WORK IN PROGRESS

Overview

This document describes a draft proposal for NuttX continuous integration.

Email context

Part of the info in this document is related to threads in the dev@nuttx.apache.org mailing list. Some related threads by subject are the following:

Subject "Jenkins CI": generic talk about the Jenkins CI and this document
Subject "Build testing": info about the build testing scripts
Subject "Code review tool": info about the code review tool (nxstyle)
Subject "Test repository": info about were to put the CI/testing stuff.

Authors

Miguel Herranz (miguel@midokura.com)
... (collaborate and put your name here too )

Goals

The goals of continuous integration are the following:

Avoid "integration hell" by provide a quick but thoughtful automated check of new code addition so that the Code Contribution Workflow can be done seamlessly.
- Definition of quick is difficult, but some hints are:
  - At maximum, a few minutes to rebase local developer work with latest master. Usually a developer get a lot more code than he produces, so is important the rebase to be easy to do, or the developer will prefer to just keep working for a few days, leading to potential "integration hell".
  - At maximum, a couple hours to do minimum set of checks in the CI server. This is needed to not saturate CI service.
  - Less than a day for a full regression including all the tests (nightly tests). This is needed to have a daily stable base.
  - Fail fast approach: the fastest tests should be the first one to be tested, and they should provide feedback right away, stopping any further processing until they are addressed so that CI resources could be freed for testing other changes. Nevertheless, the CI pipeline should be reproducible by local developer environment, allowing the developer to test before submit, so minimizing the developer coding loop.
- Definition of thoughtful is even more difficult, but some hints are:
  - Check the source code style
  - Check it builds and executes in the main architectures
  - Check it don't break the main functionalities
  - Check that all the new code is covered by the existing or new added tests.
  - Provides useful logs, errors and reports, so that is easy for the developer to fix the problems.
  - The results are consistent (the result of a given commit does not change over time or execution environment)
- By automated, it would be executed in a CI like ASF Jenkins service. However, the CI platform specific part should be clearly separated from the parts that should be common to the testing in local developer environment. For example, if there is a Jenkins pipeline that is executed, it should use some external script that can be use without a Jenkins environment, and a tool to run the script locally in a similar way to the Jenkins pipeline should be provided.
Use standardized tools and methods whenever it is possible and practical so that more people can contribute.

In general, other common practices should be considered as goals too.

To some extension, the following table should give a hint about what to include in each situation:

Case	Maximum duration	Commit message format & code style	Build of nuttx and/or apps	Minimal testing	Full testing
Rebase from master in local developer environment	~1-5 minutes	always	always	optional	not recommended
Prepare local code for pull request	~1 hour	always	always	recommended	optional
Continuous integration triggered by a pull request update	~2-3 hours	always	always	always	optional
Nightly build	~12 hours	always	always	always	always

Pipeline

The pipeline term is used here in a loose way. It means whatever arrange of tools and scripts to execute the CI. Specifically it does not restrict to the Jenkins CI pipelines.

In principle, all the code for the pipeline should be stored in the separate repository, so that the main NuttX code is not "contaminated" by it (see thread "Test repository" in dev@nuttx.apache.org mail list). Currently the repository to use is: https://github.com/apache/incubator-nuttx-testing

The continuous integration pipeline consist in the following steps and substeps.

Filter
Build
Test
1. Static code analysis
2. Unit tests
3. Functional tests
4. Integration tests
5. Performance tests
6. Coverage tests

Each one of these steps should be usually reflected by a step in the pipeline of the chosen CI tool, so that its UI allows easy identification of the step. They also should have easy access to

Admit

This step will basically check that the commit is well formed and that the source code follows the NuttX coding standards.

The intention is to discard as early as possible that the code is not fulfilling the coding standards. Building and testing stages are way more costly in resources, so it would be wasteful to test all the pipeline just to discover that the code is not acceptable to start with.

The output of this step will be a report that will include all the detected problems, so that the contributor could address them all: in this case it should not fail-fast until all the errors are detected. For example, if 2 files have been modified, the tool should not stop after finding one error in one of the files. The idea of fail-fast is to save resources, but in this case, the compute resource wasted is minimal, and the weight of the "contributor's time resource" is higher.

C language

See NuttX C Coding Standard.

Proposed tools:

nxstyle: this is a tool available under tools/nxstyle.c in incubator-nuttx repository.

Build

This step will check that source code can be build in all or at least the main architectures.

Proposed tools:

tools/testbuild.sh in incubator-nuttx repository

In principle not all the architectures should be tested, so at least one configuration for minimal test and another for full one is expected.

The output of this step should be any artifact that could be used by the test step. They should be complete enough to be able to be downloaded and successfully tested on local developer environment, if desired.

Test

Static code Analysis

Currently no static code analysis tests are done in NuttX, so this will kind of tests are out of the scope for the initial CI pipeline.

Unit test

These tests should test individual functions, like simple API calls, etc.

Some of them are under testing/ directory in incubator-nuttx-apps repository.

Functional tests

It seems most of the current test available in NuttX are in this category, despite this classification could be discussed. Most of them are under testing/ directory in incubator-nuttx-apps repository.

Integration tests

These tests should test NuttX integration with external and third-parties code.

Currently no integration tests are done in NuttX (to be checked), so this will kind of tests are out of the scope for the initial CI pipeline.

Performance tests

Currently no performance test are done in NuttX, so this will kind of tests are out of the scope for the initial CI pipeline.

Coverage tests

These tests should gather the coverage reports of previous test steps, and generate a report about the coverage (absolute and relative to the just code in the contribution (GIT commit) under test).

Currently no coverage output is generated in previous tests, so this will kind of tests are out of the scope for the initial CI pipeline.

Implementation

Current proposal for implementation is to generate a Jenkins pipeline job, called nuttx.

Pipeline versioning

The execution will consist on a Jenkins Pipeline defined by the contents of Jenkinsfile file in the root the ci-configuration branch of incubator-nuttx-testing repository (see https://jenkins.io/doc/book/pipeline/getting-started/#defining-a-pipeline-in-scm).

Triggers

The triggers of the pipeline will be:

Pull requests master branch (or release/topic branch) of incubator-nuttx repository
Pull requests master branch (or release/topic branch) of incubator-nuttx-apps repository
Pull requests master branch (or release/topic branch) of incubator-nuttx-testing repository

WARNING: the special branch in incubator-nuttx-testing repository, called ci-configuration, will have a special functionality as explained before, and must not be used as trigger source. This branch will not trigger any automated job from its pull request. Instead, it should be tested using another Jenkins pipeline job, called nuttx-testing-ci-configuration, that should be triggered manually by Apache NuttX committers with enough privileges. This will allow to test in a more controlled way the changes in the pipeline itself.

Environment

The jobs in the pipeline will have the following filesystem as part of their environment:

nuttx/ will contain the contents of the master branch (or release/topic branch) of incubator-nuttx repository, or the trigger commit, in case it belongs to that repository.
apps/ will contain the contents of the master branch (or release/topic branch) of incubator-nuttx-apps repository, or the trigger commit, in case it belongs to that repository.
testing/ will contain the contents of the master branch (or release/topic branch) of incubator-nuttx-testing repository, or the trigger commit, in case it belongs to that repository.

Pipeline example

Here is a draft of what could be the pipeline script:

pipeline {

agent any

stages {

stage('Admit') {

steps {

sh './testing/ci/check-commit-message.sh'

sh './nuttx/tools/nxstyle ...'

}

stage('Build') {

steps {

sh './nuttx/tools/testbuild.sh check'

}

stage('Test') {

steps {

sh './apps/testing/foo ...'

sh './apps/testing/bar ...'

sh './testing/ci/test-report.sh ...'

}

TODO

This is a TODO list with questions and issues to discuss:

Is there a "POSIX compliance" test suite? Making a quick web search reveals some projects exists (http://posixtest.sourceforge.net/, https://github.com/juj/posixtestsuite, etc) aside the official one (http://www.opengroup.org/testing/testsuites/posix.html) so maybe it could be used in testing NuttX compliance to standards.
In order to test NuttX in more platforms and real boards, specialized CI tools could be added. For example, like open source ones like https://kernelci.org/ (that leverages https://lavasoftware.org/).