WORK IN PROGRESS

Overview

This document describes a draft proposal for NuttX continuous integration.

Email context

Part of the info in this document is related to threads in the dev@nuttx.apache.org mailing list. Some related threads by subject are the following:

• Subject "Jenkins CI": generic talk about the Jenkins CI and this document
• Subject "Build testing": info about the build testing scripts
• Subject "Code review tool": info about the code review tool (nxstyle)
• Subject "Test repository": info about were to put the CI/testing stuff.

Authors

• Miguel Herranz (miguel@midokura.com)
• ... (collaborate and put your name here too )

Goals

The goals of continuous integration are the following:

• Avoid "integration hell" by provide a quick but thoughtful automated check of new code addition so that the Code Contribution Workflow can be done seamlessly.
  • Definition of quick is difficult, but some hints are:
    • At maximum, a few minutes to rebase local developer work with latest master. Usually a developer get a lot more code than he produces, so is important the rebase to be easy to do, or the developer will prefer to just keep working for a few days, leading to potential "integration hell".
    • At maximum, a couple hours to do minimum set of checks in the CI server. This is needed to not saturate CI service.
    • Less than a day for a full regression including all the tests (nightly tests). This is needed to have a daily stable base.
    • Fail fast approach: the fastest tests should be the first one to be tested, and they should provide feedback right away, stopping any further processing until they are addressed so that CI resources could be freed for testing other changes. Nevertheless, the CI pipeline should be reproducible by local developer environment, allowing the developer to test before submit, so minimizing the developer coding loop.
  • Definition of thoughtful is even more difficult, but some hints are:
    • Check the source code style
    • Check it builds and executes in the main architectures
    • Check it don't break the main functionalities
    • Check that all the new code is covered by the existing or new added tests.
    • Provides useful logs, errors and reports, so that is easy for the developer to fix the problems.
    • The results are consistent (the result of a given commit does not change over time or execution environment)
  • By automated, it would be executed in a CI like ASF Jenkins service. However, the CI platform specific part should be clearly separated from the parts that should be common to the testing in local developer environment. For example, if there is a Jenkins pipeline that is executed, it should use some external script that can be use without a Jenkins environment, and a tool to run the script locally in a similar way to the Jenkins pipeline should be provided.
• Use standardized tools and methods whenever it is possible and practical so that more people can contribute.

In general, other common practices should be considered as goals too.

To some extension, the following table should give a hint about what to include in each situation:

Pipeline

The pipeline term is used here in a loose way. It means whatever arrange of tools and scripts to execute the CI. Specifically it does not restrict to the Jenkins CI pipelines.

In principle, all the code for the pipeline should be stored in the separate repository, so that the main NuttX code is not "contaminated" by it (see thread "Test repository" in dev@nuttx.apache.org mail list). Currently the repository to use is: https://github.com/apache/incubator-nuttx-testing

The continuous integration pipeline consist in the following steps and substeps.

1. Filter
2. Build
3. Test
   1. Static code analysis
   2. Unit tests
   3. Functional tests
   4. Integration tests
   5. Performance tests
   6. Coverage tests

Each one of these steps should be usually reflected by a step in the pipeline of the chosen CI tool, so that its UI allows easy identification of the step. They also should have easy access to 

Admit

This step will basically check that the commit is well formed and that the source code follows the NuttX coding standards.

The intention is to discard as early as possible that the code is not fulfilling the coding standards. Building and testing stages are way more costly in resources, so it would be wasteful to test all the pipeline just to discover that the code is not acceptable to start with.

The output of this step will be a report that will include all the detected problems, so that the contributor could address them all: in this case it should not fail-fast until all the errors are detected. For example, if 2 files have been modified, the tool should not stop after finding one error in one of the files. The idea of fail-fast is to save resources, but in this case, the compute resource wasted is minimal, and the weight of the "contributor's time resource" is higher.

C language

See NuttX C Coding Standard.

Proposed tools:

• nxstyle: this is a tool available under tools/nxstyle.c in incubator-nuttx repository.

Build

This step will check that source code can be build in all or at least the main architectures.

Proposed tools:

• tools/testbuild.sh in incubator-nuttx repository

In principle not all the architectures should be tested, so at least one configuration for minimal test and another for full one is expected.

The output of this step should be any artifact that could be used by the test step. They should be complete enough to be able to be downloaded and successfully tested on local developer environment, if desired.

Test

Static code Analysis

Currently no static code analysis tests are done in NuttX, so this will kind of tests are out of the scope for the initial CI pipeline.

Unit test

These tests should test individual functions, like simple API calls, etc. 

Some of them are under testing/ directory in incubator-nuttx-apps repository.

Functional tests

It seems most of the current test available in NuttX are in this category, despite this classification could be discussed. Most of them are under testing/ directory in incubator-nuttx-apps repository.

Integration tests

These tests should test NuttX integration with external and third-parties code. 

Currently no integration tests are done in NuttX (to be checked), so this will kind of tests are out of the scope for the initial CI pipeline.

Performance tests

Currently no performance test are done in NuttX, so this will kind of tests are out of the scope for the initial CI pipeline.

Coverage tests

These tests should gather the coverage reports of previous test steps, and generate a report about the coverage (absolute and relative to the just code in the contribution (GIT commit) under test).

Currently no coverage output is generated in previous tests, so this will kind of tests are out of the scope for the initial CI pipeline.

Implementation

Current proposal for implementation is to generate three Jenkins pipeline jobs: incubator-nuttx, incubator-nuttx-apps, incubator-nuttx-testing (specially the first one).

The execution will consist on a Jenkins multibranch pipeline defined by the contents of tools/Jenkinsfile file in the root each repository.

Using a multibranch pipeline allows to properly track the master, release and other branches and also the ones created due to pull requests. It also integrates smoothly with new Jenkins interfaces (Blue Ocean) for better/simpler visualization of the pipelines.

Pipeline versioning

As commented, a Jenkins multibranch pipeline defined by the contents of tools/Jenkinsfile file in the root each repository. But importantly, as demanded by explicit requirement, that file will not contain the logic of the pipeline itself. Instead, it will use a pipeline library defined in the incubator-nuttx-testing  repo.

This way, the new changes to the CI logic will not need changes in the main repos, like incubator-nuttx or incubator-nuttx-apps and all the changes will go to incubator-nuttx-testing.

Triggers

The triggers of the pipelines will be:

• Pull requests master branch (or release/topic branch) of incubator-nuttx repository
• Pull requests master branch (or release/topic branch) of incubator-nuttx-apps repository
• Pull requests master branch (or release/topic branch) of incubator-nuttx-testing repository

NOTE: the Jenkins multibranch pipeline jobs will be configured to restrict pull requests that modify the tools/Jenkinsfile to committers, to avoid potential security issues by abuse of not trusted users.

Environment

The jobs in the pipeline will have the following filesystem as part of their environment:

• nuttx/ will contain the contents of the master branch (or release/topic branch) of incubator-nuttx  repository, or the trigger commit, in case it belongs to that repository.
• apps/ will contain the contents of the master branch (or release/topic branch) of incubator-nuttx-apps  repository, or the trigger commit, in case it belongs to that repository.
• testing/ will contain the contents of the master branch (or release/topic branch) of incubator-nuttx-testing  repository, or the trigger commit, in case it belongs to that repository.

Pipeline example

Here is a draft of what could be the pipeline script:

TODO

This is a TODO list with questions and issues to discuss:

• Is there a "POSIX compliance" test suite? Making a quick web search reveals some projects exists (http://posixtest.sourceforge.net/, https://github.com/juj/posixtestsuite, etc) aside the official one (http://www.opengroup.org/testing/testsuites/posix.html) so maybe it could be used in testing NuttX compliance to standards.
• In order to test NuttX in more platforms and real boards, specialized CI tools could be added. For example, like open source ones like https://kernelci.org/ (that leverages https://lavasoftware.org/).