Page History

...

1. Add Pipeline:AWS Steps plugin to Jenkins CI prod account
2. Create bucket on S3 CI prod account
3. Handling permissions
   1. Create a write policy specific for this bucket
      Currently, we only want to write to this specific bucket. To make it secure, only "PutObject" permission is given to this specific bucket (via ARN)
   2. Attach the policy to bucket
   3. Attach the policy to IAM Role (jenkins-slave)
4. Set Global property on Jenkins console

   1. Jenkins → Manage Jenkins → Configure System → Global Properties (helps setup global environment variables visible in every job)

      ENV Variable : Bucket name

      MXNET_CI_UNITTEST_ARTIFACT_BUCKET : <bucket-name>

5. Call S3 upload with required parameters
   1. Functions:
      1. ci/Jenkinsfile_utils.groovy → collect_test_results_unix()
      2. ci/Jenkinsfile_utils.groovy → collect_test_results_windows()
   2. PR : https://github.com/apache/incubator-mxnet/pull/16336

Design Choices

Instead of uploading unittest data to S3 for every single PR, design choice was made to upload it only for PR-merge (when PR gets merged into the master i.e. commit made to master).

Why? Tracking for every commit of PRs would be unfeasible, and unstable. On the other hand, PR merges are generally stable and would be "relevant" to tracking health of the master build.

FAQ

Q. Why upload to S3 instead of directly putting in CloudWatch?

...

Q. What happens to S3 upload if jobs fails?

A. Way CI handles job failures is - jenkins master retriggers the job on Jenkins slaveAs it stands, if the job fails, S3 upload fails. We catch the exception. We don't want the CI job to fail if data doesn't get uploaded to S3. Master build will be re triggered for next PR merge.

Stage 2: WIP

Stage 3: WIP