This document describes the architecture of the current CI system hosted at http://jenkins.mxnet-ci.amazon-ml.com/

State persistence and versioning

Analysis

To preserve the state of the Jenkins Master, the various files in the Jenkins configuration directory at /var/lib/jenkins have to be considered. In general, these files can be categorized as follows:

Config: Configuration files
Cache: Temporary cache files to improve performance
State: Related to the current and past execution
Secrets: Special configuration files which have to be handled with pre-caution (aka must not be stored on a versioning system but rather in a secure storage like KMS).

While Config and Secrets are a crucial part of the CI system and thus require detailed backups and versioning, the Cache can be stored in a non-persistent way. It would be desirable to have the State persisted as this allows to have a seamless experience in between deployments. Due to the fact that the State changes very frequently, no explicit versioning is required.

The analysis and categorization of the existing files is as follows:

caches: Cache
fingerprints: State
HOOK.groovy.d: Config
jobs: Config
	incubator-mxnet: Config
		branches: State
		indexing: State
		config.xml: Config
		state.xml: Config

	... 
logs: State
nodes: Config
plugins: Config
secrets: Secrets
updates: Config
userContent: Config
users: Secret State
workflow-libs: Config
workspace: State


com.cloudbees.jenkins.plugins.bitbucket.endpoints.BitbucketEndpointConfiguration.xml: Config
com.orctom.jenkins.plugin.buildtimestamp.BuildTimestampWrapper.xml: Config
config.xml: Config AND Secret
credentials.xml: Secret
github-plugin-configuration.xml: Config
hudson.model.UpdateCenter.xml: Config
hudson.plugins.ansicolor.AnsiColorBuildWrapper.xml: Config
hudson.plugins.build_timeout.operations.BuildStepOperation.xml: Config 
hudson.plugins.emailext.ExtendedEmailPublisher.xml: Config
hudson.plugins.git.GitSCM.xml: Config
hudson.plugins.git.GitTool.xml: Config
hudson.plugins.jira.JiraProjectProperty.xml: Config
hudson.plugins.timestamper.TimestamperConfig.xml: Config
hudson.tasks.Mailer.xml: Config
hudson.tasks.Shell.xml: Config
hudson.triggers.SCMTrigger.xml: Config
identity.key.enc: Secret
javaposse.jobdsl.plugin.GlobalJobDslSecurityConfiguration.xml: Config
jenkins.CLI.xml: Config
jenkins.install.InstallUtil.lastExecVersion: Config
jenkins.install.UpgradeWizard.state: Config
jenkins.metrics.api.MetricsAccessKey.xml: Config
jenkins.model.ArtifactManagerConfiguration.xml: Config
jenkins.model.DownloadSettings.xml: Config
jenkins.model.JenkinsLocationConfiguration.xml: Config
jenkins.security.QueueItemAuthenticatorConfiguration.xml: Config
jenkins.security.UpdateSiteWarningsConfiguration.xml: Config
nodeMonitors.xml: Config
org.jenkinsci.plugins.ghprb.GhprbTrigger.xml: Config and Secret
org.jenkinsci.plugins.pipeline.modeldefinition.config.GlobalConfig.xml: Config
org.jenkinsci.plugins.resourcedisposer.AsyncResourceDisposer.xml: Config
org.jenkinsci.plugins.workflow.flow.FlowExecutionList.xml: State
org.jenkinsci.plugins.workflow.libs.GlobalLibraries.xml: Config
org.jvnet.hudson.plugins.SSHBuildWrapper.xml: Config
queue.xml.bak: State
scriptApproval.xml: Config
secret.key: Secret
secret.key.not-so-secret: Secret

Solution

asdasd