You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 59 Next »

Status
 WIP
Version
Issue(s)
Unable to render Jira issues macro, execution error.
Sources
Developer(s)
 Hervé Boutemy Thomas Lorblanchès

https://reproducible-builds.org/ (see mailing list)
Reproducible builds are a set of software development practices that create a verifiable path from human readable source code to the binary code used by computers

How?

First, the build system needs to be made entirely deterministic: transforming a given source must always create the same result. Typically, the current date and time must not be recorded and output always has to be written in the same order.

Second, the set of tools used to perform the build and more generally the build environment should either be recorded or pre-defined.

Third, users should be given a way to recreate a close enough build environment, perform the build process, and verify that the output matches the original build.

Tooling like diffoscope have been created to measure differences between archives content.

Java builds are naturally not immediately reproducible: timestamps in jar files are the first source of non-idempotence (if you do a build twice with just javac and jar commands, the result won't be the same bit for bit).

But Maven plugins in the whole ecosystem (not only provided by Apache Maven team) sometimes adds some variable parts that adds to the problem: timestamp text or username in MANIFEST.MF, ...

reproducible-build-maven-plugin has been created to try to fix issues after packaging.

The goal of this proposal is to prepare a set of configuration and practices to have reproducible/verifiable builds at packaging time, both by enhancing java natural build behaviour and by removing some variability introduced by some Maven plugins (core plugins at first, but also in the Maven eco-system).
In parallel to this proposal, "Reproducible Maven Builds" site has been created to work on prototypes.

Use cases

  1. As a user of artifacts published on repositories like Maven Central, I want to be able to check that the binary version of the artifact matches its source version.
    On a software QA point of view, this would allow to detect quality problems in the build/publish process.
    On a computer security point of view, this would allow to detect the introduction of a backdoor during the build/publish process (instead of other solutions based on checking signatures like envisioned in MNG-6026).
  2. As a developer voting on an Apache source release against a staging repository, I want to verify that the binary from my local build from sources is the same as the binary that is staged and signed by the release manager

Sources of unreproducible bits

  • Timestamps:
    • Timestamp in pom.properties generated by maven-archiver (MSHARED-494 (tick))
    • Timestamp in plugin.xml and plugin-help.xml descriptors generated by maven-plugin-tools-generator (MPLUGIN-326 (tick))
    • Timestamp in MANIFEST.MF (Bnd-LastModified) generated by Felix maven-bundle-plugin
    • Timestamps in ZIP/JAR files: file last modification time/date in central directory and file entry headers + possible optional fields "X5455_ExtendedTimestamp" (PLEXUS-ARCHIVER-48)
    • Timestamps in generated javadoc HTML files (can be disabled with javadoc options "notimestamp" and "bottom")
    • Timestamps in bytecode generated from Groovy code (added by GroovyClassLoader.addTimeStamp())
  • Username:
    • Username in MANIFEST.MF (Built-By) generated by maven-archiver (MSHARED-661 (tick))
    • UID/GID in tar file entries
  • Ordering:
    • Order of the file entries in a ZIP/JAR file (depends on file system order)
    • Order of the entries in the MANIFEST (MSHARED-511(tick))
    • Order of goals in plugin.xml generated by maven-plugin-tools (MPLUGIN-261(tick))
    • Order of the methods of the ObjectFactory.java file generated by JAXB/xjc (JAXB-598(tick))
    • Order of components in META-INF/plexus/components.xml generated by plexus metadata (issue #8(tick))
  • Tools Versions:
    • exact JDK version used to build in MANIFEST.MF (Build-Jdk) generated by maven-archiver (MSHARED-797(tick))
      Notice that keeping the major version of the JDK used still makes sense, since it has an influence on generated bytecode: with the same source code and defined -target version, javac from JDK 6, 7, 8, ... do not produce the same bytecode. If we want to isolate the generated binary from JDK used, the compiler used will have to not be javac provided by running JDK (see Using Non-Javac Compilers)
    • exact Maven version used to build in MANIFEST.MF (Created-By) generated by maven-archiver (MSHARED-799(tick))
    • exact Maven version used to build in META-INF/.../pom.properties generated by maven-archiver (MSHARED-800(tick))

Line endings is also a problem, and even if we could force given line endings for build-generated text files (MANIFEST, pom.properties...), it would be hazardous to try to change the line endings of the resource files.

The underlying problem is that the pom file does not capture all the configuration of the build environment: it includes the plugins used during the build with their version number, but it does not include the version of Maven and the JDK, the Operating System and the architecture used to produce the artifact, etc...

What can we do? (non-exhaustive list):

  • Drop the Maven/JDK version numbers in MANIFEST: most of the time you should get exactly the same result with two different versions of Maven and/or JDK (if you keep the same major version number). But you have to restrain to a given OS family because of line endings. However, with the new development roadmap of OpenJDK (2 "major" versions per year, each one increasing the class file version number), it may be difficult to find 2 versions of javac that produce the same class files in the near future.
  • Keep the version numbers in the MANIFEST: there are not easily accessible here. Moreover the semantic is poor (e.g. there is no JDK vendor so you can have different results if you use OpenJDK/Oracle JDK/Eclipse compiler, no reference of the Operating System used for line endings).
  • Add a "reproducible build bill of materials" in an external file. This is the way Debian took to manage reproducible builds: they record the "build environment" in an external ".buildinfo" file that has all the information required to reproduce the build environment. If the frame of Maven we can think of several ways to achieve that (non-exhaustive list):
    • Create a secondary artifact (e.g. *-buildinfo.xml) with the required information
    • "Patch" the published pom file to add properties with the required information, something like:
<properties>
  <maven.reproducible.build.maven.version>3.5.0</maven.reproducible.build.maven.version>
  <maven.reproducible.build.jdk.version>8u123</maven.reproducible.build.jdk.version>
  <maven.reproducible.build.jdk.vendor>openjdk</maven.reproducible.build.jdk.vendor>
  <maven.reproducible.build.arch>amd64</maven.reproducible.build.arch>
  <maven.reproducible.build.os>linux</maven.reproducible.build.os>
</properties>

This buildinfo for the jvm is currently a work in progress on reproducible-builds.

Another way to ease the reproducibility would be to use a wrapper script that would download from Maven Central the exact Maven & JDK versions that should be used to build the project. It is the same kind of idea than the maven-wrapper tool, but extended to the JDK itself. This feature would also benefit people not interested in reproducible builds because it would ease the computer setup of every developer and erase most of the discrepancies between developers builds and CI builds.

What are the issues to solve?

issue trackingdescription
MSHARED-661 ((tick) maven-archiver 3.4.0)

META-INF/MANIFEST.MF

maven-archiver adds "Built-By: <username>" Manifest entry: the entry was removed

MSHARED-796 ((tick) maven-archiver-3.4.0)

META-INF/MANIFEST.MFmaven-archiver adds "Built-Jdk: <detailed java version>" Manifest entry: better replaced with "Built-Jdk: <java specification version>"
MSHARED-494 ((tick) maven-archiver 3.1.0)META-INF/maven/$groupId/$artifactId/pom.propertiesTimestamp in pom.properties
MSHARED-800 (tick)META-INF/maven/$groupId/$artifactId/pom.propertiesMaven version in pom.properties
MPLUGIN-261 ((tick) maven-plugin-plugin 3.3)META-INF/maven/plugin.xmlgenerated plugin.xml is non-deterministic
MPLUGIN-326 ((tick) maven-plugin-plugin 3.5.1)META-INF/maven/plugin.xml
META-INF/maven/$groupId/$artifactId/plugin-help.xml		Timestamp in plugin.xml and plugin-help.xml descriptors generated by maven-plugin-tools-generator
plexus-containers issue #8 ((tick) plexus-component-metadata 2.0.0)
META-INF/plexus/components.xmlsort components when generating META-INF/plexus/components.xml

META-INF/MANIFEST.MF

"Private-Package" manifest entry content (created by felix:bundle) has not the same order between builds

sisu-maven-plugin PR#5META-INF/sisu/javax.inject.Named

META-INF/sisu/javax.inject.Named content (created by sisu-maven-plugin) has non reproducible order for content

zip entries timestamp and order
COMPRESS-485 ((tick) commons-compress 1.19)keep entries order when gathering ParallelScatterZipCreator
plexus-archiver issue #48, PR #49 ((tick) plexus-archiver 4.2.0)avoid timestamp issues in archives created by plexus-archiver (widely used in Maven plugins creating jar, zip, war, tar... archives)
plexus-archiver issue #114 ((tick) plexus-archiver 4.2.0)To enable reproducible builds `AbstractArchiver#addFileSet` should add the files in order

MSHARED-505

support SOURCE_DATE_EPOCH environment variable or equivalent: see https://reproducible-builds.org/docs/timestamps/

remove variation based on user's umask on Unixes
MSOURCES-120apply reproducible zip (entries order and timestamp) to maven-source-plugin
MASSEMBLY-921apply reproducible archive (entries order and timestamp) to maven-assembly-plugin
MJAR-263apply reproducible zip (entries order and timestamp) to maven-jar-plugin
issues fixed in maven-archiver will have to be picked by 9 other plugins managed by Apache Maven team (acr, ear, ejb, jlink, rar, war, site, javadoc, assembly) and perhaps other plugins managed outside Apache Maven team

Debian approach

Debian has a strong reproducible builds structure working on the topic for a few years: see BuildinfoFiles for environment info recording.

On java and Maven issues, Debian maintains a serie of patches that perhaps could be integrated (thank you Emmanuel Bourg for the summary):

REX on Clojure: source .clj must have one second difference with .class, or Clojure will recompile

  • No labels