Versions Compared

Old Version 71

changes.mady.by.user Herve Boutemy

Saved on

compared with

New Version Current

changes.mady.by.user Herve Boutemy

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

( short link: https://s.apache.org/reproducible-builds )

Page properties


Status

 

Status
colour

Blue

Green
title

WIP

DONE
works with usual plugins, artifact:check-buildplan to track plugins with issues

Versionunrelated to Maven itself, see plugins versions
Issue(s)
Jira
serverASF JIRA
columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId5aa69414-a9e9-3523-82ec-879b028fb15b
keyMNG-6276
Sources
Developer(s)
 Hervé Boutemy Thomas Lorblanchès



Background Color
coloryellow

see also the Maven - Guide to Configuring for Reproducible Builds

Context

https://reproducible-https://reproducible-builds.org/ (see mailing list)
Reproducible builds are a set of software development practices that create a verifiable path from human readable source code to the binary code used by computers

How?

First, the build system needs to be made entirely deterministic: transforming a given source must always create the same result. Typically, the current date and time must not be recorded and output always has to be written in the same order.

Second, the set of tools used to perform the build and more generally the build environment should either be recorded or pre-defined.

Third, users should be given a way to recreate a close enough build environment, perform the build process, and verify that the output matches the original build.

...

But Maven plugins in the whole ecosystem (not only provided by Apache Maven team) sometimes add some variable parts that adds to the problem: timestamp text or username in MANIFEST.MF, ...

In 2015, reproducible-build-maven-plugin has been created to try to fix issues after packaging, by rewriting the archive and reworking content known for variable parts.

The goal of this proposal started in 2017 is to prepare a set of configuration and practices to have reproducible/verifiable builds at packaging time, both by enhancing java natural build behaviour and by removing some variability introduced by some Maven plugins (core plugins at first, but also in the Maven eco-system).
In parallel to this proposal, "Reproducible Maven Builds" site has been created to work on prototypes.

...

  • Timestamps:
    • Timestamps in ZIP/JAR files: file last modification time/date in central directory and file entry headers + possible optional fields "X5455_ExtendedTimestamp" (PLEXUS-ARCHIVER-48plexus-archiver PR #121)
    • Timestamp in pom.properties generated by maven-archiver (MSHARED-494 (tick))
    • Timestamp in plugin.xml and plugin-help.xml descriptors generated by maven-plugin-tools-generator (MPLUGIN-326 (tick))
    • Timestamp in MANIFEST.MF (Bnd-LastModified) generated by Felix maven-bundle-plugin
    • Timestamps in generated javadoc HTML files (can be disabled with javadoc options "notimestamp" and "bottom")
    • Timestamps in bytecode generated from Groovy code (added by GroovyClassLoader.addTimeStamp())
  • Username:
    • UID/GID in tar file entries
    • Username in MANIFEST.MF (Built-By) generated by maven-archiver (MSHARED-661 (tick))
  • Ordering:
    • Order of the file entries in a ZIP/JAR file (depends on file system order)
    • Order of the entries in the MANIFEST (MSHARED-511(tick))
    • Order of goals in plugin.xml generated by maven-plugin-tools (MPLUGIN-261(tick))
    • Order of the methods of the ObjectFactory.java file generated by JAXB/xjc (JAXB-598(tick))
    • Order of components in META-INF/plexus/components.xml generated by plexus metadata (issue #8(tick))
  • Tools Versions:
    • exact JDK version used to build in MANIFEST.MF (Build-Jdk) generated by maven-archiver (MSHARED-797(tick))
      Notice that keeping the major version of the JDK used still makes sense, since it has an influence on generated bytecode: with the same source code and defined -target version, javac from JDK 6, 7, 8, ... do not produce the same bytecode. If we want to isolate the generated binary from JDK used, the compiler used will have to not be javac provided by running JDK (see Using Non-Javac Compilers)
    • exact Maven version used to build in MANIFEST.MF (Created-By) generated by maven-archiver (MSHARED-799(tick))
    • exact Maven version used to build in META-INF/.../pom.properties generated by maven-archiver (MSHARED-800(tick))

...

Life would become easier if there was a dedicated POM element like ${project.build.outputTimestamp} (with an ISO - 8601 formatted date+time) which could be used to specify the timestamp value once per entire project. Every plugin could use it as default value, like it has been done with source files encoding:

/**
 * Timestamp for reproducible output archive entries, either formatted as ISO 8601
 * <code>yyyy-MM-dd'T'HH:mm:ssXXX</code> or as an int representing seconds since the epoch (like
 * <a href="https://reproducible-builds.org/docs/source-date-epoch/">SOURCE_DATE_EPOCH</a>).
 */
@Parameter( defaultValue = "${project.build.outputTimestamp}" )
private String outputTimestamp;

Adding this element to the POM structure without breaking backward compatibility can only happen in a future version, yet to be defined (at least after Maven 3.0, see POM Model Version 5 proposal):

<project>
  ...
  <build>
    <!-- NOTE: This is just a vision for the future, it's not yet implemented: see MNG-xxx -->
    <outputTimestamp>2019-10-02T08:04:00Z</outputTimestamp>
    ...
  </build>
  ...
</project>

For Maven 2.x and 3.x, the value can be defined as an equivalent property:

<project>
  ...
  <properties>
    <project.build.outputTimestamp>2019-10-02T08:04:00Z</project.build.outputTimestamp>
    ...
  </properties>
  ...
</project>

Thus plugins could immediately be modified to use ${project.build.outputTimestamp} default value, whatever Maven version is used.

...

What are the issues to solve?

adds "Built-Jdk: <detailed java version>" Manifest entry: better replaced with "Built-Jdk: <java specification version>"Timestamp in Maven version in META-INF/plexus/components.xmlsort components when generating sort components when merging discovered components with manually crafted component filesMETA-INF/sisu/javax.inject.Named content (created by ) has non reproducible order for contentplexus-archiver issue #114 plexus-archiver #124 issues fixed in maven-archiver will have to be picked by 9 other plugins managed by Apache Maven team (acr, ear, ejb, jlink, rar, war, site, javadoc, assembly) and perhaps other plugins managed outside Apache Maven team
issue trackingdescription
MSHARED-661 ((tick) maven-archiver 3.4.0)

META-INF/MANIFEST.MF

maven-archiver adds "Built-By: <username>" Manifest entry: the entry was removed

MSHARED-796 ((tick) maven-archiver-3.4.0)

META-INF/MANIFEST.MF
MSHARED-494 ((tick) maven-archiver MSHARED-494 ((tick) maven-archiver 3.3.1.0)META-INF/maven/$groupId/$artifactId/pom.propertiespom.properties
MSHARED-800 (tick)META-INF/maven/$groupId/$artifactId/pom.propertiespom.properties
MPLUGIN-261 ((tick) maven-plugin-plugin 3.3)META-INF/maven/plugin.xmlgenerated plugin.xml is non-deterministic
MPLUGIN-326 ((tick) maven-plugin-plugin 3.5.1)META-INF/maven/plugin.xml
META-INF/maven/$groupId/$artifactId/plugin-help.xml		Timestamp in plugin.xml and plugin-help.xml descriptors generated by maven-plugin-tools-generator
plexus-containers plexus-containers issue #8 ((tick) plexus-component-metadata 2.0.0)

META-INF/plexus/components.xml
plexus-containers issue #27 ((tick) plexus-component-metadata 2.1.0)META-INF/plexus/components.xml
bnd-maven-plugin #3521 ((tick) bnd-maven-plugin configuration)META-INF/MANIFEST.MF

"Private-Package" manifest entry content (created by felix:bundle) has not the same order between builds

FELIX-6269 ((tick) maven-bundle-plugin:manifest & bundle 4.2.2)META-INF/MANIFEST.MF
FELIX-6203 ((tick) maven-bundle-plugin:bundle 4.2.2)META-INF/maven/$groupId/$artifactId/pom.properties
sisu-maven-plugin 5e2377c ((tick) sisu.inject 0.3.4)sisu-maven-plugin PR#5META-INF/sisu/javax.inject.Named
sisu-maven-plugin
zip entries timestamp and order
COMPRESS-485 ((tick) commons-compress 1.19)keep entries order when gathering ParallelScatterZipCreator
plexus-archiver issue #48, PR #49 ((tick) plexus-archiver 4.2.1)avoid timestamp issues in archives created by plexus-archiver (widely used in Maven plugins creating jar, zip, war, tar... archives)
annotation processor 570e81 ((tick) sisu.inject 0.9.0.M1)META-INF/sisu/javax.inject.Named
MRRESOURCES-114 ((tick) maven-remote-resources-plugin 1.7.0)projectTimespan, as often printed in META-INF/NOTICE
JDK-8240734 ((tick)JDK 15)module-info.class
zip entries timestamp and order
COMPRESS-485 ((tick) commons-compress 1.19)keep entries order when gathering ParallelScatterZipCreator

plexus-archiver
- PR #118

((tick) plexus-archiver 4.2.0)

To enable reproducible builds `AbstractArchiver#addFileSet` should add the files in order

MSHARED-837 ((tick) maven-archiver 3.5.0)

support SOURCE_DATE_EPOCH environment variable or equivalent: see https://reproducible-builds.org/docs/timestamps/

=> see "Output Archive Entries Timestamp" section of the proposal

: predefined timestamp
- PR #121 ((tick) plexus-archiver 4.2.0): manage TZ
- PR #127 ((tick) plexus-archiver 4.2.1): manage DST

avoid timestamp issues in archives created by plexus-archiver (widely used in Maven plugins creating jar, zip, war, tar... archives)
plexus-archiver issue #114 remove variation based on user's umask on Unixes ((tick) plexus-archiver 4.2.0)remove variation based on uid/gid & userName/groupName in tar
MSOURCES-120apply reproducible zip (entries order and timestamp) to maven-source-plugin
MASSEMBLY-921apply reproducible archive (entries order and timestamp) to maven-assembly-plugin
MJAR-263apply reproducible zip (entries order and timestamp) to maven-jar-plugin
To enable reproducible builds `AbstractArchiver#addFileSet` should add the files in order

MSHARED-837 ((tick) maven-archiver 3.5.0)

support SOURCE_DATE_EPOCH environment variable or equivalent: see https://reproducible-builds.org/docs/timestamps/

=> see "Output Archive Entries Timestamp" section of the proposal

plexus-archiver #271 ((tick) plexus-archiver 2.7.0)remove variation based on user's umask on Unixes (particularly group write)
plexus-archiver #124 ((tick) plexus-archiver 4.2.0)remove variation based on uid/gid & userName/groupName in tar
MSOURCES-120 ((tick) maven-source-plugin 3.2.0)apply reproducible zip (entries order and timestamp) to maven-source-plugin
MSOURCES-137 ((tick) maven-source-plugin 3.3.1)apply 022 umask (to ease RB check: independent from env)
MASSEMBLY-921 ((tick) maven-assembly-plugin 3.2.0)apply reproducible archive (entries order and timestamp) to maven-assembly-plugin
MASSEMBLY-989 ((tick) maven-assembly-plugin 3.6.0)apply 022 umask (to ease RB check: independent from env)
MJAR-263 ((tick) maven-jar-plugin 3.2.0)apply reproducible zip (entries order and timestamp) to maven-jar-plugin
MSITE-851 ((tick) maven-site-plugin 3.9.0)apply reproducible zip (entries order and timestamp) to site:jar 
MJAVADOC-627 ((tick) maven-javadoc-plugin 3.2.0)apply reproducible zip (entries order and timestamp) to javadoc:*jar 
MSHADE-347 ((tick) maven-shade-plugin 3.2.2)apply reproducible zip (entries order and timestamp) to shade:shade
MSHADE-352 ((tick) maven-shade-plugin 3.2.3)keep reproducible timestamp when shading with transformer
MSHADE-420mtime from extra field data from shaded dependency makes result builder's timezone sensitive
ARCHETYPE-590 ((tick) maven-archetype-plugin 3.2.0)apply reproducible zip (entries order and timestamp) to archetype:jar 
MWAR-432 ((tick) maven-war-plugin 3.3.0)apply reproducible zip (entries order and timestamp) to war:jar 
MACR-53 ((tick) maven-acr-plugin 3.2.0)
MEAR-280 ((tick) maven-ear-plugin 3.1.0)
MEJB-128 ((tick) maven-ejb-plugin 3.1.0)
MRAR-86 ((tick) maven-rar-plugin 3.0.0)
MJLINK-75 ((tick) maven-jlink-plugin 3.0.0)apply reproducible zip to the zip file created by the plugin
issues fixed in maven-archiver will have been picked by 9 other plugins managed by Apache Maven team (acr, ear, ejb, jlink, rar), probably other plugins creating zip/jar/tar archives managed outside Apache Maven team will require to do the same
FELIX-6304 ((tick) maven-bundle-plugin:bundle 5.1.1)order and timestamp of jar entries for bundle goal
FELIX-6404 ((tick) maven-bundle-plugin 5.1.3)regression in 5.1.2
FELIX-6495 ((tick) maven-bundle-plugin 5.1.4)bundle:manifest adds Bnd-LastModified even if outputTimestamp is defined
FELIX-6496 ((tick) maven-bundle-plugin 5.1.5)
require bnd #5021((tick)6.2)

non-reproducible Export-Package and Private-Package values

FELIX-6602 ((tick) maven-bundle-plugin 5.1.9)

non-reproducible Include-Resource entry

FELIX-6681

non-reproducible Export-Service entry

spring-boot-maven-plugin:repackage #20176 ((tick) 2.3.0-M4)timestamp
org.jboss.jandex:jandex-maven-plugin #26 ((tick) 1.1.1)unsorted files (notice: not sure that sorting files at plugin level is sufficient: it seems Jandex indexer itself produces non-reproducible output)

org.jboss.jandex:jandex-maven-plugin #35 (error) 1.2.3

replaced by io.smallrye:jandex-maven-plugin #286 ((tick) 3.1.0)

jandex.idx output is not stable/reproducible
MJAR-275 ((tick)maven-jar-plugin 3.3.0)

outputTimestamp not applied to module-info.class

moditect-maven-plugin:add-module-info #185 ((tick) 1.0.0.Final)

creates module-info.class with non-reproducible timestamp

moditect #222

timestamp is dependant on timezone

SM-5021 ServiceMix depends-maven-plugin:generate-depends-file ((tick)1.5.0)timestamp in generated depends.txt
XBEAN-335 Geronimo maven-xbean-plugin timestamp in generated /META-INF/spring-* (Properties format)
KARAF-7367 karaf-maven-plugin:kar #1492 ((tick) 4.3.7)does not take outputTimestamp into account
quarkus-extension-maven-plugin #38364timestamp in generated properties META-INF/quarkus-extension.properties, META-INF/quarkus-javadoc.properties and sources *.jdp
tracking of plugins with issues and fixes is now also done in artifact:check-buildplan

Debian approach

Debian has a strong reproducible builds structure working on the topic for a few years: see BuildinfoFiles for environment info recording.

...