This project uses Scala Steward to automatically create pull requests to update the project when new dependencies are released.

Required Checks

Before provding a +1 and merging pull requests created by Scala Steward, the following checks should be performed

  • Do all automated continuous integration checks pass?
  • Is the update a patch, minor, or major update?
    • Major updates need more stringent review criteria than the other updates, at least a manual reading of the library's release notes / changelog to check for API changes.
  • Is the license still compatible with ASF License Policy?
  • Have any changes been made to LICENSE/NOTICE files that need to be incorporated? 
    • Note that something as simple as a date change in a LICENSE/NOTICE file requires an update.
  • Have any transitive dependencies been added or changed?
    • If so, we must perform all the same checks above for those dependenies as well.

Making Updates

If changes are needed, such as updates to allow the integration tests to pass or to update a LICENSE/NOTICE file, the following steps can be performed:

  1. Add the Scala Steward daffodil fork as a new remote:

    git remote add scala-steward git@github.com:scala-steward/daffodil.git

  2. Fetch the branch associated with the new dependency:

    git fetch scala-steward

  3. Checkout the branch:

    git checkout update/jackson-core-2.11.4
  4. Commit any necessary changes to this branch.
       

  5. Push changes back to the scala-steward remote:

    git push scala-steward update/jackson-core-2.11.4
  6. Follow the Code Contributor Workflow just as usual, eventually squashing changes, and using push --force to push those changes back to the scala-steward remote to update pull request before being merged.
  • No labels

3 Comments

  1. Mike Beckerle

    I'm not exactly sure how to codify these things into the checklist, but to me these are essential also:

    • For apache dependencies, we know we can do all these things:
      • verify binaries of dependency are available from maven-central
      • verify digital signatures, and hashes of binaries
      • verify signing keys - e.g., who exactly is signing?
        • Note: we need to establish web-of-trust relationships with those signers at these key-exchanges where people swap keys at face-to-face events (which have to be virtual these days)
    • For apache dependencies - we may not need to do all of the above. We may just want to verify in the dev list for that project that these steps were verified by that project's participants/PMC.
    • For non-apache dependencies, we need to figure out the equivalent level of trust in binaries, but this may vary by the dependency itself, depending on where it is hosted, where the binaries are served from, etc.

    Another thing to check:

    • Verify that dependency versions from daffodil are for specific exact versions

    These things may be beyond state of the art, but I wanted to comment on them anyway:

    • Verify that transitive dependencies are also for specific exact versions.
      • Why? - this notion of dependencies on "at least version X" or "from version X to Y are allowed" is the source of numerous supply-chain compromises. At this point I consider only exact dependencies to be sensible.
    • I'd like to have a way to verify the complete set of servers from which dependencies were satisfied are all acceptable trusted servers, and a way to verify those servers really ARE the servers they say. I.e., DNS attacks can't have redirected traffic to a false site where dependencies are being proxied, but some are potentially being satisfied with substituted binaries.
      • Why? - this has been a way that supply-chain compromises work.



  2. Steve Lawrence

    We might want to look into https://github.com/josephearl/sbt-verify for validating dependencies. Unfortnately, this requires a bit of work and I think manual update of dependencies, but an automated method like that is likely the best way to handle verification of maven binaries. I'll also point of that it's not even possible for all Apache binaries. Technically, ASF only requires that you sign release sources. It's very likely many Apache release don't sign helper binaries or thouse published to maven central.

    Verify that dependency versions from daffodil are for specific exact versions

    Does sbt even allow you to specify a non exact version? Something to confirm. This may be a non-issue.

    Verify that transitive dependencies are also for specific exact versions.

    I suspect sbt-verify can also handle this. Though, that raise a good point, we need to verify that transitive dependencies have not changed, and if they have, then we must perform all the same checks for those as well.

    I'd like to have a way to verify the complete set of servers from which dependencies were satisfied are all acceptable trusted servers.

    The list of servers that are used come from sbt. Though, I would argue we don't really care where things come from, especially since new mirors might be spun up all the time. What we really care about is that what we get matches some known hashes, which I think sbt-verify can handle.

    1. Mike Beckerle

      Hmmm. If projects have unsigned helper binaries, ultimately we need to be building them from source then.

      As an Owl engineer (with that hat on), I'd say this is ok, as we have lots of requirements to only bring source code (signed) across from untrusted environments onto trusted development networks, so this should be something we are willing to make routine and automated.