Context for this meeting: White House national security adviser asks software companies to discuss cybersecurity.

The content below was developed on the security-discuss@community.apache.org mailing list, and perusing the archives there from late December 2021 to early January 2022 may provide important context.  Everyone is invited to participate there, but be aware that posts to that mailing list are publicly archived.

The best place to get started to know what the ASF is is here: https://apache.org/. That page is full of statistics, videos, and links.

The following links in particular are important background to this discussion:

• https://www.apache.org/foundation/how-it-works.html
• https://www.apache.org/dev/
• https://www.apache.org/dev/pmc.html
• https://apache.org/security/
• https://apache.org/theapacheway/index.html
• https://www.apache.org/foundation/getinvolved.html

Key bullet points

• The entire "supply chain" needs to be considered:

  • Software developed at the ASF is made available at no cost and without warranty

  • Commercial products may include this software without entering into any form of contract with the ASF, or even notifying us

  • End users may purchase these products but have little interest or ability to apply fixes
• This will require collective action:
  • There are things we can do, both individually and together, to reduce the number of vulnerabilities.
  • There are things, such as SBOMs, that can help identify what is affected once a vulnerability is found.
  • Much of this is moot if patches are never applied.
• On the topic of volunteers/community/participation:
  • Our contributors tend to be seasoned software professionals whose employers include ASF releases in their commercial products.
  • Our communities are healthy, open, and transparent.
  • As a result of vulnerability reports, more attention is focused on the specific product, at least for a period of time.  Open source enables a more diverse group of people to participate in these efforts.
  • Companies and government agencies that want to help don't need money or formal contracts to do so.  Join our mailing lists, review our code, contribute fixes.

Background reading:

• EO - Mark Cox - https://s.apache.org/3nctr
• SBOM - David Nalley - https://s.apache.org/hccur
• Applying updates - Mark Thomas - https://s.apache.org/5jqab
• Collective action - Phil Steitz - https://s.apache.org/ljzn0
• Volunteers - Sam Ruby - https://s.apache.org/3vkpr
• Contributors/maintenance - Dominik Psenner - https://s.apache.org/3lrk1
• CISA - David Nalley - https://s.apache.org/1gr1c
• White box testing - John Kinsella - https://s.apache.org/hn2ba
• 2016 EU Audit - Dirk-Willem van Gulik - https://s.apache.org/fhoji

There are really three main supply-chain related ideas that people need to get:

• While we may not all agree at this point that it works, our biggest defense against vulnerabilities getting introduced is eyeballs, which our open development process supports.  Any interested human (or bot) can be a QC inspector in our direct contribution part of the supply chain.  We have always welcomed this, especially when reports come with patches.
• There is a natural nesting that happens as software dependencies propagate through applications.  Addressing vulnerabilities in base level components (e.g. log4j) has a cascading impact.  Unless and until all downstream systems have effectively automated build, test and deployment systems, this creates systemic risk which has nothing to do with OSS per se.
• End of life is an industry standard concept that we apply at the ASF.  End of life means no more patches.  End of life software running in critical systems is another systemic risk.

Annotated notes and links

• Foundation organization and governance structure (https://www.apache.org/foundation/governance/)

The ASF is an operating US 501(c)3  organization that supports several hundred open source projects.  The operating part of the ASF is managed by the President, supported by the Treasurer, Vice Presidents, committees and third parties.  The ASF board provides oversight for operating functions as well as ASF project communities.  The Legal Affairs, Security, and Treasurer departments report directly to the board, which means, for example that the Security team acts with the authority of the board.

• How oversight works (https://www.apache.org/foundation/how-it-works.html)

ASF projects are run by Project Management Committees (PMCs).  PMC  members review and cast binding votes on releases, nominate and vote on  new PMC members and collectively manage community, security, trademarks  or other problems.  The PMCs act on behalf of the ASF when they cut  releases.  The PMC plays a key role in technical project oversight, but not the only role.  The full project community contributes by working creating and reviewing) patches and contributing to discussion.

All ASF community members, including PMC members, officers and the board, participate in our communities as individuals, not as representatives of companies or other organizations.  Some community members are paid by their employers to contribute to ASF projects but in their ASF work they are expected to act in the best interest of the community, exercising their own best judgement.  Impact on project decisions is based on publicly earned merit which accrues to individuals, based only on their contributions at the ASF.

• How development works (https://www.apache.org/dev/pmc.html)

ASF projects don’t all work the same and the ASF does not prescribe any kind of SDLC for our projects.  What we do require is that everything happens in the open:

• • All bug reports are visible to all interested parties (with exception of reported security vulnerabilities, which are kept private until patched)
  • All commits to project source code are tracked and visible to all interested parties
  • All project-related discussion that can be public is public

ASF projects differ in how they do things, but most have documented process for working bug reports, introducing new features and planning and executing releases. As new features are developed and bugs are addressed, code changes are made that will go into releases.  Typically, proposed changes are submitted as “pull requests” (PRs) that explicitly describe how the source will change if they are applied.  The original source, the PR and any changes that result from applying it are all publicly visible.  Any interested party can comment on proposed changes and it frequently happens that performance, security or other issues are discovered in the open development process that creates and modifies ASF software.

• How releases are versioned, validated and distributed (https://www.apache.org/dev/#releases)

ASF software is distributed and meant to be consumed in the form of versioned releases.  ASF PMCs have various ways of packaging software for distribution, but the core asset being released is always source code. Releases must be voted on by PMCs, who are responsible for validating release candidates (RCs).  Most PMCs also make RCs available to the public prior to or during release votes. Library or infrastructure projects sometimes make backward-incompatible changes as they develop new versions of their software.  Sometimes, a backward-compatible version of the software is added and maintained in parallel, but it often happens that the actively supported version is not backward compatible with an older version.   When library or infrastructure components make backward incompatible changes, downstream users may have to make code changes to upgrade their applications.

• How ASF code is incorporated in other software

ASF software is used in running systems in basically 2 ways:

• • A distributed system includes a running ASF product (e.g. Apache web server)
  • ASF software is integrated with other software to build applications.  When applications created in 2. are themselves used to create other applications, nesting happens, causing the impact of a vulnerability or incompatible change at the lowest level to have a cascading impact.   When a new release is made available by an ASF project, downstream users need to minimally make configuration changes and test their systems.  In case 2. above they typically have to rebuild and redeploy their applications.  In nested situations, this often has to happen sequentially.

• How security reports are handled (https://www.apache.org/security/)

The ASF encourages responsible disclosure of security vulnerabilities discovered in software managed by ASF projects.  The ASF security team sets a common policy, maintains security contacts for PMCs, and provides support for projects responding to security issues. Reporters are encouraged to use the designated security contacts to report vulnerabilities privately.  PMCs are required to respond to security reports promptly, working with reporters to investigate and if necessary develop patches.   The ASF Security team assist the PMC in publishing a CVE once the vulnerability has been patched and a release containing the patch has been made available.  The security team oversee all reported issues across all ASF projects.

• How end of support works

Like all software products, ASF code lines go through a lifecycle that terminates in end of support.  Typically, ASF PMCs communicate end of support dates for products or code lines with ample time (one year plus) for users to plan upgrades or find alternatives.  Once communicated end of life dates arrive, the PMC is no longer expected or required to accept or apply patches to the end of life versions. This includes security patches against identified vulnerabilities. Therefore in some cases, when users who are depending on no longer supported versions of ASF software,  a version upgrade will be required to obtain a fix of an identified vulnerability and that may require code changes to integrate.