*** Unsent and unapproved draft ***
About the Apache Software Foundation
The ASF is a US 501(c)3 non-profit charity that acts as a steward for several hundred open source projects. The ASF is one of the largest open source organizations in the world, and is home to prolific projects such as Apache Hadoop, Apache Tomcat, Apache Cassandra, and scores of others. The ASF provides a known and vetted set of governance, intellectual property, and release processes as well as providing a vendor-neutral place for contributors to collaborate. However, within those processes there is a high degree of autonomy for each project. Projects decisions are driven by the people doing the work. The ASF Board of Directors is responsible for project oversight, ensuring the health and vitality of each project.
In brief numbers the ASF has:
- 227 Million lines of code
- 630,000+ contributors
- 8,370+ committers (individuals who have earned the status of being able to directly change source code).
- 200+ projects
How the ASF keeps our projects secure
- ASF projects are run by Project Management Committees (PMCs). PMC members nominate and vote on new PMC members and collectively manage community, security, trademarks, or other problems.
- Projects to have to adhere to various foundation-wide policies including trademarks, release policy, security vulnerability handling policy. ASF policies require PMC reviews with minimum levels of voting required for new releases.
- All changes to project source code are tracked and visible to all interested parties.
- The ASF has a security team and a VP of security and acts with board authority. The ASF security team sets a common policy, maintains security contacts for PMCs, and provides support for projects responding to security issues.
About the recent issues in log4j
Before jumping directly into recommendations, it is worth describing the factors contributing to the recent log4j vulnerability. Our recommendations are based on those experiences as well as over 20 years of dealing with vulnerabilities in open source software.
- The IETF, W3C, and WHATWG are examples of standards organizations. Companies and invited experts collaborate there to produce specifications of standards, formats, and protocols. Along the way, those companies and individuals contribute their intellectual property to make this happen. Those standards are then implemented by commercial and open source products.
- LDAP is an example of one such standard. It is defined at the IETF. It's original use was for things like phone books and organization charts.
- Java is a programming language, and OpenJDK is the open source implementation of that language. Java contains a component, named JNDI ,that implements LDAP as well as a number of other standards such as CORBA and RMI. Those other standards provide the ability to do things like remote code execution which was popular in the 1990s, but now isn't quite so important.
- The Apache Software Foundation is an example of an open source organization. Companies and invited experts collaborate there to produce releases of software components and products. Along the way, those companies and individuals contribute their intellectual property to make this happen. Those releases are then incorporated into commercial and other open source products.
- Log4J is one such example of a software component produced by the ASF. It does the mundane job of producing an audit trail of important activities. Augmenting the audit trail with organization data from LDAP seemed like a good idea at the time. As Log4J is written in Java, it made use of JNDI to access that data.
- Put together, and you have a mundane and almost invisible function which can be tricked into doing remote code execution. There have been prior efforts to identify critical functions, yet somehow log4j never ranked high on any list. Furthermore, this vulnerability was not a bug (like a buffer overflow), but a combination of intentionally designed features when put together can be exploited. This went unnoticed for many years.
- Since the vulnerability was disclosed, we had a number of inquiries from people trying to determine out if they were affected by this vulnerability. It turns out many were running log4j version 1, which was discontinued in 2015. The good news is that this particular vulnerability does not affect them. The bad news is that the code they are running has not benefited from any security fixes in many years.
- The handling of the issues in log4j followed ASF processes and the fixes were timely.
Recommendations
These experiences inform our positions on a number of items.
- We support the notion of identifying critical components. As our software is without cost and can be downloaded by anybody, we have no way of knowing how widely our projects are used. This means that by necessity determining if a component is critical can only either be determined by automated means which is always going to be incomplete and imperfect, or more manually such as by identifying critical products first, and then getting an inventory of what components those products embed. This problem is not unique to the ASF, so it makes sense for there to be industry standards and best practices. We are prepared to participate in these activities.
The owner of a system is best placed to determine the criticality of that system and the degree to which individual components contribute to that criticality.
Criticality depends on usage. We need to avoid solutions that create burdens for all system owners using a software component when the component is only critical for some systems.
- We eagerly welcome audits and fixes from any source. We have a process defined for doing so: https://www.apache.org/security/. Our one caution is that we have prior experiences with audits and it is not productive when those audits produce a number of false positives. See https://s.apache.org/fhoji. It is worth noting that many automated means of performing an audit are more suited to detecting bugs inside a single component rather than identifying a vulnerability that is composed of combination of intentional features such as the one we saw with log4j. It is better when these audits are performed by skilled resources who are able to accurately identify actual issues, produce fixes, and therefore reduce the burden on each project.
- It is particularly frustrating to us that we can produces fixes for critical issues in a matter of days or weeks and not to have those fixes be picked up for months or years, despite having a common and industry standard process for disclosing vulnerabilities and fixes. This is the problem that we see as most pressing.
- "We also find that a typical zero-day attack lasts 312 days on average and that, after vulnerabilities are disclosed publicly, the volume of attacks exploiting them increases by up to 5 orders of magnitude." – https://dl.acm.org/doi/10.1145/2382196.2382284
- This is a really hard problem to solve. Like with criticality, whether a system is exposed to a particular vulnerability depends on usage. We don't want to introduce something that impacts all systems when only some are vulnerable.
- Adding automatic update checks could work for products but not libraries. Even then there are plenty of system owners (including most of the US Federal Government) who don't want their software phoning home for any reason.
- Logging regular warnings once the software reaches a certain age requires assumptions to be made in terms of expose to vulnerabilities that are likely to be wrong for a reasonable proportion of systems. And, again, is only really workable for products.
- The most viable approach is likely to be to incentivize system owners to ensure that they are not exposed to vulnerabilities via using a product within their system:
- with known vulnerabilities;
- that has no established process for handling vulnerability reports and publishing details of confirmed vulnerabilities; and/or
- has reached End of Life without effective mitigation for the associated security risks.
The incentive has to be some form of liability (civil and/or criminal) if a system owner does any of the above. Such liability may already exist, for example in the FTC. Expanding the funding of the organizations tasked with enforcement in one possible option
Further reading