On February 8, the Senate Homeland Security and Government Affairs Committee held a full hearing to address the Log4j software vulnerability made public in November of 2021 and to better understand the status of remediation and the ongoing impact. The news was not heartening.
Hearing attendees included David Nalley, President of Apache Software Foundation which developed Log4j; Brad Arkin, Senior Vice President and Chief Security Officer of Cisco Systems; and Jen Miller-Osborn from Palo Alto Networks. The sobering testimony offered by these expert witnesses made it clear that there is no easy path to fully address the Log4j vulnerability in part because Log4j is so pervasive – found in hundreds of millions devices worldwide – and because the Log4j flaw can be used by the most unsophisticated of hackers to control devices. The testimony laid bare the benefits and vulnerability of open-source software, which both is foundational to the world’s digital infrastructure but also creates opportunities for exploiting security flaws on a global scale.
David Nalley, President of Apache Software Foundation, indicated in his testimony that, “Given the near ubiquity of Log4j’s use, it may be months or even years before all deployed instances of this vulnerability are eliminated.” Log4j was released by Apache in 2001. As its footprint grew, so did its functionalities. According to Nalley, it was a 2013 addition to Log4j, along with a part of the Java programming environment, that combined in such a way that exposed this security flaw. He recommended that every stakeholder in the software industry – including its largest customers, like the federal government – should be investing in software supply chain security and supported legislative efforts to address vulnerabilities. However, he cautioned that, “While ideas like the Software Bills of Materials won’t prevent vulnerabilities, they can mitigate the impact by accelerating the identification of potentially vulnerable software. However, the ability to quickly update to the most secure and up-to-date versions remains a significant hurdle for the software industry.”
Indeed, while the use of open-source software to develop common solutions to common problems is a highly efficient programming model, remediation of flaws such as Log4j, once in use and embedded in the infrastructure of multiple organizations, is a logistical nightmare. Virtually every major technology company world-wide was forced to assess their software applications to identify vulnerable versions of Log4j in use.
Even global organizations such as Cisco Systems with its massive resources can struggle to address these issues. As stated in his testimony to the Committee, Brad Arkin recounted an earlier similar vulnerability in 2014 which took Cisco 50 days to identify all software that required updates to remediate the vulnerability and several additional weeks to apply the necessary software patches. With Log4j, Cisco was able to identify the use of the Log4j library within its products and provide software patches for affected products within 10 days. Arkin attributes the greatly reduced response time to several factors, including lessons learned in 2014, building better and more secure software, and having data about which specific applications and software we use. As he explained to the Committee,
“We were able to move quicker, eliminate risk faster, and have the agility needed to manage our own security and resilience. For Cisco, the key differentiator was our improved visibility into the software applications and thirdparty products that we use as a company. Additionally, we now use tooling to allow us to see the software maintenance status of the applications we use, identifying whether a particular piece of software is the latest version. We strongly recommend the use of tools and technology that allow companies and government agencies to have this kind of visibility into the applications they employ and their maintenance status.”
Unfortunately, not all companies have taken the steps necessary to remediate the vulnerabilities created by Log4j. In fact, as noted by Sonatype (which owns the largest repository of Java packages) in a recent article, more than 35% of users continue to download vulnerable versions of Apache Log4j, despite Apache’s concerted efforts to offer remediated software. Why would companies do this? The Chief Technology Officer of Sonatype suggested that “many companies don’t actually know what is in their applications so instead of launching targeted updates they are doing busy work. In short, they are still trying to build their inventory before they start to react.”
The danger of this approach was specifically addressed in the testimony of Jen Miller-Osborn from Palo Alto Networks:
“I cannot stress enough the foundational importance of accurately understanding the size of your internet-exposed attack surface. If you do not understand the totality of your digital footprint through the eyes of the adversary, your security baseline is inherently incomplete. The ability to rapidly discover and remediate new vulnerabilities – Log4Shell or otherwise – is going to be predicated on production level mapping of your networks. This is true for organizations large and small, public and private.”
As noted in the testimony of Jen Miller-Osborn, the Log4j flaw has been a very fruitful tool for attackers. “We have subsequently seen exploitation for coin mining (to commandeer computing horsepower from the victim to perform cryptocurrency mining), for hijacking victim networks as part of larger botnet systems (defined as network of computers controlled as a group without the owners’ knowledge), and to infect systems with ransomware.” Threat researchers have also reported that a number of foreign governments, including North Korea, China, Iran and Turkey, are exploiting Log4j for more sophisticated attacks. By mid-December, attackers made more than 1.8 million attempts to exploit the Log4j flaw, according to Check Point Software, targeting almost half of the corporate networks that Check Point Software tracks worldwide.
Conclusion
Notwithstanding the somber testimony offered about the challenges of open-source software and addressing both the Log4j vulnerability as well as similar situations that will arise in the future, the industry experts also offered some solid recommendations to the Committee.
Brad Arkin of Cisco stressed that to address future vulnerabilities, the following improvements are needed in both the private and public sectors: 1) baselines for software security, including open-source software; 2) speed and efficiency at finding and fixing problems when they arise; and 3) resilience against attacks, particularly in the window between identification of a vulnerability and application of a fix or mitigation.
Jen Miller-Osborn’s testimony also offered the Committee tangible steps to immediately reduce risks in response to Log4j and future vulnerabilities:
- Enumerate an accurate denominator of your digital infrastructure. This should be a foundational aspect of any national-level incident response and is applicable across federal and non-federal entities. You can’t protect what you can’t see.
- Look for ways to automate compliance with vulnerability management policies.
- Drive industry-wide commitment to Development Security Operations (DevSecOps). Impressive work is already being done in this arena, but the community would be well-served by increasing adoption of existing development tools to control access to open-source components. These tools can scan all of the open-source packages for both integrity and security before they are approved and allowed for engineering teams to use in products.
- Promote common visibility and automated security capabilities across your entire environment. Data from cloud, endpoint, and on-premise systems should be seamlessly integrated.
- Lastly, cyber hygiene basics still remain as important as ever.
The ongoing impact of Apache’s Log4j flaw was starkly presented to the Committee and the subtext of all the testimony was clear – there is much work to be done both by legislators and the business community to mitigate, if not eliminate, the impact of future threats.
NOTE: An earlier article on Log4j is available for a deep dive into the vulnerability and how it was initially addressed. In summary, Apache’s Log4J library is used in the development of software when logging is needed on a variety of systems. This library intentionally included the ability to remotely fetch executable code from a remote server and then execute it. However, this design also allowed attackers to force a vulnerable system to use this functionality without the knowledge of its owner and, thereby, download and deliver what’s known as the “payload” – the part of the malware that is crafted for a specific malicious purpose.
