Addressing the Log4j Vulnerability with Verve

Addressing Log4j Vulnerability

Over the past week, the cyber security community has been chasing remediation measures for the Log4j vulnerability present in the libraries of many applications. This note is not intended to parse the details of this vulnerability and its potential impact on the huge array of systems impacted. There are many very good articles, including those by CISA and others that do a great job of that. For more information we recommend the following articles:

• The Log4j project security page
• CISA guidance
• Veracode’s technical analysis and remediation guidance

Our focus is how our customers and other industrial organizations similarly situated might address the challenges most effectively.

First, a quick note on the challenge of this particular vulnerability. Log4j exists within a library that other applications use for a variety of tasks. Most end-users – and in fact many software companies themselves – are unaware that the applications utilize or require this library, or at least the vulnerable version of it.

Second, the library cannot be updated without updating the application. Third, identifying where the library has significant challenges due to the depth and breadth one must search to find it.

There are many methods that have been offered to protect organizations from the threat posed by Log4j, the CISA guidance being a very good source.

However, to truly protect the endpoint, the organization must find these libraries and patch or update them in most cases by updating the application.  Many vendors have issued alerts to their customers and publicly announced their applications are vulnerable, offering customers an initial list to search. But many vendors are uncertain if the vulnerability will impact their products, so the list is still very incomplete.

Efforts in IT to find this have included adding the publicly announced software vulnerabilities into the databases of common vulnerability assessment products, trying to monitor traffic to find active exploits or presence of the library, and through scanning devices to identify the presence of the library on that system.

The first approach is a great start but has significant gaps and potential errors depending on the level of automation desired. The second helps prioritize where to go deeper and could identify an active exploit but may occur after the horse has left the barn, so to speak.

The third approach via scanning creates significant stress and system impact:

• Verve observed a common scanner taking 45 minutes, 400 MB RAM, and an entire CPU core to scan a single system.
• It is risky in an enterprise data center with network-mounted SCSI drives, CPU, and disk oversubscription.
• It consumes these resources on devices whether or not they even run Java.
• Many of these are not cross-platform, requiring separate scanners for Windows, Linux, etc.
• Even at the end of all this, they may miss applications on attached drives, network drives, etc.

In the environments in which Verve and its clients operate – critical infrastructure, industrial controls, etc. – these scanning technologies would cause significant operational challenges and there is no way an organization could run the common scanners on all devices given the length of time and system impact.  Therefore, we have taken a different approach.

The approach leverages the 360-degree risk view that is part of the Verve platform. Using our agent and agentless capabilities as well as our real-time monitoring of network AND endpoint behavior, we narrow down the targets, reduce the system impact and sequentially work through this challenge.

The approach, in short, is as follows:

• We developed a very OT-sensitive “agent-based discovery” approach that doesn’t utilize significant CPU or RAM and runs in a matter of a few seconds rather than 45 minutes. This has been shared with the community.
• This “agent-detection” approach allows the identification of devices you KNOW are vulnerable. While this is not a comprehensive list, it initially eliminates the false positives and begins to make traction. Importantly, this is cross-platform, so you only need one detector across all device-types
• Simultaneously, our dashboards are updated with the latest list from CISA on all the publicly announced vulnerable applications which we use to help customers identify where those applications exist in their environment. This list is being updated daily.
• We use the Verve platform to triage these devices through a range of remediating actions from patching to managing ports and services.
• We leverage our historical endpoint behavior data captured in our OT Threat Detection component to look at process creation and metrics to identify machines with transient Java services running. This provides us with the next batch of target devices.
• The threat detection capability allows monitoring of IOCs and attacks. IOCs and attacks pivot over time. Simple rules alone aren’t enough to keep up with the threat landscape. Verve’s Threat Feed is constantly updating with the latest available information about emerging threats and automatically detects when a host was found communicating with a known, malicious server.
• Finally, we leverage the rest of our threat detections built into the platform to identify post-exploitation events i.e., using log4shell to install and execute other malicious software.
• Once the initial triage is complete, the user can go back to the Internet-facing devices and begin further investigation.

The industry needs lots of ideas to address this significant risk. This is our contribution to the effort. We look forward to continuing to work with others to push the boundaries on what is feasible, especially in sensitive OT environments.