Security Operations

Log4Shell Response and Mitigation Recommendations

Written by
December 17, 2021
Security Operations featured Log4J Log4shell MTR Sophos Managed Threat Response (MTR) Sophos MTR

Last updated 2021-12-20 UTC 20:06

Update: Added new open source scanning tool, adjusted open sockets query

Update: Added information on log4j 2.17.0, adjusted scheduled query to enumerate more versions of log4j

Summary and Background

Log4j is an open-source logging framework developed by the Apache Foundation which is incorporated into many Java-based applications on both servers and end-user systems. Initially released, on December 9, 2021, Log4Shell (the nickname given to this vulnerability) is a pervasive and widespread issue due to the integrated nature of Log4j in many applications and dependencies. It’s classified as an unauthenticated remote code execution vulnerability and listed under CVE-2021-44228

Proof of concept code and weaponization became available almost immediately following publication.

From NIST:

Apache Log4j2 2.0-beta9 through 2.12.1 and 2.13.0 through 2.15.0 JNDI features used in configuration, log messages, and parameters do not protect against attacker-controlled LDAP and other JNDI related endpoints. An attacker who can control log messages or log message parameters can execute arbitrary code loaded from LDAP servers when message lookup substitution is enabled. From Log4j 2.15.0, this behavior has been disabled by default. From version 2.16.0, this functionality has been completely removed. Note that this vulnerability is specific to Log4j-core and does not affect log4net, log4cxx, or other Apache Logging Services projects.

UPDATE: On December 18th, 2021, a denial-of-service vulnerability (CVE-2021-45105) affecting Log4j versions from 2.0-beta9 to 2.16.0 (Fixed in version 2.17.0) was discovered.

For a more detailed breakdown on this vulnerability, please see the Sophos News article: Inside the code: How the Log4Shell exploit works

Mitigation Guidance

Identification

The primary thing an organization should be focusing on is to determine if and where Log4j exists within their estate. Even more specifically, focusing on externally facing applications which may use Log4j. This is a non-trivial task given that this is a library, rather than an installed application, and it can be integrated with vendors and third-party tools.

For organizations who have robust asset inventories, these should be used to identify where Java, the JVM (Java Virtual Machine), and the Log4j libraries exist within their inventory.

There are currently several resources being maintained which document known vulnerable application and vendors. This is a great first place to start. Review these lists and determine if the software is present in your estate. If it is, ensure that you have applied the latest patches.

Another method of identification would be to perform internal/external vulnerability scans, many of which now provide signatures or plugins for identifying vulnerable Log4j instances.

Sophos XDR can be used to help identify devices with possible vulnerable Log4j JAR files; the following Live Discover query can be used. Given Log4j can be embedded in other applications and dependencies, this query serves as a starting point to help organizations identify where there are instances of Java running in their estate.

It is suggested to begin with running this query on servers to identify possible usage of Java in the estate. These queries will work across operating systems.

SELECT DISTINCT name, cwd, cmdline
FROM processes
WHERE (cmdline LIKE '%Log4j%' OR cmdline LIKE '%java%')
AND LOWER(cwd) NOT IN ('c:\windows\', 'c:\windows\system32\', '/')

Extending on the query above, we can subsequently enumerate possibly vulnerable versions of the Log4j library with the query below. This query will recursively look in the working directory of the running Java processes to identify files with the name `Log4j%.jar`

Note: This query will miss instances where Log4j is bundled into other third-party JAR files as well as non-standard named JAR files. But it can provide a good initial triage of possible exposure.

WITH java_processes AS (
SELECT DISTINCT name, cwd, cmdline
FROM processes
WHERE (LOWER(cmdline) LIKE '%log4j%' OR LOWER(cmdline) LIKE '%java%')
AND LOWER(cwd) NOT IN ('c:\windows\', 'c:\windows\system32\', '/')
)

SELECT
    p.name,
    p.cwd,
    p.cmdline,
    f.path,
    f.directory,
    f.filename,
CASE
    WHEN f.filename LIKE 'log4j-1%' THEN 'log4j 1.X is no longer maintained supported, but is not vulnerable to CVE-2021-44228'
    WHEN f.filename LIKE 'log4j-%-2.17.%.jar 'THEN' Patched against CVE-2021-44228 #log4shell, CVE-2021-45046, and CVE-2021-45105 and on the latest version'
    WHEN f.filename LIKE 'log4j-%-2.16.%.jar 'THEN 'Patched against CVE-2021-44228 #log4shell and CVE-2021-45046, but susceptible to CVE-2021-45105. Upgrade to log4j 2.17.0'
    WHEN f.filename LIKE 'log4j-%-2.15.%.jar 'THEN 'Patched against CVE-2021-44228 #log4shell, but susceptible to CVE-2021-45046. Upgrade to log4j 2.17.0'
    WHEN f.filename LIKE 'log4j-%-2.14.%.jar 'THEN 'Not Patched against CVE-2021-44228 #log4shell and susceptible to CVE-2021-45046. Upgrade to log4j 2.17.0'
    WHEN f.filename LIKE 'log4j-%-2.13.%.jar 'THEN 'Not Patched against CVE-2021-44228 #log4shell and susceptible to CVE-2021-45046. Upgrade to log4j 2.17.0'
    WHEN f.filename LIKE 'log4j-%-2.12.%.jar 'THEN 'Not Patched against CVE-2021-44228 #log4shell and susceptible to CVE-2021-45046. Upgrade to log4j 2.17.0'
    WHEN f.filename LIKE 'log4j-%-2.11.%.jar 'THEN 'Not Patched against CVE-2021-44228 #log4shell and susceptible to CVE-2021-45046. Upgrade to log4j 2.17.0'
    WHEN f.filename LIKE 'log4j-%-2.10.%.jar 'THEN 'Not Patched against CVE-2021-44228 #log4shell and susceptible to CVE-2021-45046. Upgrade to log4j 2.17.0'
    WHEN f.filename LIKE 'log4j-%-2.9.%.jar' THEN 'Not Patched against CVE-2021-44228 #log4shell and susceptible to CVE-2021-45046. Upgrade to log4j 2.17.0'
    WHEN f.filename LIKE 'log4j-%-2.8.%.jar' THEN 'Not Patched against CVE-2021-44228 #log4shell and susceptible to CVE-2021-45046. Upgrade to log4j 2.17.0'
    WHEN f.filename LIKE 'log4j-%-2.7.%.jar' THEN 'Not Patched against CVE-2021-44228 #log4shell and susceptible to CVE-2021-45046. Upgrade to log4j 2.17.0'
    WHEN f.filename LIKE 'log4j-%-2.6.%.jar' THEN 'Not Patched against CVE-2021-44228 #log4shell and susceptible to CVE-2021-45046. Upgrade to log4j 2.17.0'
    WHEN f.filename LIKE 'log4j-%-2.5.%.jar' THEN 'Not Patched against CVE-2021-44228 #log4shell and susceptible to CVE-2021-45046. Upgrade to log4j 2.17.0'
    WHEN f.filename LIKE 'log4j-%-2.4.%.jar' THEN 'Not Patched against CVE-2021-44228 #log4shell and susceptible to CVE-2021-45046. Upgrade to log4j 2.17.0'
    WHEN f.filename LIKE' log4j-%-2.3.%.jar' THEN 'Not Patched against CVE-2021-44228 #log4shell and susceptible to CVE-2021-45046. Upgrade to log4j 2.17.0'
    WHEN f.filename LIKE' log4j-%-2.2.%.jar' THEN 'Not Patched against CVE-2021-44228 #log4shell and susceptible to CVE-2021-45046. Upgrade to log4j 2.17.0'
    WHEN f.filename LIKE' log4j-%-2.1.%.jar' THEN 'Not Patched against CVE-2021-44228 #log4shell and susceptible to CVE-2021-45046. Upgrade to log4j 2.17.0'
    ELSE' Unknown version or unrecognized filename, need to investigate manually'
END AS patch_status
FROM
    java_processes p
JOIN
    file f
WHERE
    f.path LIKE p.cwd || '%%'
    AND f.filename LIKE 'log4j%.jar'

To solve the challenge of nested JAR files, a local scanner can be used to enumerate vulnerable Log4j libraries which exist on the filesystem. Several open-source projects have been released which serve this purpose. For all systems identified above that have a running Java process, a more in-depth scan can be conducted using the following open source tools.

Update Vulnerable Versions

The most thorough mitigation will be updating and patching the vulnerable Log4j libraries discovered, but understandably this is not possible in all scenarios. As of writing this writing, version 2.16.0 is the most up to date version, and more information regarding fixes for the latest Log4j vulnerabilities can be found at: https://logging.apache.org/Log4j/2.x/security.html.

On December 14, 2021 a second CVE (CVE-2021-45046) was discovered in the Log4j library which made the 2.15.0 version ineffective in some default configurations. To address both CVE-2021-44228 and CVE-2021-45046 it is recommended to update to Log4j version 2.16.0. In instances where updating is not possible, please see the Other Mitigations section below.

UPDATE: The current list of known vulnerabilities and suggested fixes are listed below:

  • CVE-2021-44228 (CVSS score: 10.0) – A remote code execution vulnerability affecting Log4j versions from 2.0-beta9 to 2.14.1 (Fixed in version 2.15.0)
  • CVE-2021-45046 (CVSS score: 9.0) – An information leak and remote code execution vulnerability affecting Log4j versions from 2.0-beta9 to 2.15.0, excluding 2.12.2 (Fixed in version 2.16.0)
  • CVE-2021-45105 (CVSS score: 7.5) – A denial-of-service vulnerability affecting Log4j versions from 2.0-beta9 to 2.16.0 (Fixed in version 2.17.0)

It is suggested to follow the guidance of Apache, and upgrade to log4j version 2.17.0

Other Mitigations

In instances where updating or identification of the Log4j library is not possible or just as an extra precaution, consider implementing the following:

  1. This attack requires that the affected host make an initial outbound connection to the adversary’s malicious LDAP server to load the malicious Java class. Blocking the standard ports for LDAP, LDAPS, and RMI (389, 636, 1099, 1389) while not exhaustive because the port can be randomized, can reduce the attack surface.
  2. IPS and WAF, while shown to be non-comprehensive due to the various evasion techniques which are possible in this attack, should still be implemented to assist with detection and response
  3. If the vulnerable version of Log4j cannot be upgraded, in any release other than 2.16.0, you may remove the JndiLookup Class from the classpath:
zip -q -d Log4j-core-*.jar org/apache/logging/Log4j/core/lookup/JndiLookup.class

Note: The previously recommended mitigations by configuring the “Log4j2.formatMsgNoLookups” property or setting the environmental variable “LOG4J_FORMAT_MSG_NO_LOOKUPS” to True, are no longer recommended, as they have been proven ineffective.

Monitoring and Threat Hunting

With the challenges in fully enumerating the log4j usage in an estate, the below Data Lake queries can be used to hunt for suspicious processes and network connections.

Java Spawning Suspicious Processes

The below Data Lake queries will enumerate suspicious child processes of Java that should be investigated. These could be an indication that a vulnerable log4 instance was successfully exploited. Any suspicious commands should be investigated further.

Windows

SELECT
    meta_hostname,
    parent_path,
    parent_sophos_pid,
    path,
    sophos_pid,
    cmdline
FROM xdr_data
WHERE query_name = 'running_processes_windows_sophos'
AND LOWER(parent_name) = 'java.exe'
AND name IN (
    'cmd.exe',
    'powershell.exe',
    'rundll32.exe',
    'whoami.exe',
    'certutil.exe',
    'ipconfig.exe',
    'net.exe'
)

Linux

Note: Because we have to reconstruct the parent / child relationship, there can be some inconsistencies in results due to process ID reuse.

SELECT DISTINCT
   X3.meta_hostname AS ep_name, 
   X3.meta_os_platform,
   replace(replace(CAST(from_unixtime(X3.time) AS VARCHAR),'.000','Z'),' ','T') Date_Time, -- add the T to help excel understand this is a date and time
   X3.name, 
   X3.cmdline, 
   X3.path, 
   S1.Single_PID||':'||CAST(X3.time AS VARCHAR) AS sophos_pid,
   S1.Single_PID pid,
   X4.name Child_Name,
   X4.cmdline Child_cmdline,
   X4.path Child_Path,
   -- WE DID NOT unnest the PIDS so are only showing FIRST PID
   SPLIT_PART(X4.pids,',',1)||':'||CAST(X4.time AS VARCHAR) AS Child_Sophos_PID,
   SPLIT_PART(X4.pids,',',1) Child_PID,
   P1.Single_Parent||':'||CAST(X4.time AS VARCHAR) AS Child_Parent_Sophos_PID,
   P1.Single_Parent Child_Parent_PID,
   X3.query_name
FROM xdr_data X3 CROSS JOIN UNNEST(SPLIT(X3.pids,',')) AS S1 (Single_PID)
LEFT JOIN xdr_data X4 CROSS JOIN UNNEST(SPLIT(X4.parents,',')) AS P1 (Single_Parent) ON P1.Single_Parent = S1.Single_PID 
   AND X4.query_name = 'running_processes_linux_events'
   AND X3.meta_boot_time = X4.meta_boot_time  -- Try and avoid PID collisions
   AND X4.time <= X3.time
   AND LOWER(X4.name) IN ('sh', 'bash', 'dash', 'ksh', 'zsh', 'curl', 'wget')
WHERE X3.query_name = 'running_processes_linux_events'
   AND LOWER(X3.name) = 'java'

Unexpected Outbound Network Traffic

This query can help identify abnormal traffic from Java processes over LDAP, RMI, and DNS standard ports (389, 636, 1099, 1389, 53, 5353). It’s looking for traffic with a destination to a non RFC1918 address over the protocol’s standard port. This can also help identify devices where there is a vulnerable version of Log4j reaching out to an adversary’s LDAP server. This can and will also identify benign traffic; therefore, the results will need validated.

Windows / Linux / OSX

SELECT
    meta_hostname,
    name,
    cmdline,
    pid,
    parent,
    path,
    remote_address,
    remote_port,
    local_address
FROM
    xdr_data
WHERE
    query_name = 'open_sockets'
    AND (LOWER(name) = 'java.exe' OR LOWER(name) = 'java')
    AND remote_port IN (389, 636, 1099, 1389, 53, 5353)
    AND remote_address NOT LIKE '10.%'
    AND remote_address NOT LIKE '192.168.%.%'
    AND remote_address NOT LIKE '172.16.%.%'
    AND remote_address NOT LIKE '172.17.%.%'
    AND remote_address NOT LIKE '172.18.%.%'
    AND remote_address NOT LIKE '172.19.%.%'
    AND remote_address NOT LIKE '172.20.%.%'
    AND remote_address NOT LIKE '172.21.%.%'
    AND remote_address NOT LIKE '172.22.%.%'
    AND remote_address NOT LIKE '172.23.%.%'
    AND remote_address NOT LIKE '172.24.%.%'
    AND remote_address NOT LIKE '172.25.%.%'
    AND remote_address NOT LIKE '172.26.%.%'
    AND remote_address NOT LIKE '172.27.%.%'
    AND remote_address NOT LIKE '172.28.%.%'
    AND remote_address NOT LIKE '172.29.%.%'
    AND remote_address NOT LIKE '172.30.%.%'
    AND remote_address NOT LIKE '172.31.%.%'
    --Exclude localhost
    AND remote_address NOT LIKE '127.%.%.%'
    --Exlude self-signed IP Addresses
    AND remote_address NOT LIKE '169.254.%.%'
    AND remote_address NOT LIKE '-'
    AND remote_address NOT LIKE '::1'
    AND remote_address NOT LIKE '%::%'

Contributions to this article were made by members of the Managed Threat Response team (Tyler Wojcik)

Resources

Associated CVEs

Apache Logging Security

Sophos Central Live Discover

Author photo of Mat Gangwer
About the Author

VP of Managed Detection and Response (MDR) at Sophos. Focused on efficient and effective security operations, threat hunting, and adversary countermeasures.

Read Similar Articles

Leave a Reply

Your email address will not be published. Required fields are marked *