Log4j / Log4Shell Vulnerabilities

CVE-2021-45105 | CVE-2021-45046 | CVE-2021-44228 | CVE-2021-42550

Google Translate | en English
| Be aware that Google Translate may change command syntax, as well

1. Executive Summary

Updated: December 30, 2021 @ 3:31 AM CT

The discovery of a new type of attack has opened the door for critical exploits from bad actors and a rush of fixes from Open Source developers to prevent these types of attacks. This attack takes advantage of a series of features in enterprise logging software to call a remote server for sharing configuration across servers. When this feature is instructed to call an LDAP Server, bad actors are able to take advantage of LDAP’s ability to store a specific type of Java data and deliver an exploit to the source system.

After the initial exploit was remediated, several other attacks were uncovered by security researchers in the widely used Log4j2 and Logback software libraries. These exploits impact Java-based software which is widely used in the enterprise and Internet technology companies.

1.1 What to expect

When any new attack vector is discovered, additional exploits will be discovered across other applications and open source frameworks. Technical leaders should prepare for more exploits in the coming months.

1.2 Course of action

Technical leaders should ensure they have visibility to the software running in their environment and the ability to verify the version number of libraries and components. As new exploits are reported, technical teams are then well positioned to know which systems need to be upgraded and to immediately begin upgrading to remediate the security risk.

2. Background

A zero-day vulnerability was disclosed in the world’s most widely adopted logging framework, Log4j. Within 24 hours of the disclosure, global organizations were reporting exploits. Apache has identified the issue and published a fix.

This is a commonly used package and the impact of the vulnerability has been covered by many on the Internet. HYTE Support would like to provide some resources regarding this vulnerability in the event that you haven’t seen this information yet.

The issue is readily mitigated from exploitation with a simple configuration property. HYTE is working with Open Source communities to release updates that include the patch moving forward.

The Apache Log4j project has released a new version (v2.17.0) which requires that users explicitly enable the problematic lookups. This version will be used in the HYTE releases moving forward starting with v5.2.30 and v5.4.0.

Table 1. CVE Matrix

CVE Number Component Severity Score
CVE-2021-42550 LogBack High 8.5
CVE-2021-44228 Log4j2 Critical 10.0
CVE-2021-44832 Log4j2 Moderate 6.6
CVE-2021-45046 Log4j2 Critical 9.0
CVE-2021-45105 Log4j2 Moderate 5.9

3. Exposure

Log4j2 is massively popular and widely used. Virtually every enterprise and Internet technology company will be impacted directly or indirectly.

This popularity is seen as a massive opportunity for bad actors. Reports from Sonatype, (which operates the most popular Java library repository in the world-- Maven Central) have reported the Log4j2 was downloaded over 28 million times in the last 4 months and is used by almost 7,000 open source projects-- including the helicopter launched by the Mars rover (Ingenuity).

The feature that is being exploited is also very popular. Almost 800 other projects borrow the problematic JndiManager class from Log4j2 for use in their system.

4. Exploits

Unsavory actors are actively exploiting this vulnerability with identified hosts in Russia and other hosts running on public Internet addresses.

4.1 Blocking public Internet access

Environments that do not permit their enterprise systems from accessing public Internet are at reduced risk for immediate exploitation.

4.2 Next wave of attacks

Computer security experts are alerting that the next wave of attacks will include chaining other vulnerabilities to exploit environments, including ransomware attacks.

Blocking public Internet access alone will not be sufficient in preventing future attacks.

Fixes need to be applied in order to mitigate attacks where the Log4Shell vulnerability is combined with other security exploits to compromise systems

5. Resolution

The fundamental nature of the logging software being the affected party means that it will require some time to ensure proper compatibility and stability of all layers throughout HYTE products.

HYTE will be providing customers with resolution in three phases

Phase 1 - Mitigation solution available immediately

Customers are encouraged to apply the appropriate mitigation solution to all instances in all environments.

Phase 2 - Mitigation solution configured by default

HYTE Platform v5.2.28 will include the mitigation flag enabled by default. Customers may choose to upgrade their environments to this version.

Phase 3 - Permanent patches applied in all releases

HYTE Platform v5.2.30 and upcoming v5.4.x releases will include the permanent patches from upstream Open Source communities.

6. Mitigation Technical Details

The Apache Log4j2 community responded quickly with updates to address the vulnerabilities. After additional exploits were discovered, the problematic feature was removed entirely in v2.16.0.

6.1 Apache Log4j2 mitigation matrix

Table 2. Apache Log4j2 version Mitigation

Log4j Version Mitigation
2.17.1 Fourth release, addresses moderate risk flaw
2.17.0 Third release, fixes Critical and High risk flaws
2.16.0 Second release in response to Log4Shell, improvements
2.15.0 First release in response to Log4Shell, Option 1 set by default
>= 2.10 Option 3
2.7 =< version used =< 2.14.1 Option 3
2.0-beta9 =< version used =< 2.10.0 Option 3

6.2 Apache ActiveMQ mitigation matrix

Apache ActiveMQ distributions are not impacted by the Log4Shell vulnerability, as all the current releases use Log4j v1.x

The upcoming Apache ActiveMQ 5.17.0 release includes an upgrade to Log4j2, and the upstream source tree has been patched to the fixed version of Log4j2 (v2.17.1).

Table 3. Apache ActiveMQ version Mitigation

Apache ActiveMQ Version Mitigation
5.17.0 (unreleased) Fixed. Upstream source patched to Log4j2 2.17.1
5.x (all versions) Not vulnerable

6.3 HYTE Product mitigation matrix

HYTE products that use the HYTE Runtime are powered by Apache Karaf, which in turn utilizes Log4j2 for features and performance benefits. HYTE does not use LogBack.

  • HYTE Console
  • HYTE MQ
  • HYTE Runtime

This table provides HYTE Customers with a quick reference to identify which fix is required, based on the version used.

Table 4. HYTE Product Matrix

HYTE Product Version HYTE OSS Version Log4j Version Mitigation
5.2.28 4.2.11.hyte-42113 2.14.0 Option 2 + 3 (enabled by default)
5.2.x 4.2.11.hyte+ 2.14.0 Option 2 + 3
5.0.10 4.2.10.hyte-42101 2.13.3 Option 2 + 3
N/A 4.2.5.hyte-4200+ 2.8.2 Option 2 + 3
N/A 4.1.5.hyte-4005+ 2.8.2 Option 2 + 3
N/A Older Varies Option 2 + 3

7. Mitigation Options

Table 5. Mitigations

Mitigation Approach Effective Notes
Option 1 Disable lookup feature No Not effective
Option 2 Disable lookups in log pattern No Must be combined with Option 3
Option 3 Remove problematic class Yes
Option 4 Upgrade to Log4j2 2.17.0 Yes

7.1 Option 1: Disable message format lookups

Option 1 is ineffective

7.2 Option 2: Update logging message format

Must be combined with Option 3

This change disables message format lookups for individual log configurations.

Configuration macro:

{nolookups}

The vulnerability is mitigated from exploitation by updating logging configuration file:

etc/org.ops4j.pax.logging.cfg

Before:

log4j2.pattern = %d{ISO8601} | %-5p | %-16t | %-32c{1} | %X{bundle.id} - %X{bundle.name} - %X{bundle.version} | %m%n

After

log4j2.pattern = %d{ISO8601} | %-5p | %-16t | %-32c{1} | %X{bundle.id} - %X{bundle.name} - %X{bundle.version} | %m{nolookups}%n

This needs to be changed for every pattern used

7.3 Option 3: Remove JndiLoookup class

This removes the class from the installation to prevent exploitation.

This does not effect logging moving forward, as Log4j2 quietly skips lookups when this class is missing.

Steps:

1. Identify the Log4j bundle id
karaf@root()> list -t 0 | grep -i log4j
 7 │ Active   │   8 │ 1.11.9			│ OPS4J Pax Logging - Log4Jv2
implementation, Fragments: 14
14 │ Resolved │  12 │ 1.11.9			│ OPS4J Pax Logging - Log4j v2 Extra
packages, Hosts: 7
karaf@root()>

The log4j2 bundle is '7' in this example

2. Confirm the JndiLookup class exists

karaf@root()> classes 7 | grep -i  jndilookup
org/apache/logging/log4j/core/lookup/JndiLookup.class | exported: false
karaf@root()>

3. Stop the process

./bin/stop

4. Remove the class from the system/ folder

$ zip -q -d system/org/ops4j/pax/logging/pax-logging-log4j2/1.11.9/pax-logging-log4j2-1.11.9.jar org/apache/logging/log4j/core/lookup/JndiLookup.class

5. Remove the class from the deployed bundle in the data/cache folder

$ zip -q -d data/cache/bundle7/version0.0/bundle.jar org/apache/logging/log4j/core/lookup/JndiLookup.class

6. Start the process

./bin/start

7. Confirm the class is removed

karaf@root()> classes 7 | grep -i jndilookup
karaf@root()>

7.4 Option 4: Upgrade Log4j2 to 2.17.1

As of 2021-12-21, the Log4j2 2.17.0 release fixes all known security issues.

HYTE is planning on releasing two updates with fixes built-in. This approach provides a buffer from the large changeset introduced in Log4j2 2.16.0 and 2.17.0, (which was understandably rushed to release).

Table 6. HYTE Release Matrix

HYTE Product Version HYTE OSS Version Log4j Version Mitigation
5.2.28 4.2.11.hyte-42113e 2.14.0 Option 2 + 3 (enabled by default)
5.2.30 4.2.14.hyte-42140 2.17.0 Option 4
5.4.0 4.3.5.hyte-43050 2.17.0 Option 4


8. Ineffective Options

HYTE performed internal testing to validate and recommend mitigation options. The following options have been proposed by various parties across various forums on the Internet. These options have not been able to be proven by HYTE Engineering, or incur significant security risk.

HYTE recommends customers do NOT rely on any of the following options

8.1 Option 1: Disable message format lookups

Option 1 mitigation is no longer sufficient

This change disables message format lookups for the entire process.

The vulnerability is mitigated from exploitation by setting a Java system property at boot time:

-Dlog4j2.formatMsgNoLookups=true
Steps:

1. Update the bin/setenv file:

export LOG4SHELL_OPTS="-Dlog4j2.formatMsgNoLookups=true"
export EXTRA_JAVA_OPTS="$JAVA_G1_GC_OPTS $JAVA_GC_LOG_OPTS $JAVA_SP_LOG_OPTS
$JAVA_HEAPDUMP_OPTS $JAVA_NET_OPTS -Dlog4j2.formatMsgNoLookups=true"

2. Restart the HYTE process

./bin/stop
./bin/start

3. Verify the property is set

$ ps -aef | grep -i hyte

Confirm property is present:

hyte  2429254 2429192 99 18:24 pts/0    00:00:21 /opt/hyte/java/active/bin/java ...
-Dlog4j2.formatMsgNoLookups=true ... org.apache.karaf.main.Main

8.2 Java JDK flags

Early reports indicated that making configuration changes to the Java JDK would mitigate the flaw. HYTE’s internal testing has NOT been able to successfully mitigate the flaw using these options.

1. Ineffective configuration properties

-Dcom.sun.jndi.ldap.object.trustURLCodebase=false
-Dcom.sun.jndi.rmi.object.trustURLCodebase=false
-Dcom.sun.jndi.cosnaming.object.trustURLCodebase=false

8.3 Flaw specific Java Agents

Several prominent technical influencers have produced Java Agents and other alternate tools to address the flaw. HYTE does NOT recommend customers apply these approaches for the following reasons:

1. Risk of bringing in Trojan horse or other exploit payload masked as a 'fix'

2. This is brand new code and requires additional operation steps to uninstall once the permanent fixes are available

3. Potentially introduce new bugs or security flaws


9. Frequently Asked Questions

What is Log4j?

Log4j (or Log4j2) is the most popular logging library for applications written in Java.

What is LogBack?

Logback is a very popular alternative logging library for applications written in Java.

Why does HYTE use Log4j2?

Log4j2 provides very high performance logging without dropping log messages under heavy load. This is critical for high-performing messaging environments.

https://logging.apache.org/log4j/2.x/performance.html

https://www.loggly.com/blog/benchmarking-java-logging-frameworks/


10. Change Log

Table 7. Changes

Date Note
2021-12-29 Updated for Log4j2 2.17.1
2021-12-21 Updated to account for latest updates
2021-12-13 Updated mitigation matrix
2021-12-12 JDK upgrade secondary mitigation option questioned
2021-12-11 JDK upgrade secondary mitigation option communicated
2021-12-10 Initial publication


12. Contact HYTE

For technical assistance, please open a support ticket in the HYTE Portal

You are solely responsible for determining the appropriateness of using and distributing any publicly available HYTE information and you assume all risks associated with its use, including but not limited to the risks and costs of program errors, compliance with applicable laws, damage to or loss of data, programs or equipment, and the unavailability or interruption of operation. This information is not intended to be used in any situation where a failure could cause risk of injury or damage to property. This is not intended as legal advice and no warranty is provided with this information. Use at your own risk.

When using Google Translate, please note that commands may need to be executed with English syntax depending on your system configuration.

Check out our new YouTube channel, The Uplink, if you are interested in the latest discussions regarding messaging, streaming, middleware, integration and all technologies that surround those topics in the Enterprise.