Log4Shell

Owned by Julian Ladisch
Last updated: Jan 17, 2022Version comment

FOLIO is affected by the Log4Shell security issue (CVE-2021-44228 and CVE-2021-45046) and a denial of service attack issue (CVE-2021-45105) in the log4j-core library used by many FOLIO backend modules and FOLIO's Okapi.

FOLIO is not affected by the remote code execution (RCE) attack using JDBC Appender (CVE-2021-44832).

Advisories of the Log4j project: https://logging.apache.org/log4j/2.x/security.html

This page gives advice for FOLIO installations and FOLIO developers to mitigate the risks of these issues.

Please contribute, this is a wiki page!

Sysops/Devops

Okapi upgrade

All existing installations should upgrade to Okapi v4.11.1.

This version works for all releases (Lotus, Kiwi, Juniper, Iris), there were no breaking changes.

No environment variables or system variables are needed to fix Okapi.

Okapi protects all modules where it proxies requests to (the mod-* modules) from the denial-of-service attack of CVE-2021-45105 (infinite recursion in MDC lookup evaluation) by filtering malicious values in the fields used for MDC (OKAPI-1058), therefore those modules don't need log4j >= 2.17.0 and can safely continue to use log4j 2.16.0.

Okapi doesn't protect the edge-* modules, edge-* modules that might be affected need to upgrade to log4j >= 2.17.0.

Hot-fix preview

Sysops that don't want to wait for the official hot-fix release can install the module versions of the platform-complete branch:

https://github.com/julianladisch/platform-complete/actions/workflows/log4shell-scan.yml scans these branches for vulnerable log4j versions. Click on the release and on "Run cat result.txt" to see the results.  The scan runs daily. The result history shows that

  • all platform-complete modules of Juniper (R2-2021) have been fixed since December 22, 2021
  • all platform-complete modules of Kiwi (R3-2021) have been fixed since December 22, 2021

There are no plans to completely fix Iris (R1-2021) because Iris hasn't officially been declare to receive long term support and FOLIO supports only the latest two flower releases.

Configuration variables for back-end modules

Many back-end modules are affected by the Log4Shell issue, until new jar files and new docker containers with a fixed version are ready the existing back-end modules should be reconfigured with an environment variable that disables the flaw for most cases in log4j:

  • LOG4J_FORMAT_MSG_NO_LOOKUPS=true
  • Append -Dlog4j2.formatMsgNoLookups=true to the JAVA_OPTIONS variable

Pick one of the two options.

Example for the second option: If the existing configuration has JAVA_OPTIONS="-XX:MaxRAMPercentage=66.0" then the new configuration should be
JAVA_OPTIONS="-XX:MaxRAMPercentage=66.0 -Dlog4j2.formatMsgNoLookups=true"

SQL query to do this, posted by Lucy Menon on #sys-ops Slack channel, assuming that all modules in use already have a JAVA_OPTIONS env entry:

okapi=# with foo as (select ('{descriptor,env,'||index-1||',value}')::text[] as path, to_jsonb(env->>'value'||' -Dlog4j2.formatMsgNoLookups=true') as nval, json->'instId' as instId from deployments, jsonb_array_elements(json->'descriptor'->'env') with ordinality arr(env, index) where env->>'name' = 'JAVA_OPTIONS') update deployments set json = jsonb_set(deployments.json, foo.path, foo.nval) from foo where deployments.json->'instId' = foo.instId;
okapi=# with foo as (select ('{launchDescriptor,env,'||index-1||',value}')::text[] as path, to_jsonb(env->>'value'||' -Dlog4j2.formatMsgNoLookups=true') as nval, modulejson->'id' as modid from modules, jsonb_array_elements(modulejson->'launchDescriptor'->'env') with ordinality arr(env, index) where env->>'name' = 'JAVA_OPTIONS') update modules set modulejson = jsonb_set(modules.modulejson, foo.path, foo.nval) from  foo where modules.modulejson->'id' = foo.modid;

If you use Okapi to deploy the modules you may use this method to set the LOG4J_FORMAT_MSG_NO_LOOKUPS variable and redeploy them:

curl -w '\n' -D - -X POST -H "Content-Type: application/json" -d '{"name":"LOG4J_FORMAT_MSG_NO_LOOKUPS","value":"true"}' http://localhost:9130/_/env
systemctl restart okapi

Is using configuration variables secure?

Not completely. It only limits exposure while leaving some attack vectors open. Using the configuration variables is a temporary measure for the time until patched FOLIO modules are available. Please upgrade to patched modules as soon as possible.

From:  https://logging.apache.org/log4j/2.x/security.html 

History

Older (discredited) mitigation measures

This page previously mentioned other mitigation measures, but we discovered that these measures only limit exposure while leaving some attack vectors open.

Other insufficient mitigation measures are: setting system property log4j2.formatMsgNoLookups or environment variable LOG4J_FORMAT_MSG_NO_LOOKUPS to true for releases >= 2.10, or modifying the logging configuration to disable message lookups with %m{nolookups}, %msg{nolookups} or %message{nolookups} for releases >= 2.7 and <= 2.14.1.

The reason these measures are insufficient is that, in addition to the Thread Context attack vector mentioned above, there are still code paths in Log4j where message lookups could occur: known examples are applications that use Logger.printf("%s", userInput), or applications that use a custom message factory, where the resulting messages do not implement StringBuilderFormattable. There may be other attack vectors.

The safest thing to do is to upgrade Log4j to a safe version, or remove the JndiLookup class from the log4j-core jar.

Are Kafka and Zookeeper affected?

No, because they use log4j 1.x and FOLIO doesn't set the JMS configuration TopicBindingName or TopicConnectionFactoryBindingName. For details see https://kafka.apache.org/cve-list and https://issues.apache.org/jira/browse/ZOOKEEPER-4423

Is ElasticSearch affected?

No, because FOLIO uses version 7.10. For details see https://discuss.elastic.co/t/apache-log4j2-remote-code-execution-rce-vulnerability-cve-2021-44228-esa-2021-31/291476

Developers

Modules that use a vulnerable log4j version must be upgraded to >= 2.17.0, however, mod-* modules that already have upgraded to 2.16.0 don't need to upgrade to 2.17.0. Okapi protects all modules where it proxies requests to from the denial-of-service attack of CVE-2021-45105 (infinite recursion in MDC lookup evaluation) by filtering malicious values in the fields used for MDC (OKAPI-1058), therefore those modules don't need log4j 2.17.0 and can safely continue to use log4j 2.16.0.

Modules don't need to be upgraded to log4j 2.17.1 because they are not affected by the remote code execution (RCE) attack using JDBC Appender (CVE-2021-44832) that can only be triggered through malicious configuration files; since these are hard-coded into each module, sysop permissions would be required to change them and, if an attacker has sysop permissions, they can do far more than exploit log4j. Sysop permissions are needed to inject a configuration file and/or set it via the -Dlog4j.configurationFile system property; a search through the source code shows that no FOLIO module (re)configures log4j using code (programmatic API, setConfigLocation) and therefore cannot be (re)configured at runtime without sysop permissions.

Each edge-* module must upgrade to log4j >= 2.17.0 unless you know that it doesn't use MDC lookups.

Please do the work in this order: Juniper, Kiwi. Work on edge module before back-end modules.

After all mod-* and edge-* modules have been upgraded to log4j >= 2.16.0 upgrade edge-* modules with log4j 2.16.0 to log4j 2.17.0 unless they don't use MDC lookups.

https://github.com/julianladisch/platform-complete/actions/workflows/log4shell-scan.yml scans the platfrom-complete branches R1-2021, R2-2021 and R3-2021 for vulnerable log4j versions. Click on the release and on "Run cat result.txt" to see the results. The scan runs daily.

RMB based modules

Modules based on Raml Module Builder (RMB) should upgrade to a fixed version:

RMB based modules should remove any direct log4j dependency from their pom.xml and use the log4j version provided by RMB if they want the log4j version being automatically updated when updating RMB.

Spring Boot based modules

For modules based off of Spring Boot are vulnerable as spring-boot-starter-log4j2 includes the vulnerable version of log4j.  Spring Boot has updated their version to 2.17.0 (as of 12/23/2021), therefore, pulling in the newest spring-boot-starter-log4j2 should be sufficient.

In you need to specifically override the version provided by Spring Boot, you can override the variable through the log4j2.version property like so (per their instructions for Maven, as FOLIO base uses their parent POM):

<properties>
  ...
  <log4j2.version>2.17.1</log4j2.version>
</properties>

Maven based modules

For modules that use maven run

mvn dependency:tree -Dincludes=org.apache.logging.log4j

or – for verbose output –

mvn dependency:tree -Dincludes=org.apache.logging.log4j -Dverbose

to see whether log4j is used and which version. Example output:

[INFO] org.folio:mod-users:jar:18.2.0-SNAPSHOT
[INFO] +- org.folio:domain-models-runtime:jar:33.2.0-SNAPSHOT:compile
[INFO] |  +- org.apache.logging.log4j:log4j-core:jar:2.14.1:compile
[INFO] |  \- org.apache.logging.log4j:log4j-api:jar:2.14.1:compile
[INFO] \- org.folio:folio-service-tools-test:jar:1.7.1:test
[INFO]    \- org.apache.logging.log4j:log4j-slf4j-impl:jar:2.14.1:compile

This shows log4j 2.14.1 comes from domain-models-runtime (= RMB) so upgrading RMB resolves it.

Non-RMB modules might use other ways to update log4j.

If a module needs to directly set the version of log4j it should use the a <dependencyManagement> entry with log4j-bom to keep all log4j components (log4j-api, log4j-core, ...) in sync:

  <dependencyManagement>
    <dependencies>
      <dependency>
        <groupId>org.apache.logging.log4j</groupId>
        <artifactId>log4j-bom</artifactId>
        <version>2.17.1</version>
        <type>pom</type>
        <scope>import</scope>
      </dependency>
    </dependencies>
  </dependencyManagement>

Example: https://github.com/folio-org/mod-authtoken/blob/v2.9.1/pom.xml#L60-L100

If not using the log4j-bom be sure to specify the version in all of your log4j dependencies, check with mvn dependency:tree.

Should developers set environment or system variables in modules?

No, updating to log4j >= 2.16.0 is sufficient. LOG4J_FORMAT_MSG_NO_LOOKUPS=true or -Dlog4j2.formatMsgNoLookups=true should only be used by sysops for unpatched modules as a temporary fix. Don't add them to the ModuleDescriptors or LaunchDescriptors a module ships with. For details see section "Is using configuration variables secure?" above.

Should developers remove JndiLookup.class from modules?

No, updating to log4j >= 2.16.0 is sufficient. 

For the time while waiting for a patched module the advice from Apache (see above) suggest that the safest thing to do may be to remove JndiLookup.class from the classpath (usually from a module's fat jar).  However, the effort to remove the class and release this stripped module is as big as updating log4j and releasing a patched module.

Modules without log4j

Modules that don't use log4j don't need to do anything.