<p>I'm running Flink 1.11 on k8s cluster and getting the following error when trying to update the log4j-console.properties file:</p>
<pre><code>Starting Task Manager
Enabling required built-in plugins
Linking flink-s3-fs-hadoop-1.11.1.jar to plugin directory
Successfully enabled flink-s3-fs-hadoop-1.11.1.jar
sed: couldn't open temporary file /opt/flink/conf/sedl2dH0X: Read-only file system
sed: couldn't open temporary file /opt/flink/conf/sedPLYAzY: Read-only file system
/docker-entrypoint.sh: 72: /docker-entrypoint.sh: cannot create /opt/flink/conf/flink-conf.yaml: Permission denied
sed: couldn't open temporary file /opt/flink/conf/sede0G5LW: Read-only file system
/docker-entrypoint.sh: 120: /docker-entrypoint.sh: cannot create /opt/flink/conf/flink-conf.yaml.tmp: Read-only file system
Starting taskexecutor as a console application on host flink-taskmanager-c765c947c-qx68t.
Exception in thread &quot;main&quot; java.lang.NoClassDefFoundError: com/fasterxml/jackson/databind/ser/FilterProvider
    at org.apache.logging.log4j.core.layout.JsonLayout.&lt;init&gt;(JsonLayout.java:158)
    at org.apache.logging.log4j.core.layout.JsonLayout.&lt;init&gt;(JsonLayout.java:69)
    at org.apache.logging.log4j.core.layout.JsonLayout$Builder.build(JsonLayout.java:102)
    at org.apache.logging.log4j.core.layout.JsonLayout$Builder.build(JsonLayout.java:77)
    at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:122)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:1002)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:942)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:934)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:934)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:552)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:241)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:288)
    at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:579)
    at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:651)
    at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:668)
    at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:253)
    at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:153)
    at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45)
    at org.apache.logging.log4j.LogManager.getContext(LogManager.java:194)
    at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:138)
    at org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:45)
    at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:48)
    at org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:30)
    at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:329)
    at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:349)
    at org.apache.flink.runtime.taskexecutor.TaskManagerRunner.&lt;clinit&gt;(TaskManagerRunner.java:89)
Caused by: java.lang.ClassNotFoundException: com.fasterxml.jackson.databind.ser.FilterProvider
    at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(Unknown Source)
    at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(Unknown Source)
    at java.base/java.lang.ClassLoader.loadClass(Unknown Source)
</code></pre>
<p>my log4j-console.properties:</p>
<pre><code>rootLogger.level = INFO
#rootLogger.appenderRef.console.ref = ConsoleAppender

appender.console.type = Console
appender.console.name = STDOUT
appender.console.layout.type = PatternLayout
appender.console.layout.pattern = %d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n

appender.kafka.type = Kafka
appender.kafka.name = Kafka
appender.kafka.topic = test
appender.kafka.layout.type = JsonLayout
appender.kafka.layout.complete = false
appender.kafka.additional1.type = KeyValuePair
appender.kafka.additional1.key=app
appender.kafka.additional1.value=TEST
appender.kafka.additional2.type = KeyValuePair
appender.kafka.additional2.key=subsystem
appender.kafka.additional2.value=TEST
appender.kafka.additional3.type = KeyValuePair
appender.kafka.additional3.key=deployment
appender.kafka.additional3.value=TEST
appender.kafka.property.bootstrap.servers=***

rootLogger.appenderRef.console.ref = STDOUT
rootLogger.appenderRef.kafka.ref = Kafka
</code></pre>
<p>Im using &quot;flink:1.11.1-scala_2.11-java11&quot; docker image and validated that all log4j2 dependencies are in the classpath.</p>
<p>I have also tried to create a new docker image from the above base image and add to it the missing dependency and yet nothing happened.</p>

<p>I too suffered from this bug. The issue here is that when the task manager and job managers start they are running with a modified classpath, not the JAR that you've built via your build system.</p>
<p>See the <code>constructFlinkClassPath</code> in the <a href="https://github.com/apache/flink/blob/af681c63821c027bc7a233560f7765b686d0d244/flink-dist/src/main/flink-bin/bin/config.sh#L20" rel="nofollow noreferrer">flink source code</a>. To prove this out, revert the JSON logging pattering and check out the classpath in the tm/jm logs on startup. You'll notice that your JAR isn't on the classpath.</p>
<p>To fix this issue you need to provide the dependencies (in this case you'll need <code>jackson-core</code> <code>jackson-annotations</code> and <code>jackson-databind</code>) to the <code>lib</code> folder within the tm/jm nodes (the <code>lib</code> folder is included by default in the flink classpath).</p>
<p>If you are using docker, you can do this when you build the container (<code>RUN wget...</code>).</p>
