<p>I want to disable kafka logs from spark running in kubernetes. I have set in Spark application log4j properties below entries</p>
<pre><code>log4j.rootLogger=INFO, stdout, stderr, kafkaAppender
  log4j.logger.Client=INFO, stdout, stderr
  log4j.appender.stdout=org.apache.log4j.ConsoleAppender
  log4j.appender.stdout.Threshold=INFO
  log4j.appender.stdout.Target=System.out
  log4j.appender.stdout.immediateFlush=true
  log4j.logger.kafka=ERROR, WARN
  log4j.logger.org.apache.kafka=ERROR, WARN
  log4j.logger.org.apache.kafka.clients.producer.KafkaProducer=ERROR, WARN
  log4j.logger.org.apache.kafka.streams=ERROR, WARN
  log4j.additivity.org.apache.kafka.streams=false
  log4j.logger.org.apache.kafka.clients.consumer=ERROR, WARN
  log4j.logger.org.apache.kafka.clients.producer=ERROR, WARN
  log4j.logger.org.apache.kafka.clients.consumer.internals.SubscriptionState=ERROR, WARN 
</code></pre>
<p>Still I see INFO logs like below in kubernetes driver pod</p>
<pre><code>INFO SubscriptionState: [Consumer... 
INFO Metadata
INFO AbstractCoordinator
INFO MemoryStore
INFO CodeGenerator
INFO KafkaConsumer: [Consumer
INFO Metadata
</code></pre>
<p>Am I adding the correct configs in</p>

Panda
<p>Spark <a href="https://spark.apache.org/docs/latest/configuration.html#configuring-logging" rel="nofollow noreferrer">uses log4j2</a> now, which is why log4j.properties will not be picked up.</p>
<p>Inside the containers, you should have some environment variable <code>SPARK_CONF_DIR</code>, and in that path, you can set the <code>log4j2.properties</code> file.</p>
<p>You can find the default template in the Spark source code - <a href="https://github.com/apache/spark/blob/master/conf/log4j2.properties.template" rel="nofollow noreferrer">https://github.com/apache/spark/blob/master/conf/log4j2.properties.template</a></p>
<p>So, turning off kafka might look something like this</p>
<pre><code>logger.kafkaClients.name = org.apache.kafka.clients
logger.kafkaClients.level = off
</code></pre>
