title: How do I properly externalize spring-boot kafka-streams configuration in a properties file?
body: <p>I'm trying to externalize the configuration of a spring-kafka application that I currently have written in Java code. 
  Should I be putting the <code>ProducerConfig</code> and <code>ConsumerConfig</code> values into <code>spring.kafka.streams.properties</code>, or will they be properly configured if I provide them through<code>spring.kafka.producer</code> and <code>spring.kafka.consumer</code>?</p>
  
  <p>So far, it appears that I am supposed to get all of my configuration into a bean of type <code>KafkaStreamsConfiguration</code> in order to configure my kafka-streams app. Currently, I am doing that by setting <code>ProducerConfig</code> and <code>ConsumerConfig</code> values directly in code.</p>
  
  <p>When I externalize this configuration, it appears that setting the property values from <code>ProducerConfig</code> and <code>ConsumerConfig</code> in the <code>application.properties</code> file does not correlate with them being in the <code>KafkaStreamsConfiguration</code> created by spring-boot (I confirmed this by autowiring the configuration somewhere and taking a look at it).</p>
  
  <p>If I instead provide the <code>ProducerConfig</code> and <code>ConsumerConfig</code> values through <code>spring.kafka.streams.properties</code> they show up in the <code>KafkaStreamsConfiguration</code>.</p>
  
  <p>Here is my old Java configuration:</p>
  
  <pre class="lang-java prettyprint-override"><code>@Bean(name = KafkaStreamsDefaultConfiguration.DEFAULT_STREAMS_CONFIG_BEAN_NAME)
      public KafkaStreamsConfiguration kStreamsConfigs() {
          Map&lt;String, Object&gt; props = new HashMap&lt;&gt;();
  
          props.put(StreamsConfig.APPLICATION_ID_CONFIG, applicationId);
          props.put(ConsumerConfig.GROUP_ID_CONFIG, groupId);
          props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);
          props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass().getName());
          props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.String().getClass().getName());
          props.put(StreamsConfig.DEFAULT_TIMESTAMP_EXTRACTOR_CLASS_CONFIG, WallclockTimestampExtractor.class.getName());
          props.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, commitInterval);
          props.put(AbstractKafkaAvroSerDeConfig.SCHEMA_REGISTRY_URL_CONFIG, registryUrl);
          props.put(StreamsConfig.DEFAULT_DESERIALIZATION_EXCEPTION_HANDLER_CLASS_CONFIG, LogAndContinueExceptionHandler.class.getName());
          props.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "lz4");
          props.put("replication.factor", replicationFactor);
          props.put(StreamsConfig.STATE_DIR_CONFIG, "/var/lib/kafka-streams");
          props.put(StreamsConfig.STATE_CLEANUP_DELAY_MS_CONFIG, "600000");
          props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "latest");
  
          return new KafkaStreamsConfiguration(props);
      }
  </code></pre>
  
  <p>this ends up with <code>ProducerConfig</code> and <code>ConsumerConfig</code> values not being in the <code>KafkaStreamsConfiguration</code> at runtime:</p>
  
  <pre><code>spring.kafka.streams.bootstrap-servers=localhost:9092
  spring.kafka.streams.properties.schema.registry.url=http://localhost:8081
  spring.kafka.streams.application-id=&lt;application_id&gt;
  spring.kafka.consumer.group-id=&lt;group_id&gt; #this won't show up in KafkaStreamsConfiguration
  spring.kafka.streams.replication-factor=1
  spring.kafka.streams.properties.commit.interval.ms=100
  spring.kafka.streams.properties.default.key.serde=org.apache.kafka.common.serialization.Serdes$StringSerde
  spring.kafka.streams.properties.default.value.serde=org.apache.kafka.common.serialization.Serdes$StringSerde
  spring.kafka.streams.properties.default.deserialization.exception.handler=org.apache.kafka.streams.errors.LogAndContinueExceptionHandler
  spring.kafka.producer.compression-type=lz4 #this won't show up in KafkaStreamsConfiguration
  spring.kafka.streams.properties.state.dir=/var/lib/kafka-streams
  spring.kafka.streams.properties.state.cleanup.delay.ms=600000
  spring.kafka.consumer.auto-offset-reset=latest #this won't show up in KafkaStreamsConfiguration
  spring.kafka.streams.properties.timestamp.extractor=org.apache.kafka.streams.processor.WallclockTimestampExtractor
  </code></pre>
  
  <p>This, however, does result in <code>KafkaStreamsConfiguration</code> having the values as expected:</p>
  
  <pre><code>spring.kafka.streams.bootstrap-servers=localhost:9092
  spring.kafka.streams.properties.schema.registry.url=http://localhost:8081
  spring.kafka.streams.application-id=&lt;application_id&gt;
  spring.kafka.streams.properties.group-id=&lt;group_id&gt; #this shows up in KafkaStreamsConfiguration
  spring.kafka.streams.replication-factor=1
  spring.kafka.streams.properties.commit.interval.ms=100
  spring.kafka.streams.properties.default.key.serde=org.apache.kafka.common.serialization.Serdes$StringSerde
  spring.kafka.streams.properties.default.value.serde=org.apache.kafka.common.serialization.Serdes$StringSerde
  spring.kafka.streams.properties.default.deserialization.exception.handler=org.apache.kafka.streams.errors.LogAndContinueExceptionHandler
  spring.kafka.streams.properties.compression-type=lz4 #this shows up in KafkaStreamsConfiguration
  spring.kafka.streams.properties.state.dir=/var/lib/kafka-streams
  spring.kafka.streams.properties.state.cleanup.delay.ms=600000
  spring.kafka.streams.properties.auto-offset-reset=latest #this shows up in KafkaStreamsConfiguration
  spring.kafka.streams.properties.timestamp.extractor=org.apache.kafka.streams.processor.WallclockTimestampExtractor
  </code></pre>
  
  <p>I was expecting the <code>ProducerConfig</code> and <code>ConsumerConfig</code> values to propagate to the <code>KafkaStreamsConfiguration</code> when being set through <code>spring.kafka.producer</code> and <code>spring.kafka.consumer</code> respectively. Especially since I get Intellisense in IntelliJ for the Producer and Consumer configs in <code>application.properties</code>.</p>
  
  <p>That said, do I need to make sure I'm setting those through <code>spring.kafka.streams.properties</code> in order for the app to be properly configured?</p>