<p>I have my own Spring Cloud Data Flow processor with Python inside, I used this sample as a guide: <a href="https://dataflow.spring.io/docs/recipes/polyglot/processor/" rel="nofollow noreferrer">https://dataflow.spring.io/docs/recipes/polyglot/processor/</a>.
Then I want to scale and create three of these processors, so using <code>spring.cloud.deployer.myApp.count=3</code> I create 3 pods with Python inside.
I slightly modified a code in sample: when I create a Kafka consumer, I also pass a group id, so messages should be load balanced.</p>

<pre><code>consumer = KafkaConsumer(get_input_channel(), group_id=get_consumer_group(), bootstrap_servers=[get_kafka_binder_brokers()])
</code></pre>

<p>The issue is that SCDF creates a Kafka topic with only 1 partition, so messages arrive to one pod only. 
So I am wondering:</p>

<ul>
<li>Should I somehow configure SCDF to create a Kafka topic with 3 partitions?</li>
<li>Or should I not rely on SCDF and create topics on my own in Python? I suppose this will be redundant, since SCDF also creates this topic.</li>
<li>What component in SCDF is actually responsible for Kafka topics creation? And how can I influence it regarding number of partitions?</li>
<li>If I stop this stream and launch again with 4 processor steps, should the topic be extended with the 4th partition? Because currently no new partitions get created.</li>
</ul>

Maks Shal
<p>Please take a moment to review Spring Cloud Data Flow's <a href="https://dataflow.spring.io/docs/concepts/architecture/#server-components" rel="nofollow noreferrer">responsibilities</a>. In case it is not clear, SCDF doesn't either interact with the backing messaging middleware like Kafka or uses it at runtime. In other words, SCDF doesn't create topics or partitions associated with it — it merely automates the configuration of the Spring Cloud Stream (SCSt) properties.</p>

<p>If you're using SCSt in a custom processor, however, the framework automates the binding of the desired channels to the underlying topics in the middleware. The framework also has the facility to alter the partitioning behavior. You can deploy the processor with the over-partitioned topic, too. There are <a href="https://cloud.spring.io/spring-cloud-static/spring-cloud-stream-binder-kafka/3.0.0.RELEASE/reference/html/spring-cloud-stream-binder-kafka.html#_configuration_options" rel="nofollow noreferrer">several other configuration options</a> to build the desired streaming data processing behavior.</p>

<p>The Python sample that you're looking at doesn't have all the features that SCSt provides. The recipe is an example walkthrough of how someone can build a native processor-style application in Python, where the producer and consumer configurations are manually created within the Python code itself. Neither SCDF nor SCSt influences the application behavior in this recipe.</p>

<blockquote>
  <p>Should I somehow configure SCDF to create a Kafka topic with 3 partitions?</p>
</blockquote>

<p>As discussed previously, SCDF does not interact with Kafka. </p>

<blockquote>
  <p>Or should I not rely on SCDF and create topics on my own in Python? I suppose this will be redundant, since SCDF also creates this topic.</p>
</blockquote>

<p>If your custom processor is not a Spring Cloud Stream application, yes, it is your responsibility to define the topics + partitions in your code explicitly.</p>

<blockquote>
  <p>What component in SCDF is actually responsible for Kafka topics creation? And how can I influence it regarding number of partitions?</p>
</blockquote>

<p>Spring Cloud Stream. See the explanation above.</p>

<blockquote>
  <p>If I stop this stream and launch again with 4 processor steps, should the topic be extended with the 4th partition? Because currently no new partitions get created.</p>
</blockquote>

<p>You don't necessarily need to restart the streaming data pipeline. If your topic is over-partitioned upfront, yes, any additional consumers at runtime should be able to participate in the competing consumer relationship automatically. Keep an eye on <a href="https://github.com/spring-io/dataflow.spring.io/issues/156" rel="nofollow noreferrer">spring-io/dataflow.spring.io#156</a> — we are in the process of adding a recipe to demonstrate manual and autoscaling possibilities using SCSt + SCDF + Kafka.</p>
