title: Prevent repartitioning in spark when using mapgroupswithstate in spark structured streaming
body: <p>I am processing data from kafka with spark structured streaming, and am using mapgroupswithstate as I need to keep a state as the data is arriving and is analysed.</p>
  <p>In kafka there is one partition per user.</p>
  <p>In spark I am processing this data, and I was hoping that I wouldn't need to do any shuffle because as the documentation states: &quot;By default, Spark has a 1-1 mapping of topicPartitions to Spark partitions consuming from Kafka.&quot; but since I need to do a groupByKey before using mapgroupswithstate I can't find a way to stop spark from doing a shuffle in this stage if I have more than one partition in spark, even if the number of partitions in kafka is the same as the number of partitions in spark. I am finding that having a single partition in spark has more performance than having multiple, as this way the data isn't shuffled from what I see in the spark UI. This is obviously not ideal because a single partition won't be processed in parallel(in case of multiple users simultaneously). So what I wanted was to keep the same partitions in spark as those that come from kafka and use mapgroupswithstate in each partition.</p>
  <p>The number of partitions used in kafka is variable, because some users might be generating data while others aren't.</p>
  <p>Is there a way to do this inside the same spark app?</p>
  <p>Is the best solution to have multiple single partition spark apps, one for each user(reading from a single kafka partition)? This is obviously far from ideal because as the user base grows, the spark apps have to be manually scaled. Having 500 users would require 500 spark apps...</p>
  <p>Thank you.</p>