title: new spark.sql.shuffle.partitions value not used after checkpointing
body: <p>I have a Spark's Structured Streaming application with checkpointing to write output in parquet and using the default spark.sql.shuffle.partitions = 200. I need to change the shuffle partitions but the new value is not used. Here is the content of a checkpoint offset after the application is restarted:  </p>
  
  <pre><code>{"batchWatermarkMs":1520054221000,"batchTimestampMs":1520054720003,"conf":{"spark.sql.shuffle.partitions":"200"}}
  </code></pre>
  
  <p>Do I need to set the number of partitions in the code instead of setting it with --conf?</p>