title: stateSnapshots on demand in mapWithState
body: <p>I am streaming data from Kafka (batch interval 10 sec), convert the RDD to a PairRDD, and then storing the RDD into the state using mapWithState(). Below is the code:</p>
  
  <pre><code>    JavaPairDStream&lt;String, Object&gt; transformedStream = stream
              .mapToPair(record -&gt; new Tuple2&lt;&gt;(record.getKey(), record))
              .mapWithState(StateSpec.function(updateDataFuncGDM).numPartitions(32)).stateSnapshots();
  
  transformedStream.foreachRDD(rdd -&gt; {
          //if flag is true, put the RDD to a SQL table, and run a query to do some aggregations liek sum, avg etc
         // if flag is false, return;
          }
  </code></pre>
  
  <p>Now, i keep on updating data in the state, and on a certain event, i change the flag to true, and i put this data in the table, and do my calculations.</p>
  
  <p>The problem here is that since i am getting "stateSnapshots" in every batch, its not efficient, and mapWithState keeps lots of data in memory, and as the state grows it will become even worse. Also, since mapWithState checkpoints data after every 10 iterations, it takes a lot of time because the data is very large.</p>
  
  <p>I want to get a stateSnapshot of the state only on demand (i.e. only in the iteration of foreachRDD when the flag is true)
  But i didn't find a lot of ways to play around with the state</p>