title: Restoring Large State in Apache Flink Streaming Job
body: <p>We have a cluster running Hadoop and YARN on AWS EMR with one core and one master, each with 4 vCores, 32 GB mem, 32 GB disc. We only have one long-running YARN application, and within that, there are only one or two long-running Flink applications, each with a parallelism of 1. Checkpointing has a 10 minute interval with a minimum of 5 minutes between. We use EventTime with a window of 10 minutes and a watermark duration of 15 seconds. The state is stored in S3 through the FsStateBackend with async snapshots enabled. Exactly-Once checkpointing is enabled as well.</p>
  
  <p>We have UUIDs set up for all operators but don't have HA set up for YARN or explicit max parallelism for the operators.</p>
  
  <p>Currently, when restoring from a checkpoint (3GB) the processing holds at the windowing until a <code>org.apache.flink.util.FlinkException: The assigned slot &lt;container_id&gt; was removed</code> error is thrown during the next checkpoint. I have seen that all but the operator with the largest state (which is a ProcessFunction directly after the windowing), finish checkpointing.</p>
  
  <p>I know it is <strong>strongly suggested</strong> to use RocksDB for production, but is that mandatory for a state that most likely won't exceed 50GB?</p>
  
  <p>Where would be the best place to start addressing this problem? Parallelism?</p>