title: Best practices for serializing complex models in Flink
body: <p>I am using a UDF to apply some ML models to a data stream. Because the <code>Model</code> class (from a third party library) cannot be serialized by Flink automatically, I am using two variables as illustrated below:</p>
  
  <pre class="lang-scala prettyprint-override"><code>class MyUDF extends KeyedCoProcessFunction[String, ModelDef, Data, Prediction]
    with CheckpointedFunction {
  
    // To hold loaded models
    @transient private var models: HashMap[(String, String), Model] = _
  
    // For serialization purposes
    @transient private var modelsBytes: MapState[(String, String), Array[Bytes]] = _
  
    ...
  }
  
  </code></pre>
  
  <p>Where:</p>
  
  <ul>
  <li><code>models</code> holds the loaded (running) models (created from a <code>ModelDef</code>, basically a string)</li>
  <li><code>modelsBytes</code> is the real (keyed) state, which holds the same models but as a blob of bytes so that checkpoints work well.</li>
  </ul>
  
  <p>The overall solution is simple (just requires to invoke <code>fromBytes</code>/<code>toBytes</code> on my models when restoring/saving them) but I don't know if this is a common/best practice. Having two variables for essentially the same thing looks like a quirk. For instance, <a href="https://github.com/typesafehub/fdp-modelserver/blob/master/flinkserver/src/main/scala/com/lightbend/modelserver/typeschema/ModelTypeSerializer.scala" rel="nofollow noreferrer">here</a> you can find an example which uses a <code>TypeSerializer[Option[Model]]</code> instead, which looks cleaner but also more complicated to implement.</p>
  
  <p>So, basically:</p>
  
  <ul>
  <li><p>Should I use the <code>TypeSerializer</code> approach or having a somehow duplicated state for running/serialized models is ok?</p></li>
  <li><p>Also, if you could please point me to some docs/examples regarding custom types serialization in Flink, that would be great, generally I find the official docs a bit lacking on this regard.</p></li>
  </ul>