title: Spark No Encoder found for java.io.Serializable in Map[String, java.io.Serializable]
body: <p>I am writing a spark job that the dataset is pretty flexible, it's defined as <code>Dataset[Map[String, java.io.Serializable]]</code>.</p>
  
  <p>now the problem start to show up, spark runtime complains about <code>No Encoder found for java.io.Serializable</code>. I've tried kyro serde, still showing the same error message. </p>
  
  <p>the reason why I have to use this weird Dataset type is because I have flexible fields per Row. and the map looks like:</p>
  
  <pre><code>Map(
    "a" -&gt; 1,
    "b" -&gt; "bbb",
    "c" -&gt; 0.1,
    ...
  )
  </code></pre>
  
  <p>is there anyway in Spark to handle this flexible dataset type? </p>
  
  <p>EDIT: 
  here is the solid code anyone can try.</p>
  
  <pre><code>import org.apache.spark.sql.{Dataset, SparkSession}
  
  object SerdeTest extends App {
    val sparkSession: SparkSession = SparkSession
      .builder()
      .master("local[2]")
      .getOrCreate()
  
  
    import sparkSession.implicits._
    val ret: Dataset[Record] = sparkSession.sparkContext.parallelize(0 to 10)
      .map(
        t =&gt; {
          val row = (0 to t).map(
            i =&gt; i -&gt; i.asInstanceOf[Integer]
          ).toMap
  
          Record(map = row)
        }
      ).toDS()
  
    val repartitioned = ret.repartition(10)
  
  
    repartitioned.collect.foreach(println)
  }
  
  case class Record (
                    map: Map[Int, java.io.Serializable]
                    )
  </code></pre>
  
  <p>the above code will give you error Encoder not found: </p>
  
  <pre><code>Exception in thread "main" java.lang.UnsupportedOperationException: No Encoder found for java.io.Serializable
  - map value class: "java.io.Serializable"
  - field (class: "scala.collection.immutable.Map", name: "map")
  </code></pre>