title: Spark Merge schema on write - getting error: "The column number of the existing table doesn't match the data
body: <p>I am working with Spark version 2.2.1 and trying to write new data to the existing table with additional column. Now I know already that it will not work, but I wonder what is the best workaround.</p>
  <p>Here is a code example:</p>
  <pre><code>case class MyMockV1(a: String)
  case class MyMockV2(a: String, b:String)
  
  object StackOverflowSparkQuestion extends App{
    val sparkConf = new SparkConf()
    sparkConf.setMaster(s&quot;local[2]&quot;)
    val spark = SparkSession.builder().config(sparkConf).getOrCreate()
  
    val values = List(MyMockV1(&quot;test1&quot;), MyMockV1(&quot;test2&quot;))
    val values2 = List(MyMockV2(&quot;test1&quot;,&quot;test2&quot;), MyMockV2(&quot;test2&quot;,&quot;test3&quot;))
  
    val tableName = &quot;myTableTest&quot;
    spark.createDataFrame(values).coalesce(1).write.mode(SaveMode.Append).saveAsTable(tableName)
    spark.createDataFrame(values2).coalesce(1).write.mode(SaveMode.Append).saveAsTable(tableName)
  
  
  }
  </code></pre>
  <p>And I get this error:</p>
  <pre><code>Exception in thread &quot;main&quot; org.apache.spark.sql.AnalysisException: The column number of the existing table default.myTableTest(struct&lt;a:string&gt;) doesn't match the data schema(struct&lt;a:string,b:string&gt;);
      at org.apache.spark.sql.execution.datasources.PreprocessTableCreation$$anonfun$apply$2.applyOrElse(rules.scala:121)
      at org.apache.spark.sql.execution.datasources.PreprocessTableCreation$$anonfun$apply$2.applyOrElse(rules.scala:75)
      at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$2.apply(TreeNode.scala:267)
      at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$2.apply(TreeNode.scala:267)
      at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:70)
      at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:266)
      at org.apache.spark.sql.catalyst.trees.TreeNode.transform(TreeNode.scala:256)
      at org.apache.spark.sql.execution.datasources.PreprocessTableCreation.apply(rules.scala:75)
      at org.apache.spark.sql.execution.datasources.PreprocessTableCreation.apply(rules.scala:71)
      at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1$$anonfun$apply$1.apply(RuleExecutor.scala:85)
      at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1$$anonfun$apply$1.apply(RuleExecutor.scala:82)
      at scala.collection.IndexedSeqOptimized$class.foldl(IndexedSeqOptimized.scala:57)
  </code></pre>
  <p>Currently I am planning to re-write my table, if someone has a better approach I will appreciate advice.</p>