title: Spark Error - Max iterations (100) reached for batch Resolution
body: <p>I am working on Spark SQL where I need to find out Diff between two large CSV's.</p>
  
  <p>Diff should give:-</p>
  
  <ul>
  <li><p>Inserted Rows or new Record // Comparing only Id's </p></li>
  <li><p>Changed Rows (Not include inserted ones) - Comparing all column values</p></li>
  <li>Deleted rows // Comparing only Id's</li>
  </ul>
  
  <p>Spark 2.4.4 + Java</p>
  
  <p>I am using Databricks to Read/Write CSV</p>
  
  <pre><code>Dataset&lt;Row&gt; insertedDf = newDf_temp.join(oldDf_temp,oldDf_temp.col(key)
                  .equalTo(newDf_temp.col(key)),"left_anti");
  Long insertedCount = insertedDf.count();
  logger.info("Inserted File Count == "+insertedCount);
  
  
  Dataset&lt;Row&gt; deletedDf = oldDf_temp.join(newDf_temp,oldDf_temp.col(key)
                  .equalTo(newDf_temp.col(key)),"left_anti")
                  .select(oldDf_temp.col(key));
  Long deletedCount = deletedDf.count();
  logger.info("deleted File Count == "+deletedCount);
  
  
  Dataset&lt;Row&gt; changedDf = newDf_temp.exceptAll(oldDf_temp); // This gives rows (New +changed Records)
  
  Dataset&lt;Row&gt; changedDfTemp = changedDf.join(insertedDf, changedDf.col(key)
                  .equalTo(insertedDf.col(key)),"left_anti"); // This gives only changed record
  
  Long changedCount = changedDfTemp.count();
  logger.info("Changed File Count == "+changedCount);
  </code></pre>
  
  <p><em>This works well for CSV with columns upto 50 or so.</em></p>
  
  <p><code>The Above code fails for one row in CSV with 300+columns, so I am sure this is not file Size problem.</code></p>
  
  <p><strong>But if I have a CSV having 300+ Columns then it fails with Exception</strong></p>
  
  <p><strong>Max iterations (100) reached for batch Resolution – Spark Error</strong></p>
  
  <p><code>If I set the below property in Spark, It Works!!!</code></p>
  
  <p><strong>sparkConf.set("spark.sql.optimizer.maxIterations", "500");</strong></p>
  
  <p>But my question is why do I have to set this?</p>
  
  <p>Is there something wrong which I am doing?
  Or this behaviour is expected for CSV's which have large columns. </p>
  
  <p>Can I optimize it in any way to handle Large column CSV's.</p>