title: Cannot broadcast the table that is larger than 8GB : Spark 2.3
body: <p>I am trying to use the broadcast join but getting below error in Spark 2.3. However, same code is working fine in Spark 2.2.</p>
  
  <p>Upon checking the size of the dataframes its merely 50 MB and I have set the threshold to 200 MB as well. As I mentioned above same code is working fine in Spark 2.2</p>
  
  <pre><code>Error: "Cannot broadcast the table that is larger than 8GB".
  </code></pre>
  
  <p>However, Disabling the broadcasting is working fine.</p>
  
  <pre><code>'spark.sql.autoBroadcastJoinThreshold': '-1'
  </code></pre>
  
  <p>Could you please help here?</p>