title: Spark-Kafka Structured Streaming Producer: RecordTooLargeException
body: <p>I have been reading all similar posts in Stackoverflow and does not find the answer I need. Sorry if this is duplicated already.</p>
  <p>The exception is: &quot;<strong>org.apache.kafka.common.errors.RecordTooLargeException: The message is 1555146 bytes when serialized which is larger than the maximum request size you have configured with the max.request.size configuration</strong>.&quot;, when trying to write messages to Kafka.</p>
  <p>In my producer, I have set:</p>
  <pre><code>max.request.size = 1000000
  batch.size = 16384
  linger.ms = 0
  </code></pre>
  <p>The individual message I have been using the producer to send is <strong>around 4KB</strong>.I tried playing around batch.size but not working. And I think &quot;message.max.bytes&quot;, &quot;max.message.bytes&quot; are server side configs.</p>
  <p>My question is why Spark is still trying to construct a batched message larger than 1MB, with my given configuration? How could this be fixed?</p>
  <p>The issue is that I don't own the server so <strong>I am not able to change the server-side to accept larger messages.</strong> I see most of the other posts solution are suggesting altering server config.</p>