title: How to set spark consumer cache ? to fix "KafkaConsumer cache hitting max capacity of 64" error?
body: <p>I am using spark-sql 2.4.1 , spark-cassandra-connector_2.11-2.4.1.jar and java8. While inserting data from kafka topic to  C*/Cassandra table data.</p>
  
  <p>I am getting error:</p>
  
  <pre><code> org.apache.spark.sql.kafka010.KafkaDataConsumer - KafkaConsumer cache hitting max capacity of 64, removing consumer for CacheKey(spark-kafka-source-33321dde-bfad-49f3-bdf7-09f95883b6e9--1249540122-executor)
  </code></pre>
  
  <p>How to fix this issue?</p>
  
  <blockquote>
    <p><strong>Section 2:</strong></p>
  </blockquote>
  
  <p>Me using below options</p>
  
  <pre><code>Dataset&lt;Row&gt; df = sparkSession
                        .readStream()
                        .format("kafka")
                        ///other options
                        .option("startingOffsets", "latest")
                        .option("retries", 1)
                        .option("linger.ms", 10)
                        .option("enable.auto.commit", false)
                        .option("failOnDataLoss", false)
                        .option("maxOffsetsPerTrigger", 500)
                     .option("spark.streaming.kafka.consumer.cache.enabled",false)
                        .load(); 
  </code></pre>
  
  <p>Still I am getting error:</p>
  
  <pre><code> org.apache.spark.sql.kafka010.KafkaDataConsumer - KafkaConsumer cache hitting max capacity of 64, removing consumer for CacheKey(spark-kafka-source-33321dde-bfad-49f3-bdf7-09f95883b6e9--1249540122-executor)
  </code></pre>