title: Data still remains in Kafka topic even after retention time/size
body: <p>We set the <code>log retention hours</code> to 1 hour as the following (previously setting was 72H)</p>
  <p>Using the following Kafka command line tool, we set the kafka <code>retention.ms</code> to <code>1H</code>. Our aim is to purge the data that is older then 1H in topic - <code>test_topic</code>, so we used the following command:</p>
  <pre class="lang-sh prettyprint-override"><code>kafka-configs.sh --alter \
    --zookeeper localhost:2181  \
    --entity-type topics \
    --entity-name topic_test \
    --add-config retention.ms=3600000
  </code></pre>
  <p>and also</p>
  <pre class="lang-sh prettyprint-override"><code>kafka-topics.sh --zookeeper localhost:2181 --alter \
    --topic topic_test \
    --config retention.ms=3600000
  </code></pre>
  <p>Both commands ran without errors.</p>
  <p>But the problem is about Kafka data that is older then 1H and still remains!</p>
  <p>Actually no data was removed from the topic <code>topic_test</code> partitions. We have HDP Kafka cluster version 1.0x and ambari</p>
  <p>We do not understand why data on topic - <code>topic_test</code> still remained? and not decreased even after we run both cli as already described</p>
  <p>what is wrong on the following kafka cli?</p>
  <pre><code>kafka-configs.sh --alter --zookeeper localhost:2181  --entity-type topics  --entity-name topic_test --add-config retention.ms=3600000
  
  kafka-topics.sh --zookeeper localhost:2181 --alter --topic topic_test --config retention.ms=3600000
  </code></pre>
  <p>from the Kafka <code>server.log</code> we ca see the following</p>
  <pre><code>2020-07-28 14:47:27,394] INFO Processing override for entityPath: topics/topic_test with config: Map(retention.bytes -&gt; 2165441552, retention.ms -&gt; 3600000) (kafka.server.DynamicConfigManager)
  [2020-07-28 14:47:27,397] WARN retention.ms for topic topic_test is set to 3600000. It is smaller than message.timestamp.difference.max.ms's value 9223372036854775807. This may result in frequent log rolling. (kafka.server.TopicConfigHandler)
  </code></pre>
  <p>reference - <a href="https://ronnieroller.com/kafka/cheat-sheet" rel="nofollow noreferrer">https://ronnieroller.com/kafka/cheat-sheet</a></p>