<pre><code>  api.name: spark-history-server
  file.upload.path: x
  gcp.server.property.file.path: x
  git.files.update.path: x
  onprem.server.property.file.path: x
  preferred.id.deployment.file.path: x
  preferred.id.file.path: x
  server.error.whitelabel.enabled: &quot;false&quot;
  server.port: &quot;18080&quot;
  server.property.file.path: x
  server.servlet.context-path: /
  spark.history.fs.cleaner.enabled: &quot;true&quot;
  spark.history.fs.cleaner.interval: &quot;1h&quot;
  spark.history.fs.cleaner.maxAge: &quot;12h&quot;
  spring.thymeleaf.prefix: classpath:/templates/dev/
  spring.thymeleaf.view-names: index,devForm,error
  temp.repo.location: x
</code></pre>
<p>I am trying to clear my spark history server logs which I have deployed in Kubernetes using these three parameters as mentioned, I found the answer here <a href="https://stackoverflow.com/questions/42817924/cleaning-up-spark-history-logs">Cleaning up Spark history logs</a></p>
<p>it works when I restart the pods manually and deletes logs older than 12 hours but with time it starts pickingup old logs again and spark history server takes 1-2 hours to restart, is there another way I can do this so I don't have to manually restart the pods with time.</p>
<p>I asked around and found that it may be because I am using a shared starage like nfs.</p>

<p>The problem was that I was trying to add these parameters in Configmap.yaml file instead of Deployment.yaml file. Just add these paramters in SPARK_HISTORY_OPTS.</p>
<h1 id="example-ipk8">Example</h1>
<ul>
<li><p>name: SPARK_HISTORY_OPTS</p>
<p>value: &quot;-Dspark.history.fs.logDirectory=/FS/YOU/CREATED/ABOVE -Dspark.history.fs.cleaner.enabled=true -Dspark.history.fs.cleaner.interval=1d -Dspark.history.fs.cleaner.maxAge=7d&quot;</p>
</li>
</ul>
<p>This article helped me
<a href="https://wbassler23.medium.com/spark-history-server-on-dc-os-516fb71523a5" rel="nofollow noreferrer">https://wbassler23.medium.com/spark-history-server-on-dc-os-516fb71523a5</a></p>
