title: Kubernetes logs spamming Splunk
body: <p>I'm having issues with Kubernetes containers spamming Splunk with hundreds of gigabytes of logs sometimes. I would like to put together a search to track containers that have a sudden log spike and generate an alert. </p>
  
  <p>More specifically:</p>
  
  <ol>
  <li>look at the average rate of events </li>
  <li>find the peak </li>
  <li>decide a percentage of that peak </li>
  <li>and then trigger an alert when a container has breached the threshold.</li>
  </ol>
  
  <p>The closest I have come up with is the below search, which has an average rate and standard deviation of that rate by hour:</p>
  
  <pre><code>index="apps" sourcetype="kube"
  | bucket _time span=1h
  | stats count as CountByHour by _time, kubernetes.container_name
  | eventstats avg(CountByHour) as AvgByKCN stdev(CountByHour) as StDevByKCN by kubernetes.container_name
  </code></pre>