<p>I have a namespace where new short-lived pods (&lt; 1 minute) are created constantly by Apache Airflow. I want that all those new pods are annotated with <code>aws.amazon.com/cloudwatch-agent-ignore: true</code> automatically so that no CloudWatch metrics (container insights) are created for those pods.</p>
<p>I know that I can achieve that from airflow side with <a href="https://airflow.apache.org/docs/apache-airflow/stable/kubernetes.html#pod-mutation-hook" rel="nofollow noreferrer">pod mutation hook</a> but for the sake of the argument let's say that <strong>I have no control over the configuration of that airflow instance</strong>.</p>
<p>I have seen <code>MutatingAdmissionWebhook</code> and it seem that could do the trick, but it seems that it's considerable effort to set up. So I'm looking for a more of the shelf solution, I want to know if there is some &quot;standard&quot; admission controller  that can do this specific use case, without me having to deploy a web server and implement the api required by <code>MutatingAdmissionWebhook</code>.</p>
<p><strong>Is there any way to add that annotation from kubernetes side at pod creation time?</strong> The annotation must be there &quot;from the beginning&quot;, not added 5 seconds later,  otherwise the cwagent might pick it between the pod creation and the annotation being added.</p>

<p>To clarify I am posting community Wiki answer.</p>
<p>You had to use <a href="https://github.com/aws/amazon-cloudwatch-agent/blob/dd1be96164c2cd6226a33c8cf7ce10a7f29547cf/plugins/processors/k8sdecorator/stores/podstore.go#L31" rel="nofollow noreferrer"><code>aws.amazon.com/cloudwatch-agent-ignore: true</code></a> annotation.  This means the pod that has one, it will be ignored by <code>amazon-cloudwatch-agent</code> / <code>cwagent</code>.</p>
<p>Here is the excerpt of your solution how to add this annotation to Apache Airflow:</p>
<blockquote>
<p>(...) In order to force Apache Airflow to add the
<code>aws.amazon.com/cloudwatch-agent-ignore: true</code>  annotation to the task/worker pods and to the pods created by the <code>KubernetesPodOperator</code>  you will need to add the following to your helm  <code>values.yaml</code>  (assuming that you are using the &quot;official&quot; helm chart for airflow 2.2.3):</p>
</blockquote>
<pre class="lang-yaml prettyprint-override"><code>airflowPodAnnotations:
  aws.amazon.com/cloudwatch-agent-ignore: &quot;true&quot;
airflowLocalSettings: |-
  def pod_mutation_hook(pod): 
    pod.metadata.annotations[&quot;aws.amazon.com/cloudwatch-agent-ignore&quot;] = &quot;true&quot;
</code></pre>
<blockquote>
<p>If you are not using the helm chart then you will need to change the <code>pod_template_file</code>  yourself to add the  <code>annotation</code>  and you will also need to modify the  <code>airflow_local_settings.py</code>  to include the <code>pod_mutation_hook</code>.</p>
</blockquote>
<p><a href="https://stackoverflow.com/questions/71438495/how-to-prevent-cloudwatch-container-insights-metrics-from-short-lived-kubernetes/71438496#71438496">Here</a> is the link to your whole answer.</p>
