<p>I'm trying to install <strong>Datadog</strong> agent for a <strong>Kubernetes</strong> cluster using <strong>Helm</strong>.</p>
<p>This is the <strong>helm command</strong> I'm using for it:</p>
<pre><code>helm repo add datadog https://helm.datadoghq.com

helm repo update

helm upgrade --install datadog datadog/datadog \
  --namespace monitoring \
  --create-namespace \
  --atomic \
  --set datadog.apiKey=&lt;MY-DATADOG-API-KEY&gt; \
  --set targetSystem=linux \
  --values values.yaml
</code></pre>
<p><strong>Values file</strong>:</p>
<pre><code>datadog:
  kubelet:
    host:
      valueFrom:
        fieldRef:
          fieldPath: spec.nodeName
    hostCAPath: /etc/kubernetes/certs/kubeletserver.crt
    tlsVerify: false # Required as of Agent 7.35. See Notes.
</code></pre>
<p>However, when I run the <strong>liveness probe error with error 500</strong> which shows the error below:</p>
<blockquote>
<p>CLUSTER | ERROR | (pkg/forwarder/transaction/transaction.go:344 in internalProcess) | API Key invalid, dropping transaction for <a href="https://orchestrator.datadoghq.com/api/v1/orchestrator" rel="nofollow noreferrer">https://orchestrator.datadoghq.com/api/v1/orchestrator</a>.</p>
</blockquote>

<p><strong>Here's how I solved it</strong>:</p>
<p>The issue had to do with the <strong>Datadog Destination Site</strong>. The Destination site for my metrics, traces, and logs is supposed to be <code>datadoghq.eu</code>. This is set using the variable <code>DD_SITE</code>, and it defaults to <code>datadoghq.com</code> if it is not set.</p>
<p>To check what your <strong>Datadog Destination Site</strong> just look at the URL of your Datadog dashboard:</p>
<ul>
<li>For US it will be - <a href="https://app.datadoghq.com/" rel="noreferrer">https://app.datadoghq.com/</a></li>
<li>For EU it will be - <a href="https://app.datadoghq.eu/" rel="noreferrer">https://app.datadoghq.eu/</a></li>
</ul>
<p>To set this in your helm chart simply do either of the following:</p>
<pre><code>helm repo add datadog https://helm.datadoghq.com

helm repo update

helm upgrade --install datadog datadog/datadog \
  --namespace monitoring \
  --create-namespace \
  --atomic \
  --set datadog.apiKey=&lt;MY-DATADOG-API-KEY&gt; \
  --set targetSystem=linux \
  --set datadog.site=datadoghq.eu \
  --values values.yaml
</code></pre>
<p>OR set it in your values file:</p>
<pre><code>datadog:
  site: datadoghq.eu
  kubelet:
    host:
      valueFrom:
        fieldRef:
          fieldPath: spec.nodeName
    hostCAPath: /etc/kubernetes/certs/kubeletserver.crt
    tlsVerify: false # Required as of Agent 7.35. See Notes.
</code></pre>
<p><strong>References</strong>:</p>
<ol>
<li><p><a href="https://github.com/DataDog/datadog-agent/issues/1697" rel="noreferrer">Datadog Agent Forwarder fails liveness probe when new spot instance joins cluster, causing multiple restarts #1697</a></p>
</li>
<li><p><a href="https://github.com/DataDog/datadog-agent/issues/9180" rel="noreferrer">DD_SITE Set to us3.datadoghq.com, but process-agent and security-agent Still Try to Connect to non us3 endpoints #9180</a></p>
</li>
</ol>
