<p>I am incrementing a Datadog counter in python:</p>

<pre><code>from datadog import initialize
from datadog import ThreadStats
stats.increment('api.request_count', tags=['environment:' + environment])
</code></pre>

<p>And have set the metric type to "count" and the unit to "requests per none" in the metadata for the metric.</p>

<p>The code runs in a docker container on a kubernetes node in a Container Engine in Google Cloud... I have docker-dd-agent (<a href="https://github.com/DataDog/docker-dd-agent" rel="nofollow noreferrer">https://github.com/DataDog/docker-dd-agent</a>) running on each node.</p>

<p>I can move the container to any node and it logs around 200 requests per minute. But as soon as I scale it up and launch a second container, it only logs around 100 requests per minute. If I scale down to one container again, it spikes to 200 rpm again: </p>

<p><a href="https://i.stack.imgur.com/9dXzg.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/9dXzg.png" alt="enter image description here"></a></p>

<p>What could be causing the requests to drop or get overwritten from other pods?</p>

<p>Why not use dogstatsd instead of threadstats? If you're already running the dd-agent on your node in a way that's reachable by your containers, you can use the <code>datadog.statsd.increment()</code> method instead to send the metric over statsd to the agent, and from there it would get forwarded to your datadog account. </p>

<p>Dogstatsd has the benefits of being more straight-forward and of being somewhat easier to troublehsoot, at least with debug-level logging. Threadstats sometimes has the benefit of not requiring a dd-agent, but it does very little (if any) error logging, so makes it difficult to troubleshoot cases like these.</p>

<p>If you went the dogstatsd route, you'd use the following code:</p>

<pre><code>from datadog import initialize
from datadog import statsd
statsd.increment('api.request_count', tags=['environment:' + environment])
</code></pre>

<p>And from there you'd find your metric metadata with the "rate" type and with an interval of "10", and you could use the "as_count" function to translate the values to counts.</p>
