<p>I'm trying to enable AWS EKS autoscaling based on a custom Cloudwatch metric via the Kubernetes Cloudwatch adapter. I have pushed custom metrics to AWS Cloudwatch, and validated they appear in Cloudwatch console as well as are retrievable using the boto3 client get_metric_data. This is the code I use to publish my custom metric to Cloudwatch:</p>
<pre><code>import boto3
from datetime import datetime

client = boto3.client('cloudwatch')

cloudwatch_response = client.put_metric_data(
    Namespace='TestMetricNS',
    MetricData=[
        {
            'MetricName': 'TotalUnprocessed',
            'Timestamp': datetime.now(),
            'Value': 40,
            'Unit': 'Megabytes',
        }
    ]
)
</code></pre>
<p>I have the following yaml files for establishing the external metric and the hpa autoscaler in kubernetes:</p>
<p>extMetricCustom.yaml:</p>
<pre><code>apiVersion: metrics.aws/v1alpha1
kind: ExternalMetric
metadata:
  name: test-custom-metric
spec:
  name: test-custom-metric
  resource:
    resource: &quot;deployment&quot;
  queries:
    - id: sqs_test
      metricStat:
        metric:
          namespace: &quot;TestMetricNS&quot;
          metricName: &quot;TotalUnprocessed&quot;
        period: 60
        stat: Average
        unit: Megabytes
      returnData: true
</code></pre>
<p>hpaCustomMetric.yaml</p>
<pre><code>kind: HorizontalPodAutoscaler
apiVersion: autoscaling/v2beta1
metadata:
  name: test-scaler
spec:
  scaleTargetRef:
    apiVersion: apps/v1beta1
    kind: Deployment
    name: sqs-consumer
  minReplicas: 1
  maxReplicas: 4
  metrics:
  - type: External
    external:
      metricName: test-custom-metric
      targetAverageValue: 2
</code></pre>
<p>When I assess whether the Kubernetes Cloudwatch adapter is properly grabbing my custom metric (kubectl get hpa), it always displays that the metric is 0:</p>
<pre><code>NAME          REFERENCE                 TARGETS     MINPODS   MAXPODS   REPLICAS   AGE
test-scaler   Deployment/sqs-consumer   0/2 (avg)   1         4         1          161m
</code></pre>
<p>How can I properly autoscale based off my Cloudwatch custom metric?</p>

<p>Worked with OP on this out-of-band and still had the tab open for this question later in the day, so posting the outcome here for posterity for anyone that stumbles upon it.</p>
<p>The root cause of the issue was a timezone conflict. The metrics monitor was based on &quot;current&quot; metrics, but the the following line from the metric generator script was producing time stamps without a timezone specified and also was in a local timezone.</p>
<pre class="lang-py prettyprint-override"><code>            'Timestamp': datetime.now(),
</code></pre>
<p>Since there was &quot;no data&quot; for the current timezone (only data X hours in the past due to a -X UTC offset), the system did not initiate scaling because there was a value of &quot;0&quot;/nil/null effectively. Instead, a UTC time string can be specified to ensure the generated metrics are timely:</p>
<pre class="lang-py prettyprint-override"><code>            'Timestamp': datetime.utcnow(),
</code></pre>
<p>A secondary consideration was that the Kubernetes Nodes need access to poll the metrics from CloudWatch. This is done by attaching <a href="https://aws.amazon.com/blogs/compute/scaling-kubernetes-deployments-with-amazon-cloudwatch-metrics/" rel="nofollow noreferrer">this policy</a> to the nodes's IAM role:</p>
<pre class="lang-json prettyprint-override"><code>{
    &quot;Version&quot;: &quot;2012-10-17&quot;,
    &quot;Statement&quot;: [
        {
            &quot;Effect&quot;: &quot;Allow&quot;,
            &quot;Action&quot;: [
                &quot;cloudwatch:GetMetricData&quot;
            ],
            &quot;Resource&quot;: &quot;*&quot;
        }
    ]
}
</code></pre>
