<p>I am trying to set up autoscaling on a Kubernetes 1.2.3 (beta) cluster based on custom metrics. (I already tried CPU-based autoscaling on the cluster, and it worked fine.)</p>

<p>I tried to follow their <a href="https://github.com/kubernetes/kubernetes/blob/master/docs/proposals/custom-metrics.md" rel="nofollow">custom metrics proposal</a>, but I'm having problems in creating the necessary set-up.</p>

<p>This is what I have done so far:</p>

<ol>
<li><p>Added a custom metrics annotation to the pod spec being deployed
(similar to the configuration provided in their proposal):</p>

<pre><code>apiVersion: v1
kind: ReplicationController
metadata:
  name: metrix
  namespace: "default"
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: metrix
      annotations:
        metrics.alpha.kubernetes.io/custom-endpoints: &gt;
          [
            {
              "api": "prometheus",
              "path": "/status",
              "port": "9090",
              "names": ["test1"]
            },
            {
              "api": "prometheus",
              "path": "/metrics",
              "port": "9090"
              "names": ["test2"]
            }
          ]
    spec:
      containers:
      - name: metrix
        image: janaka/prometheus-ep:v1
        resources:
          requests:
            cpu: 400m
</code></pre></li>
<li><p>Created a Docker container tagged <code>janaka/prometheus-ep:v1</code> (local) running a Prometheus-compatible server on port 9090, with <code>/status</code> and <code>/metrics</code> endpoints</p></li>
<li><p>Enabled custom metrics on the kubelet by appending <code>--enable-custom-metrics=true</code> to <code>KUBELET_OPTS</code> at <code>/etc/default/kubelet</code> (based on <a href="http://kubernetes.io/docs/admin/kubelet/" rel="nofollow">the kubelet CLI reference</a>) and restarted the kubelet</p></li>
</ol>

<p>All pods (in <code>default</code> and <code>kube-system</code> namespaces) are running, and the heapster pod log doesn't contain any 'anomalous' outputs either (except for a small glitch at startup, due to temporary unavailability of InfluxDB):</p>

<pre><code>$ kubesys logs -f heapster-daftr

I0427 05:07:45.807277       1 heapster.go:60] /heapster --source=kubernetes:https://kubernetes.default --sink=influxdb:http://monitoring-influxdb:8086
I0427 05:07:45.807359       1 heapster.go:61] Heapster version 1.1.0-beta1
I0427 05:07:45.807638       1 configs.go:60] Using Kubernetes client with master "https://kubernetes.default" and version "v1"
I0427 05:07:45.807661       1 configs.go:61] Using kubelet port 10255
E0427 05:08:15.847319       1 influxdb.go:185] issues while creating an InfluxDB sink: failed to ping InfluxDB server at "monitoring-influxdb:8086" - Get http://monitoring-influxdb:8086/ping: dial tcp xxx.xxx.xxx.xxx:8086: i/o timeout, will retry on use
I0427 05:08:15.847376       1 influxdb.go:199] created influxdb sink with options: host:monitoring-influxdb:8086 user:root db:k8s
I0427 05:08:15.847412       1 heapster.go:87] Starting with InfluxDB Sink
I0427 05:08:15.847427       1 heapster.go:87] Starting with Metric Sink
I0427 05:08:15.877349       1 heapster.go:166] Starting heapster on port 8082
I0427 05:08:35.000342       1 manager.go:79] Scraping metrics start: 2016-04-27 05:08:00 +0000 UTC, end: 2016-04-27 05:08:30 +0000 UTC
I0427 05:08:35.035800       1 manager.go:152] ScrapeMetrics: time: 35.209696ms size: 24
I0427 05:08:35.044674       1 influxdb.go:177] Created database "k8s" on influxDB server at "monitoring-influxdb:8086"
I0427 05:09:05.000441       1 manager.go:79] Scraping metrics start: 2016-04-27 05:08:30 +0000 UTC, end: 2016-04-27 05:09:00 +0000 UTC
I0427 05:09:06.682941       1 manager.go:152] ScrapeMetrics: time: 1.682157776s size: 24
I0427 06:43:38.767146       1 manager.go:79] Scraping metrics start: 2016-04-27 05:09:00 +0000 UTC, end: 2016-04-27 05:09:30 +0000 UTC
I0427 06:43:38.810243       1 manager.go:152] ScrapeMetrics: time: 42.940682ms size: 1
I0427 06:44:05.012989       1 manager.go:79] Scraping metrics start: 2016-04-27 06:43:30 +0000 UTC, end: 2016-04-27 06:44:00 +0000 UTC
I0427 06:44:05.063583       1 manager.go:152] ScrapeMetrics: time: 50.368106ms size: 24
I0427 06:44:35.002038       1 manager.go:79] Scraping metrics start: 2016-04-27 06:44:00 +0000 UTC, end: 2016-04-27 06:44:30 +0000 UTC
</code></pre>

<p>However, the custom endpoints are not being scraped. (I verified it by adding stderr logs for startup and endpoint handlers of my server; only the server initialization logs are displayed on kubectl logs of the pod.)</p>

<p>As I'm a newbie to Kubernetes, any help is appreciated.</p>

<p>(From what I understood from the proposal as well as <a href="https://github.com/kubernetes/kubernetes/issues/18352" rel="nofollow">this issue</a>, we don't have to run a separate Prometheus collector in the cluster as <strong>cAdvisor</strong> should already pull data from the endpoints defined in the pod spec. Is this true, or do I need a separate Prometheus collector as well?)</p>

<p>The <a href="https://github.com/mwielgus/kubernetes.github.io/blob/custom-metrics/docs/user-guide/horizontal-pod-autoscaling/index.md" rel="nofollow">custom metrics proposal</a> is out of date.</p>

<p>Please refer to the <a href="https://github.com/mwielgus/kubernetes.github.io/blob/custom-metrics/docs/user-guide/horizontal-pod-autoscaling/index.md#support-for-custom-metrics" rel="nofollow">user guide</a>, which is currently under review.</p>
