{"action":"created","issue":{"url":"https://api.github.com/repos/luxas/kubeadm-workshop/issues/33","repository_url":"https://api.github.com/repos/luxas/kubeadm-workshop","labels_url":"https://api.github.com/repos/luxas/kubeadm-workshop/issues/33/labels{/name}","comments_url":"https://api.github.com/repos/luxas/kubeadm-workshop/issues/33/comments","events_url":"https://api.github.com/repos/luxas/kubeadm-workshop/issues/33/events","html_url":"https://github.com/luxas/kubeadm-workshop/issues/33","id":277958006,"node_id":"MDU6SXNzdWUyNzc5NTgwMDY=","number":33,"title":"custom autoscaler not working ","user":{"login":"manishrajkarnikar","id":4430507,"node_id":"MDQ6VXNlcjQ0MzA1MDc=","avatar_url":"https://avatars1.githubusercontent.com/u/4430507?v=4","gravatar_id":"","url":"https://api.github.com/users/manishrajkarnikar","html_url":"https://github.com/manishrajkarnikar","followers_url":"https://api.github.com/users/manishrajkarnikar/followers","following_url":"https://api.github.com/users/manishrajkarnikar/following{/other_user}","gists_url":"https://api.github.com/users/manishrajkarnikar/gists{/gist_id}","starred_url":"https://api.github.com/users/manishrajkarnikar/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/manishrajkarnikar/subscriptions","organizations_url":"https://api.github.com/users/manishrajkarnikar/orgs","repos_url":"https://api.github.com/users/manishrajkarnikar/repos","events_url":"https://api.github.com/users/manishrajkarnikar/events{/privacy}","received_events_url":"https://api.github.com/users/manishrajkarnikar/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":5,"created_at":"2017-11-30T00:34:32Z","updated_at":"2018-10-10T09:36:04Z","closed_at":null,"author_association":"NONE","body":"```\r\n✗ kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1/namespaces/default/services/sample-metrics-app/http_requests | jq\r\n{\r\n  \"kind\": \"MetricValueList\",\r\n  \"apiVersion\": \"custom.metrics.k8s.io/v1beta1\",\r\n  \"metadata\": {\r\n    \"selfLink\": \"/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/services/sample-metrics-app/http_requests\"\r\n  },\r\n  \"items\": [\r\n    {\r\n      \"describedObject\": {\r\n        \"kind\": \"Service\",\r\n        \"name\": \"sample-metrics-app\",\r\n        \"apiVersion\": \"/__internal\"\r\n      },\r\n      \"metricName\": \"http_requests\",\r\n      \"timestamp\": \"2017-11-29T21:31:13Z\",\r\n      \"value\": \"539m\"\r\n    }\r\n  ]\r\n}\r\n```\r\n\r\nNote that http_requests has value of \"539m\". It should be a number and not memory. In my case I was expecting 120 (that was my transaction per second i had set via my load test tool). Due to this issue hpa does not scale the replicaset appropriately. Prometheus metrics was verified to show right value. \r\n\r\n```\r\nI1129 21:27:07.806751       1 api.go:74] GET http://sample-metrics-prom.default.svc:9090/api/v1/query?query=sum%28rate%28http_requests_total%7Bservice%3D%22sample-metrics-app%22%2Cnamespace%3D%22default%22%7D%5B\r\n1m%5D%29%29+by+%28service%29&time=1511990827.805 200 OK\r\nI1129 21:27:07.806847       1 api.go:93] Response Body: {\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"service\":\"sample-metrics-app\"},\"value\":[1511990827.805,\"30.54\"]}]}}\r\n```\r\n\r\nThis is what describing the hpa shows.\r\n\r\n```\r\n➜  kubectl describe hpa sample-metrics-app-hpa\r\nName:                                             sample-metrics-app-hpa\r\nNamespace:                                        default\r\nLabels:                                           <none>\r\nAnnotations:                                      kubectl.kubernetes.io/last-applied-configuration={\"apiVersion\":\"autoscaling/v2beta1\",\"kind\":\"HorizontalPodAutoscaler\",\"metadata\":{\"annotations\":{},\"name\":\"sample-metrics-app-hpa\",\"namespace\":\"default\"...\r\nCreationTimestamp:                                Wed, 29 Nov 2017 21:29:58 +0000\r\nReference:                                        Deployment/sample-metrics-app\r\nMetrics:                                          ( current / target )\r\n  \"http_requests\" on Service/sample-metrics-app:  125300m / 100\r\nMin replicas:                                     1\r\nMax replicas:                                     10\r\nConditions:\r\n  Type            Status  Reason            Message\r\n  ----            ------  ------            -------\r\n  AbleToScale     True    ReadyForNewScale  the last scale time was sufficiently old as to warrant a new scale\r\n  ScalingActive   True    ValidMetricFound  the HPA was able to succesfully calculate a replica count from Service metric http_requests\r\n  ScalingLimited  True    TooManyReplicas   the desired replica count was more than the maximum replica count\r\nEvents:\r\n  Type     Reason                        Age              From                       Message\r\n  ----     ------                        ----             ----                       -------\r\n  Warning  FailedGetObjectMetric         8m (x9 over 2h)  horizontal-pod-autoscaler  unable to get metric http_requests: Service on default sample-metrics-app/unable to fetch metrics from API: the server could not find the requested resource (get services.custom.metrics.k8s.io sample-metrics-app)\r\n  Warning  FailedComputeMetricsReplicas  8m (x9 over 2h)  horizontal-pod-autoscaler  failed to get object metric value: unable to get metric http_requests: Service on default sample-metrics-app/unable to fetch metrics from API: the server could not find the requested resource (get services.custom.metrics.k8s.io sample-metrics-app)\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/luxas/kubeadm-workshop/issues/comments/428505672","html_url":"https://github.com/luxas/kubeadm-workshop/issues/33#issuecomment-428505672","issue_url":"https://api.github.com/repos/luxas/kubeadm-workshop/issues/33","id":428505672,"node_id":"MDEyOklzc3VlQ29tbWVudDQyODUwNTY3Mg==","user":{"login":"rickslick","id":1189101,"node_id":"MDQ6VXNlcjExODkxMDE=","avatar_url":"https://avatars2.githubusercontent.com/u/1189101?v=4","gravatar_id":"","url":"https://api.github.com/users/rickslick","html_url":"https://github.com/rickslick","followers_url":"https://api.github.com/users/rickslick/followers","following_url":"https://api.github.com/users/rickslick/following{/other_user}","gists_url":"https://api.github.com/users/rickslick/gists{/gist_id}","starred_url":"https://api.github.com/users/rickslick/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rickslick/subscriptions","organizations_url":"https://api.github.com/users/rickslick/orgs","repos_url":"https://api.github.com/users/rickslick/repos","events_url":"https://api.github.com/users/rickslick/events{/privacy}","received_events_url":"https://api.github.com/users/rickslick/received_events","type":"User","site_admin":false},"created_at":"2018-10-10T09:36:04Z","updated_at":"2018-10-10T09:36:04Z","author_association":"NONE","body":"@zihaoyu  @huxiaoliang  I have found a work around. As @zihaoyu  mentioned the server app is giving out http_requests_total while the prometheus adapter is looking out for http_request(figuring out why ) , So what we did is that we modified the application (https://github.com/luxas/kubeadm-workshop/blob/master/images/autoscaling/server.js ) to use http_request  and redeployed the container and autoscaling started working "}}
{"actor":{"display_login":"rickslick"}}