{"action":"created","issue":{"url":"https://api.github.com/repos/coreos/prometheus-operator/issues/867","repository_url":"https://api.github.com/repos/coreos/prometheus-operator","labels_url":"https://api.github.com/repos/coreos/prometheus-operator/issues/867/labels{/name}","comments_url":"https://api.github.com/repos/coreos/prometheus-operator/issues/867/comments","events_url":"https://api.github.com/repos/coreos/prometheus-operator/issues/867/events","html_url":"https://github.com/coreos/prometheus-operator/issues/867","id":286872008,"node_id":"MDU6SXNzdWUyODY4NzIwMDg=","number":867,"title":"kube-prmetheus kubelet exporter target forbidden","user":{"login":"luisyonaldo","id":6656325,"node_id":"MDQ6VXNlcjY2NTYzMjU=","avatar_url":"https://avatars1.githubusercontent.com/u/6656325?v=4","gravatar_id":"","url":"https://api.github.com/users/luisyonaldo","html_url":"https://github.com/luisyonaldo","followers_url":"https://api.github.com/users/luisyonaldo/followers","following_url":"https://api.github.com/users/luisyonaldo/following{/other_user}","gists_url":"https://api.github.com/users/luisyonaldo/gists{/gist_id}","starred_url":"https://api.github.com/users/luisyonaldo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/luisyonaldo/subscriptions","organizations_url":"https://api.github.com/users/luisyonaldo/orgs","repos_url":"https://api.github.com/users/luisyonaldo/repos","events_url":"https://api.github.com/users/luisyonaldo/events{/privacy}","received_events_url":"https://api.github.com/users/luisyonaldo/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":28,"created_at":"2018-01-08T20:20:44Z","updated_at":"2018-08-16T15:18:46Z","closed_at":"2018-01-09T13:06:26Z","author_association":"CONTRIBUTOR","body":"**What did you do?**\r\n```\r\nhelm install coreos/prometheus-operator --name prometheus-operator --namespace=monitoring\r\nhelm install coreos/kube-prometheus --name kube-prometheus --namespace=monitoring\r\n```\r\n\r\n**What did you expect to see?**\r\nAll prometheus exporter targets on prometheus dashboard\r\n\r\n**What did you see instead? Under which circumstances?**\r\ntarget on prometheus dashboard: `server returned HTTP status 403 Forbidden`\r\n\r\n**Environment**\r\nRedHat\r\n\r\n* Kubernetes version information:\r\n```\r\nClient Version: version.Info{Major:\"1\", Minor:\"8\", GitVersion:\"v1.8.1+coreos.0\", GitCommit:\"59359d9fdce74738ac9a672d2f31e9a346c5cece\", GitTreeState:\"clean\", BuildDate:\"2017-10-12T21:53:13Z\", GoVersion:\"go1.8.3\", Compiler:\"gc\", Platform:\"linux/amd64\"}\r\nServer Version: version.Info{Major:\"1\", Minor:\"8\", GitVersion:\"v1.8.1+coreos.0\", GitCommit:\"59359d9fdce74738ac9a672d2f31e9a346c5cece\", GitTreeState:\"clean\", BuildDate:\"2017-10-12T21:53:13Z\", GoVersion:\"go1.8.3\", Compiler:\"gc\", Platform:\"linux/amd64\"}\r\n```\r\n\r\n* Kubernetes cluster kind: \r\n\r\n\tkube-spray/kube-admin\r\n\r\n* Manifests:\r\nKubelet Service Monitor\r\n```\r\nkubectl get ServiceMonitor -n monitoring kube-prometheus-exporter-kubelets -oyaml\r\napiVersion: monitoring.coreos.com/v1\r\nkind: ServiceMonitor\r\nmetadata:\r\n  clusterName: \"\"\r\n  creationTimestamp: 2018-01-08T14:11:35Z\r\n  deletionGracePeriodSeconds: null\r\n  deletionTimestamp: null\r\n  generation: 0\r\n  labels:\r\n    chart: exporter-kubelets-0.1.2\r\n    component: kubelets\r\n    heritage: Tiller\r\n    prometheus: kube-prometheus\r\n    release: kube-prometheus\r\n  name: kube-prometheus-exporter-kubelets\r\n  namespace: monitoring\r\n  resourceVersion: \"66465\"\r\n  selfLink: /apis/monitoring.coreos.com/v1/namespaces/monitoring/servicemonitors/kube-prometheus-exporter-kubelets\r\n  uid: d5b50f43-f47d-11e7-9293-005056a4032b\r\nspec:\r\n  endpoints:\r\n  - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token\r\n    interval: 15s\r\n    port: https-metrics\r\n    scheme: https\r\n    tlsConfig:\r\n      caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt\r\n      insecureSkipVerify: true\r\n  - honorLabels: true\r\n    interval: 30s\r\n    port: cadvisor\r\n  jobLabel: kube-prometheus-exporter-kubelets\r\n  namespaceSelector:\r\n    matchNames:\r\n    - kube-system\r\n  selector:\r\n    matchLabels:\r\n      k8s-app: kubelet\r\n```\r\n\r\n* Prometheus Operator Logs:\r\nKubelet\r\n```\r\nJan 08 13:41:27 tapps773 kubelet[54973]: goroutine 27204 [running]:\r\nJan 08 13:41:27 tapps773 kubelet[54973]: k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog.(*respLogger).recordStatus(0xc4205ea620, 0x193)\r\nJan 08 13:41:27 tapps773 kubelet[54973]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:207 +0xdd\r\nJan 08 13:41:27 tapps773 kubelet[54973]: k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog.(*respLogger).WriteHeader(0xc4205ea620, 0x193)\r\nJan 08 13:41:27 tapps773 kubelet[54973]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:186 +0x35\r\nJan 08 13:41:27 tapps773 kubelet[54973]: k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.(*Response).WriteHeader(0xc420cc80c0, 0x193)\r\nJan 08 13:41:27 tapps773 kubelet[54973]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/response.go:201 +0x41\r\nJan 08 13:41:27 tapps773 kubelet[54973]: k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.(*Response).WriteErrorString(0xc420cc80c0, 0x193, 0xc4228c4050, 0x50, 0x4, 0xc422\r\nJan 08 13:41:27 tapps773 kubelet[54973]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/response.go:181 +0x46\r\nJan 08 13:41:27 tapps773 kubelet[54973]: k8s.io/kubernetes/pkg/kubelet/server.(*Server).InstallAuthFilter.func1(0xc422ed30e0, 0xc420cc80c0, 0xc422ed3020)\r\nJan 08 13:41:27 tapps773 kubelet[54973]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/server/server.go:246 +0x4fa\r\nJan 08 13:41:27 tapps773 kubelet[54973]: k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.(*FilterChain).ProcessFilter(0xc422ed3020, 0xc422ed30e0, 0xc420cc80c0)\r\nJan 08 13:41:27 tapps773 kubelet[54973]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/filter.go:19 +0x68\r\nJan 08 13:41:27 tapps773 kubelet[54973]: k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.(*Container).HandleWithFilter.func1(0x9a9d760, 0xc4205ea620, 0xc42304c500)\r\nJan 08 13:41:27 tapps773 kubelet[54973]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/container.go:313 +0x355\r\nJan 08 13:41:27 tapps773 kubelet[54973]: net/http.HandlerFunc.ServeHTTP(0xc420b59c60, 0x9a9d760, 0xc4205ea620, 0xc42304c500)\r\nJan 08 13:41:27 tapps773 kubelet[54973]: /usr/local/go/src/net/http/server.go:1942 +0x44\r\nJan 08 13:41:27 tapps773 kubelet[54973]: net/http.(*ServeMux).ServeHTTP(0xc4207df590, 0x9a9d760, 0xc4205ea620, 0xc42304c500)\r\nJan 08 13:41:27 tapps773 kubelet[54973]: /usr/local/go/src/net/http/server.go:2238 +0x130\r\nJan 08 13:41:27 tapps773 kubelet[54973]: k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.(*Container).ServeHTTP(0xc4211fb830, 0x9a9d760, 0xc4205ea620, 0xc42304c500)\r\nJan 08 13:41:27 tapps773 kubelet[54973]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/container.go:292 +0x4d\r\nJan 08 13:41:27 tapps773 kubelet[54973]: k8s.io/kubernetes/pkg/kubelet/server.(*Server).ServeHTTP(0xc420aaa9b0, 0x9a9d760, 0xc4205ea620, 0xc42304c500)\r\nJan 08 13:41:27 tapps773 kubelet[54973]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/server/server.go:778 +0x110\r\nJan 08 13:41:27 tapps773 kubelet[54973]: net/http.serverHandler.ServeHTTP(0xc4205a2b00, 0x9a9e9e0, 0xc420791960, 0xc42304c500)\r\nJan 08 13:41:27 tapps773 kubelet[54973]: /usr/local/go/src/net/http/server.go:2568 +0x92\r\nJan 08 13:41:27 tapps773 kubelet[54973]: net/http.(*conn).serve(0xc421ab2140, 0x9aa2160, 0xc4213ec240)\r\nJan 08 13:41:27 tapps773 kubelet[54973]: /usr/local/go/src/net/http/server.go:1825 +0x612\r\nJan 08 13:41:27 tapps773 kubelet[54973]: created by net/http.(*Server).Serve\r\nJan 08 13:41:27 tapps773 kubelet[54973]: /usr/local/go/src/net/http/server.go:2668 +0x2ce\r\nJan 08 13:41:27 tapps773 kubelet[54973]: logging error output: \"Forbidden (user=system:anonymous, verb=get, resource=nodes, subresource=metrics)\"\r\nJan 08 13:41:27 tapps773 kubelet[54973]: [[Prometheus/2.0.0] 10.94.130.14:33920]\r\nJan 08 13:41:29 tapps773 kubelet[54973]: E0108 13:41:29.322199   54973 helpers.go:468] PercpuUsage had 0 cpus, but the actual number is 4; ignoring extra CPUs\r\nJan 08 13:41:42 tapps773 kubelet[54973]: I0108 13:41:42.027127   54973 server.go:245] Forbidden (user=system:anonymous, verb=get, resource=nodes, subresource=metrics)\r\nJan 08 13:41:42 tapps773 kubelet[54973]: I0108 13:41:42.027206   54973 server.go:779] GET /metrics: (123.477µs) 403\r\n```\r\n\r\nNote: api server is able to communicate with same config. Also, it works if I change ServiceMonitor to use `http` instead of `https`.\r\n"},"comment":{"url":"https://api.github.com/repos/coreos/prometheus-operator/issues/comments/413582679","html_url":"https://github.com/coreos/prometheus-operator/issues/867#issuecomment-413582679","issue_url":"https://api.github.com/repos/coreos/prometheus-operator/issues/867","id":413582679,"node_id":"MDEyOklzc3VlQ29tbWVudDQxMzU4MjY3OQ==","user":{"login":"brancz","id":4546722,"node_id":"MDQ6VXNlcjQ1NDY3MjI=","avatar_url":"https://avatars3.githubusercontent.com/u/4546722?v=4","gravatar_id":"","url":"https://api.github.com/users/brancz","html_url":"https://github.com/brancz","followers_url":"https://api.github.com/users/brancz/followers","following_url":"https://api.github.com/users/brancz/following{/other_user}","gists_url":"https://api.github.com/users/brancz/gists{/gist_id}","starred_url":"https://api.github.com/users/brancz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/brancz/subscriptions","organizations_url":"https://api.github.com/users/brancz/orgs","repos_url":"https://api.github.com/users/brancz/repos","events_url":"https://api.github.com/users/brancz/events{/privacy}","received_events_url":"https://api.github.com/users/brancz/received_events","type":"User","site_admin":false},"created_at":"2018-08-16T15:18:46Z","updated_at":"2018-08-16T15:18:46Z","author_association":"MEMBER","body":"1) Yes, x509 TLS client cert authentication is always available, if you are able to create and manage (rotate etc.) the certificates properly, I would even encourage you to use this over token authentication, but it turns out all management aspects are much easier for most users, but if you and your organization is capable of running a proper PKI, then go for it :slightly_smiling_face:.\r\n2) Yes they could, as you would only have authentication turned on, but not authorization.\r\n3) This means that the kubelet will ask the Kubernetes API whether the requesting entity has permissions to request the respective endpoint (metrics in our case). Enabling this means the kubelet will check on any request also for those with TLS certificates not just for tokens (but you will need to grant access to a \"user\" not a serviceaccount as that is what the kubelet interprets the common name of a client certificate as. Even if you are using TLS I recommend turning this on, as otherwise that means that the Prometheus server could do all actions on the kubelet, which really only the Kubernetes API should ever do."}}
{"actor":{"display_login":"brancz"}}