{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/38716","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/38716/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/38716/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/38716/events","html_url":"https://github.com/kubernetes/kubernetes/issues/38716","id":195329789,"number":38716,"title":" API Server Panic due to .... something something audit logs","user":{"login":"SleepyBrett","id":279521,"avatar_url":"https://avatars.githubusercontent.com/u/279521?v=3","gravatar_id":"","url":"https://api.github.com/users/SleepyBrett","html_url":"https://github.com/SleepyBrett","followers_url":"https://api.github.com/users/SleepyBrett/followers","following_url":"https://api.github.com/users/SleepyBrett/following{/other_user}","gists_url":"https://api.github.com/users/SleepyBrett/gists{/gist_id}","starred_url":"https://api.github.com/users/SleepyBrett/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SleepyBrett/subscriptions","organizations_url":"https://api.github.com/users/SleepyBrett/orgs","repos_url":"https://api.github.com/users/SleepyBrett/repos","events_url":"https://api.github.com/users/SleepyBrett/events{/privacy}","received_events_url":"https://api.github.com/users/SleepyBrett/received_events","type":"User","site_admin":false},"labels":[{"id":114528068,"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/priority/critical-urgent","name":"priority/critical-urgent","color":"e11d21","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/kubernetes/kubernetes/milestones/23","html_url":"https://github.com/kubernetes/kubernetes/milestone/23","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/milestones/23/labels","id":1945978,"number":23,"title":"v1.5","description":"","creator":{"login":"lavalamp","id":647318,"avatar_url":"https://avatars.githubusercontent.com/u/647318?v=3","gravatar_id":"","url":"https://api.github.com/users/lavalamp","html_url":"https://github.com/lavalamp","followers_url":"https://api.github.com/users/lavalamp/followers","following_url":"https://api.github.com/users/lavalamp/following{/other_user}","gists_url":"https://api.github.com/users/lavalamp/gists{/gist_id}","starred_url":"https://api.github.com/users/lavalamp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lavalamp/subscriptions","organizations_url":"https://api.github.com/users/lavalamp/orgs","repos_url":"https://api.github.com/users/lavalamp/repos","events_url":"https://api.github.com/users/lavalamp/events{/privacy}","received_events_url":"https://api.github.com/users/lavalamp/received_events","type":"User","site_admin":false},"open_issues":82,"closed_issues":1027,"state":"open","created_at":"2016-08-16T21:47:28Z","updated_at":"2016-12-13T18:23:37Z","due_on":"2016-12-09T00:00:00Z","closed_at":null},"comments":1,"created_at":"2016-12-13T18:22:07Z","updated_at":"2016-12-13T18:23:59Z","closed_at":null,"body":"**Kubernetes version** (use `kubectl version`):\r\nUnable to run kubectl version, but it's a 1.5.0 cluster.\r\n\r\n**Environment**:\r\n- AWS\r\n- OS\r\nNAME=CoreOS\r\nID=coreos\r\nVERSION=1235.2.0\r\nVERSION_ID=1235.2.0\r\nBUILD_ID=2016-12-07-0039\r\nPRETTY_NAME=\"CoreOS 1235.2.0 (MoreOS)\"\r\nANSI_COLOR=\"1;32\"\r\nHOME_URL=\"https://coreos.com/\"\r\nBUG_REPORT_URL=\"https://github.com/coreos/bugs/issues\"\r\n- **Kernel** (e.g. `uname -a`):\r\ncore@ip-172-24-240-20 /etc/kubernetes/manifests $ uname -a\r\nLinux ip-172-24-240-20.cloud.nordstrom.net 4.8.6-coreos-r1 #1 SMP Wed Dec 7 00:29:42 UTC 2016 x86_64 Intel(R) Xeon(R) CPU E5-2676 v3 @ 2.40GHz GenuineIntel GNU/Linux\r\n\r\n**What happened**:\r\nApi server crashes like so ...\r\n```\r\nI1213 18:15:04.798098       1 serve.go:93] Serving securely on 0.0.0.0:443\r\nI1213 18:15:04.798272       1 serve.go:107] Serving insecurely on 127.0.0.1:8080\r\nE1213 18:15:04.799460       1 runtime.go:64] Observed a panic: \"invalid memory address or nil pointer dereference\" (runtime error: invalid memory address or nil pointer dereference)\r\n/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/runtime/runtime.go:70\r\n/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/runtime/runtime.go:63\r\n/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/runtime/runtime.go:49\r\n/usr/local/go/src/runtime/asm_amd64.s:479\r\n/usr/local/go/src/runtime/panic.go:458\r\n/usr/local/go/src/runtime/panic.go:62\r\n/usr/local/go/src/runtime/sigpanic_unix.go:24\r\n/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/filters/audit.go:119\r\n/usr/local/go/src/net/http/server.go:1726\r\n/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/genericapiserver/filters/panics.go:75\r\n/usr/local/go/src/net/http/server.go:1726\r\n/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/filters/requestinfo.go:45\r\n/usr/local/go/src/net/http/server.go:1726\r\n/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/api/requestcontext.go:107\r\n/usr/local/go/src/net/http/server.go:1726\r\n/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/genericapiserver/filters/timeout.go:78\r\n/usr/local/go/src/runtime/asm_amd64.s:2086\r\nE1213 18:15:04.799738       1 panics.go:37] APIServer panic'd on GET /apis/extensions/v1beta1/thirdpartyresources: runtime error: invalid memory address or nil pointer dereference\r\ngoroutine 675 [running]:\r\nruntime/debug.Stack(0x5ace780, 0xc4213d3500, 0x34d3646)\r\n\t/usr/local/go/src/runtime/debug/stack.go:24 +0x79\r\nk8s.io/kubernetes/pkg/genericapiserver/filters.WithPanicRecovery.func1.1(0x2e640e0, 0xc420018040)\r\n\t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/genericapiserver/filters/panics.go:37 +0x74\r\nk8s.io/kubernetes/pkg/util/runtime.HandleCrash(0xc421563da0, 0x1, 0x1)\r\n\t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/runtime/runtime.go:52 +0xe5\r\npanic(0x2e640e0, 0xc420018040)\r\n\t/usr/local/go/src/runtime/panic.go:458 +0x243\r\nk8s.io/kubernetes/pkg/apiserver/filters.WithAudit.func1(0x5ace780, 0xc4213d3500, 0xc420c390e0)\r\n\t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/filters/audit.go:119 +0x49d\r\nnet/http.HandlerFunc.ServeHTTP(0xc420800000, 0x5ace780, 0xc4213d3500, 0xc420c390e0)\r\n\t/usr/local/go/src/net/http/server.go:1726 +0x44\r\nk8s.io/kubernetes/pkg/genericapiserver/filters.WithPanicRecovery.func1(0x5ace780, 0xc4213d3500, 0xc420c390e0)\r\n\t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/genericapiserver/filters/panics.go:75 +0x24a\r\nnet/http.HandlerFunc.ServeHTTP(0xc420ba5f20, 0x7f668e329e10, 0xc420fb1c98, 0xc420c390e0)\r\n\t/usr/local/go/src/net/http/server.go:1726 +0x44\r\nk8s.io/kubernetes/pkg/apiserver/filters.WithRequestInfo.func1(0x7f668e329e10, 0xc420fb1c98, 0xc420c390e0)\r\n\t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/filters/requestinfo.go:45 +0x212\r\nnet/http.HandlerFunc.ServeHTTP(0xc420ba5fb0, 0x7f668e329e10, 0xc420fb1c98, 0xc420c390e0)\r\n\t/usr/local/go/src/net/http/server.go:1726 +0x44\r\nk8s.io/kubernetes/pkg/api.WithRequestContext.func1(0x7f668e329e10, 0xc420fb1c98, 0xc420c390e0)\r\n\t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/api/requestcontext.go:107 +0xef\r\nnet/http.HandlerFunc.ServeHTTP(0xc420bdcd40, 0x7f668e329e10, 0xc420fb1c98, 0xc420c390e0)\r\n\t/usr/local/go/src/net/http/server.go:1726 +0x44\r\nk8s.io/kubernetes/pkg/genericapiserver/filters.(*timeoutHandler).ServeHTTP.func1(0xc420bdcd60, 0x5ad7c80, 0xc420fb1c98, 0xc420c390e0, 0xc421e08840)\r\n\t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/genericapiserver/filters/timeout.go:78 +0x8d\r\ncreated by k8s.io/kubernetes/pkg/genericapiserver/filters.(*timeoutHandler).ServeHTTP\r\n\t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/genericapiserver/filters/timeout.go:80 +0x1db\r\n\r\npanic: runtime error: invalid memory address or nil pointer dereference [recovered]\r\n\tpanic: runtime error: invalid memory address or nil pointer dereference\r\n[signal SIGSEGV: segmentation violation code=0x1 addr=0x30 pc=0x1637e9d]\r\n\r\ngoroutine 675 [running]:\r\npanic(0x2e640e0, 0xc420018040)\r\n\t/usr/local/go/src/runtime/panic.go:500 +0x1a1\r\nk8s.io/kubernetes/pkg/util/runtime.HandleCrash(0xc421563da0, 0x1, 0x1)\r\n\t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/runtime/runtime.go:56 +0x126\r\npanic(0x2e640e0, 0xc420018040)\r\n\t/usr/local/go/src/runtime/panic.go:458 +0x243\r\nk8s.io/kubernetes/pkg/apiserver/filters.WithAudit.func1(0x5ace780, 0xc4213d3500, 0xc420c390e0)\r\n\t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/filters/audit.go:119 +0x49d\r\nnet/http.HandlerFunc.ServeHTTP(0xc420800000, 0x5ace780, 0xc4213d3500, 0xc420c390e0)\r\n\t/usr/local/go/src/net/http/server.go:1726 +0x44\r\nk8s.io/kubernetes/pkg/genericapiserver/filters.WithPanicRecovery.func1(0x5ace780, 0xc4213d3500, 0xc420c390e0)\r\n\t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/genericapiserver/filters/panics.go:75 +0x24a\r\nnet/http.HandlerFunc.ServeHTTP(0xc420ba5f20, 0x7f668e329e10, 0xc420fb1c98, 0xc420c390e0)\r\n\t/usr/local/go/src/net/http/server.go:1726 +0x44\r\nk8s.io/kubernetes/pkg/apiserver/filters.WithRequestInfo.func1(0x7f668e329e10, 0xc420fb1c98, 0xc420c390e0)\r\n\t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/filters/requestinfo.go:45 +0x212\r\nnet/http.HandlerFunc.ServeHTTP(0xc420ba5fb0, 0x7f668e329e10, 0xc420fb1c98, 0xc420c390e0)\r\n\t/usr/local/go/src/net/http/server.go:1726 +0x44\r\nk8s.io/kubernetes/pkg/api.WithRequestContext.func1(0x7f668e329e10, 0xc420fb1c98, 0xc420c390e0)\r\n\t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/api/requestcontext.go:107 +0xef\r\nnet/http.HandlerFunc.ServeHTTP(0xc420bdcd40, 0x7f668e329e10, 0xc420fb1c98, 0xc420c390e0)\r\n\t/usr/local/go/src/net/http/server.go:1726 +0x44\r\nk8s.io/kubernetes/pkg/genericapiserver/filters.(*timeoutHandler).ServeHTTP.func1(0xc420bdcd60, 0x5ad7c80, 0xc420fb1c98, 0xc420c390e0, 0xc421e08840)\r\n\t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/genericapiserver/filters/timeout.go:78 +0x8d\r\ncreated by k8s.io/kubernetes/pkg/genericapiserver/filters.(*timeoutHandler).ServeHTTP\r\n\t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/genericapiserver/filters/timeout.go:80 +0x1db\r\n```\r\n\r\n\r\n**What you expected to happen**:\r\nThe api server not to panic.\r\n\r\n**How to reproduce it** (as minimally and precisely as possible):\r\nMy API-Server config:\r\n```\r\napiVersion: v1\r\nkind: Pod\r\nmetadata:\r\n  name: kube-apiserver\r\n  namespace: kube-system\r\nspec:\r\n  hostNetwork: true\r\n  containers:\r\n  - name: kube-apiserver\r\n    image: \"${hyperkube_image?:\"hyperkube_image must be set\"}\"\r\n    command:\r\n    - /hyperkube\r\n    - apiserver\r\n    - --anonymous-auth=false\r\n    - --admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota\r\n    - --allow-privileged=true\r\n    - --apiserver-count=${control_plane_cluster_size?:\"control_plane_cluster_size must be set\"}\r\n    - --audit-log-maxage=30\r\n    - --audit-log-maxsize=100\r\n    - --audit-log-path=/var/log/apiserver/audit.log\r\n    - --authorization-mode=RBAC\r\n    - --authorization-rbac-super-user=kube-apiserver.${private_fqdn?:\"private_fqdn must be set\"}\r\n    - --bind-address=0.0.0.0\r\n    - --client-ca-file=/etc/kubernetes/ssl/ca.pem\r\n    - --cloud-provider=aws\r\n    - --etcd-cafile=/etc/ssl/etcd/ca.pem\r\n    - --etcd-certfile=/etc/ssl/etcd/etcd_client.pem\r\n    - --etcd-keyfile=/etc/ssl/etcd/etcd_client-key.pem\r\n    - --etcd-servers=https://127.0.0.1:2379\r\n    - --external-hostname=${private_fqdn?:\"private_fqdn must be set\"}\r\n    - --runtime-config=rbac.authorization.k8s.io/v1alpha1,batch/v2alpha1,extensions/v1beta1/thirdpartyresources,policy/v1beta1\r\n    - --secure-port=443\r\n    - --service-account-key-file=/etc/kubernetes/ssl/apiserver-key.pem\r\n    - --service-cluster-ip-range=${k8s_service_cluster_ip_range?:\"k8s_service_cluster_ip_range must be set\"}\r\n    - --tls-cert-file=/etc/kubernetes/ssl/apiserver.pem\r\n    - --tls-private-key-file=/etc/kubernetes/ssl/apiserver-key.pem\r\n    livenessProbe:\r\n      httpGet:\r\n        host: 127.0.0.1\r\n        port: 443\r\n        path: /healthz\r\n      initialDelaySeconds: 15\r\n      timeoutSeconds: 15\r\n    ports:\r\n    - containerPort: 443\r\n      hostPort: 443\r\n      name: https\r\n    - containerPort: 8080\r\n      hostPort: 8080\r\n      name: local\r\n    resources:\r\n      limits:\r\n        cpu: 250m\r\n        memory: 1Gi\r\n      requests:\r\n        cpu: 250m\r\n        memory: 1Gi\r\n    volumeMounts:\r\n    - mountPath: /etc/kubernetes/ssl\r\n      name: ssl-certs-kubernetes\r\n      readOnly: true\r\n    - mountPath: /etc/ssl/etcd\r\n      name: ssl-certs-etcd\r\n      readOnly: true\r\n    - mountPath: /etc/ssl/certs\r\n      name: ssl-certs-host\r\n      readOnly: true\r\n    - mountPath: /var/log/apiserver\r\n      name: apiserver-logs\r\n      readOnly: false\r\n  volumes:\r\n  - hostPath:\r\n      path: /etc/ssl/etcd\r\n    name: ssl-certs-etcd\r\n  - hostPath:\r\n      path: /etc/kubernetes/ssl\r\n    name: ssl-certs-kubernetes\r\n  - hostPath:\r\n      path: /usr/share/ca-certificates\r\n    name: ssl-certs-host\r\n  - hostPath:\r\n      path: /var/log/apiserver\r\n    name: apiserver-logs\r\n```\r\n\r\n\r\n**Anything else do we need to know**:\r\nDisabling audit logs causes this to no blow up. I suspect calling the api on the secure port from the controller using credentials would also cause this not to explode.\r\n"},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/266819242","html_url":"https://github.com/kubernetes/kubernetes/issues/38716#issuecomment-266819242","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/38716","id":266819242,"user":{"login":"deads2k","id":8225098,"avatar_url":"https://avatars.githubusercontent.com/u/8225098?v=3","gravatar_id":"","url":"https://api.github.com/users/deads2k","html_url":"https://github.com/deads2k","followers_url":"https://api.github.com/users/deads2k/followers","following_url":"https://api.github.com/users/deads2k/following{/other_user}","gists_url":"https://api.github.com/users/deads2k/gists{/gist_id}","starred_url":"https://api.github.com/users/deads2k/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/deads2k/subscriptions","organizations_url":"https://api.github.com/users/deads2k/orgs","repos_url":"https://api.github.com/users/deads2k/repos","events_url":"https://api.github.com/users/deads2k/events{/privacy}","received_events_url":"https://api.github.com/users/deads2k/received_events","type":"User","site_admin":false},"created_at":"2016-12-13T18:23:59Z","updated_at":"2016-12-13T18:23:59Z","body":"@saad-ali this crash the API server when auditing is turned on an a request without a user comes in."}}
{"actor":{"display_login":"deads2k"}}