{"action":"opened","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/29638","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/29638/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/29638/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/29638/events","html_url":"https://github.com/kubernetes/kubernetes/issues/29638","id":167722131,"number":29638,"title":"panic 'concurrent map writes' in kube-controller-manager in the DeploymentController","user":{"login":"yannchristensen","id":8725646,"avatar_url":"https://avatars.githubusercontent.com/u/8725646?v=3","gravatar_id":"","url":"https://api.github.com/users/yannchristensen","html_url":"https://github.com/yannchristensen","followers_url":"https://api.github.com/users/yannchristensen/followers","following_url":"https://api.github.com/users/yannchristensen/following{/other_user}","gists_url":"https://api.github.com/users/yannchristensen/gists{/gist_id}","starred_url":"https://api.github.com/users/yannchristensen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yannchristensen/subscriptions","organizations_url":"https://api.github.com/users/yannchristensen/orgs","repos_url":"https://api.github.com/users/yannchristensen/repos","events_url":"https://api.github.com/users/yannchristensen/events{/privacy}","received_events_url":"https://api.github.com/users/yannchristensen/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2016-07-26T21:58:03Z","updated_at":"2016-07-26T21:58:03Z","closed_at":null,"body":"kube-controller-manager is hitting a panic 'concurrent map writes'. We have hit this several times, our cluster has 3 worker nodes and around 10 pods so it is small. \r\n\r\nI extracted the log below with this cmd:\r\n    \r\n> kubectl logs kube-controller-manager-ip-10-16-7-10.us-west-2.compute.internal  --namespace kube-system \r\n\r\n....snip....\r\nI0726 21:39:35.154205       1 controllermanager.go:375] Starting apps/v1alpha1 apis\r\nI0726 21:39:35.154214       1 controllermanager.go:377] Starting PetSet controller\r\nI0726 21:39:35.154714       1 controller.go:237] Starting Daemon Sets controller manager\r\nI0726 21:39:35.156005       1 pet_set.go:144] Starting petset controller\r\nE0726 21:39:35.165470       1 util.go:45] Metric for serviceaccount_controller already registered\r\nI0726 21:39:35.172633       1 attach_detach_controller.go:191] Starting Attach Detach Controller\r\nW0726 21:39:35.174264       1 request.go:347] Field selector: v1 - serviceaccounts - metadata.name - default: need to check if this is versioned correctly.\r\nW0726 21:39:35.224631       1 request.go:347] Field selector: v1 - serviceaccounts - metadata.name - default: need to check if this is versioned correctly.\r\nI0726 21:39:35.239116       1 endpoints_controller.go:322] Waiting for pods controller to sync, requeuing service default/kubernetes\r\nI0726 21:39:35.239149       1 replication_controller.go:554] Waiting for pods controller to sync, requeuing rc kube-system/kube-dns-v11\r\nI0726 21:39:35.239229       1 endpoints_controller.go:322] Waiting for pods controller to sync, requeuing service kube-system/heapster\r\nI0726 21:39:35.243163       1 endpoints_controller.go:322] Waiting for pods controller to sync, requeuing service prod-alpha1-003/signalsapi-service\r\nI0726 21:39:35.243786       1 endpoints_controller.go:322] Waiting for pods controller to sync, requeuing service kube-system/kube-dns\r\nI0726 21:39:35.269144       1 controller.go:641] Waiting for pods controller to sync, requeuing ds ops/signals-fluentd\r\nI0726 21:39:35.286571       1 event.go:216] Event(api.ObjectReference{Kind:\"Deployment\", Namespace:\"prod-alpha-003\", Name:\"fing\", UID:\"762f13c2-5378-11e6-9a77-062e72edb83d\", APIVersion:\"extensions\", ResourceVersion:\"240127\", FieldPath:\"\"}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled up replica set fing-2433196982 to 1\r\nfatal error: concurrent map writes\r\n\r\ngoroutine 343 [running]:\r\nruntime.throw(0x3bb6aa0, 0x15)\r\n\t/usr/local/go/src/runtime/panic.go:547 +0x90 fp=0xc821bc7dc0 sp=0xc821bc7da8\r\nruntime.mapassign1(0x290c6a0, 0xc8212cafc0, 0xc821bc7ee8, 0xc821bc7ed8)\r\n\t/usr/local/go/src/runtime/hashmap.go:445 +0xb1 fp=0xc821bc7e68 sp=0xc821bc7dc0\r\nk8s.io/kubernetes/pkg/util/deployment.equalIgnoreHash(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\r\n\t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/deployment/deployment.go:159 +0x2c1 fp=0xc821bc8270 sp=0xc821bc7e68\r\nk8s.io/kubernetes/pkg/util/deployment.FindOldReplicaSets(0xc820a7ec00, 0xc821433000, 0xe, 0x11, 0xc820058120, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\r\n\t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/deployment/deployment.go:196 +0x418 fp=0xc821bc9998 sp=0xc821bc8270\r\nk8s.io/kubernetes/pkg/controller/deployment.(*DeploymentController).getAllReplicaSetsAndSyncRevision(0xc82103c900, 0xc820a7ec00, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)\r\n\t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/deployment/deployment_controller.go:635 +0x20c fp=0xc821bc9b00 sp=0xc821bc9998\r\nk8s.io/kubernetes/pkg/controller/deployment.(*DeploymentController).syncRollingUpdateDeployment(0xc82103c900, 0xc820a7ec00, 0x0, 0x0)\r\n\t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/deployment/deployment_controller.go:577 +0x54 fp=0xc821bc9bc8 sp=0xc821bc9b00\r\nk8s.io/kubernetes/pkg/controller/deployment.(*DeploymentController).syncDeployment(0xc82103c900, 0xc82176b260, 0x16, 0x0, 0x0)\r\n\t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/deployment/deployment_controller.go:459 +0xa6a fp=0xc821bc9d40 sp=0xc821bc9bc8\r\nk8s.io/kubernetes/pkg/controller/deployment.(*DeploymentController).(k8s.io/kubernetes/pkg/controller/deployment.syncDeployment)-fm(0xc82176b260, 0x16, 0x0, 0x0)\r\n\t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/deployment/deployment_controller.go:166 +0x42 fp=0xc821bc9d70 sp=0xc821bc9d40\r\nk8s.io/kubernetes/pkg/controller/deployment.(*DeploymentController).worker.func1(0xc82103c900)\r\n\t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/deployment/deployment_controller.go:398 +0x113 fp=0xc821bc9e18 sp=0xc821bc9d70\r\nk8s.io/kubernetes/pkg/controller/deployment.(*DeploymentController).worker(0xc82103c900)\r\n\t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/deployment/deployment_controller.go:402 +0x21 fp=0xc821bc9e28 sp=0xc821bc9e18\r\nk8s.io/kubernetes/pkg/controller/deployment.(*DeploymentController).(k8s.io/kubernetes/pkg/controller/deployment.worker)-fm()\r\n\t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/deployment/deployment_controller.go:179 +0x20 fp=0xc821bc9e38 sp=0xc821bc9e28\r\nk8s.io/kubernetes/pkg/util/wait.JitterUntil.func1(0xc8202959b0)\r\n\t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/wait/wait.go:86 +0x4b fp=0xc821bc9e68 sp=0xc821bc9e38\r\nk8s.io/kubernetes/pkg/util/wait.JitterUntil(0xc8202959b0, 0x3b9aca00, 0x0, 0x1, 0xc820058900)\r\n\t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/wait/wait.go:87 +0xb4 fp=0xc821bc9f38 sp=0xc821bc9e68\r\nk8s.io/kubernetes/pkg/util/wait.Until(0xc8202959b0, 0x3b9aca00, 0xc820058900)\r\n\t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/wait/wait.go:49 +0x43 fp=0xc821bc9f68 sp=0xc821bc9f38\r\nruntime.goexit()\r\n\t/usr/local/go/src/runtime/asm_amd64.s:1998 +0x1 fp=0xc821bc9f70 sp=0xc821bc9f68\r\ncreated by k8s.io/kubernetes/pkg/controller/deployment.(*DeploymentController).Run\r\n\t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/deployment/deployment_controller.go:179 +0x158\r\n"}}
{"actor":{"display_login":"yannchristensen"}}