{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/27385","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/27385/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/27385/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/27385/events","html_url":"https://github.com/kubernetes/kubernetes/issues/27385","id":160248599,"number":27385,"title":"Density reports/receives pod status slowly","user":{"login":"rrati","id":87555,"avatar_url":"https://avatars.githubusercontent.com/u/87555?v=3","gravatar_id":"","url":"https://api.github.com/users/rrati","html_url":"https://github.com/rrati","followers_url":"https://api.github.com/users/rrati/followers","following_url":"https://api.github.com/users/rrati/following{/other_user}","gists_url":"https://api.github.com/users/rrati/gists{/gist_id}","starred_url":"https://api.github.com/users/rrati/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rrati/subscriptions","organizations_url":"https://api.github.com/users/rrati/orgs","repos_url":"https://api.github.com/users/rrati/repos","events_url":"https://api.github.com/users/rrati/events{/privacy}","received_events_url":"https://api.github.com/users/rrati/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":11,"created_at":"2016-06-14T18:15:46Z","updated_at":"2016-06-15T12:58:15Z","closed_at":null,"body":"Running density at 95 or 100 pods/node, possibly other densities, shows that pod states are reported or recognized slowly by the density test.  There will be a prolonged period of 0 pods running, and then slowly a few pods will be recognized as running until near the end of the test where the numbers jump quickly.  If you look at the kubelet while the test is running, you will be many more pods running than the density test reports at a given time.\r\n\r\nHere's an example of the output density reports are 100 pods/node:\r\nJun 14 14:11:05.400: INFO: Density Pods: 100 out of 100 created, 0 running, 100 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady \r\nJun 14 14:11:15.400: INFO: Density Pods: 100 out of 100 created, 0 running, 100 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady \r\nJun 14 14:11:25.400: INFO: Density Pods: 100 out of 100 created, 0 running, 100 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady \r\nJun 14 14:11:35.400: INFO: Density Pods: 100 out of 100 created, 0 running, 100 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady \r\nJun 14 14:11:45.400: INFO: Density Pods: 100 out of 100 created, 0 running, 100 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady \r\nJun 14 14:11:55.400: INFO: Density Pods: 100 out of 100 created, 0 running, 100 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady \r\nJun 14 14:12:05.400: INFO: Density Pods: 100 out of 100 created, 6 running, 94 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady \r\nJun 14 14:12:15.401: INFO: Density Pods: 100 out of 100 created, 6 running, 94 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady \r\nJun 14 14:12:25.400: INFO: Density Pods: 100 out of 100 created, 7 running, 93 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady \r\nJun 14 14:12:35.400: INFO: Density Pods: 100 out of 100 created, 10 running, 90 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady \r\nJun 14 14:12:45.401: INFO: Density Pods: 100 out of 100 created, 11 running, 89 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady \r\nJun 14 14:12:55.400: INFO: Density Pods: 100 out of 100 created, 13 running, 87 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady \r\nJun 14 14:13:05.400: INFO: Density Pods: 100 out of 100 created, 15 running, 85 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady \r\nJun 14 14:13:15.400: INFO: Density Pods: 100 out of 100 created, 18 running, 82 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady \r\nJun 14 14:13:25.400: INFO: Density Pods: 100 out of 100 created, 21 running, 79 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady \r\nJun 14 14:13:35.401: INFO: Density Pods: 100 out of 100 created, 25 running, 75 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady \r\nJun 14 14:13:45.400: INFO: Density Pods: 100 out of 100 created, 32 running, 68 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady \r\nJun 14 14:13:55.400: INFO: Density Pods: 100 out of 100 created, 38 running, 62 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady \r\nJun 14 14:14:05.400: INFO: Density Pods: 100 out of 100 created, 46 running, 54 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady \r\nJun 14 14:14:15.400: INFO: Density Pods: 100 out of 100 created, 47 running, 53 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady \r\nJun 14 14:14:25.401: INFO: Density Pods: 100 out of 100 created, 47 running, 53 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady \r\nJun 14 14:14:35.400: INFO: Density Pods: 100 out of 100 created, 62 running, 38 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady \r\nJun 14 14:14:45.400: INFO: Density Pods: 100 out of 100 created, 84 running, 16 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady \r\nJun 14 14:14:55.400: INFO: Density Pods: 100 out of 100 created, 100 running, 0 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady "},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/226179103","html_url":"https://github.com/kubernetes/kubernetes/issues/27385#issuecomment-226179103","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/27385","id":226179103,"user":{"login":"rrati","id":87555,"avatar_url":"https://avatars.githubusercontent.com/u/87555?v=3","gravatar_id":"","url":"https://api.github.com/users/rrati","html_url":"https://github.com/rrati","followers_url":"https://api.github.com/users/rrati/followers","following_url":"https://api.github.com/users/rrati/following{/other_user}","gists_url":"https://api.github.com/users/rrati/gists{/gist_id}","starred_url":"https://api.github.com/users/rrati/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rrati/subscriptions","organizations_url":"https://api.github.com/users/rrati/orgs","repos_url":"https://api.github.com/users/rrati/repos","events_url":"https://api.github.com/users/rrati/events{/privacy}","received_events_url":"https://api.github.com/users/rrati/received_events","type":"User","site_admin":false},"created_at":"2016-06-15T12:58:15Z","updated_at":"2016-06-15T12:58:15Z","body":"Changing the kube-api-burst and kube-api-qps on the kubelet didn't seem to do anything.  kubelet args:\r\n/usr/bin/kubelet --logtostderr=true --v=0 --api-servers=https://192.2.5.40:443 --address=0.0.0.0 --hostname-override=192.2.5.41 --allow-privileged=true --kubeconfig=/etc/kubernetes/kubeletkubeconfig --config=/etc/kubernetes/manifests --cluster-dns=10.254.0.10 --cluster-domain=cluster.local --max-pods=250 --pods-per-core=10 --kube-api-burst=100 --kube-api-qps=50\r\n\r\nI'm still seeing a big delay:\r\nJun 15 08:54:05.517: INFO: Density Pods: 100 out of 100 created, 0 running, 100 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady \r\nJun 15 08:54:15.516: INFO: Density Pods: 100 out of 100 created, 0 running, 100 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady \r\nJun 15 08:54:25.516: INFO: Density Pods: 100 out of 100 created, 0 running, 100 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady \r\nJun 15 08:54:35.516: INFO: Density Pods: 100 out of 100 created, 0 running, 100 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady \r\nJun 15 08:54:45.517: INFO: Density Pods: 100 out of 100 created, 0 running, 100 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady \r\nJun 15 08:54:55.516: INFO: Density Pods: 100 out of 100 created, 0 running, 100 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady \r\nJun 15 08:55:05.516: INFO: Density Pods: 100 out of 100 created, 0 running, 100 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady \r\nJun 15 08:55:15.516: INFO: Density Pods: 100 out of 100 created, 0 running, 100 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady \r\nJun 15 08:55:25.516: INFO: Density Pods: 100 out of 100 created, 0 running, 100 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady \r\nJun 15 08:55:35.517: INFO: Density Pods: 100 out of 100 created, 0 running, 100 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady \r\nJun 15 08:55:45.516: INFO: Density Pods: 100 out of 100 created, 1 running, 99 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady \r\nJun 15 08:55:55.516: INFO: Density Pods: 100 out of 100 created, 7 running, 93 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady \r\nJun 15 08:56:05.520: INFO: Density Pods: 100 out of 100 created, 12 running, 88 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady \r\nJun 15 08:56:15.516: INFO: Density Pods: 100 out of 100 created, 16 running, 83 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 1 runningButNotReady \r\nJun 15 08:56:25.516: INFO: Density Pods: 100 out of 100 created, 19 running, 80 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 1 runningButNotReady \r\nJun 15 08:56:35.516: INFO: Density Pods: 100 out of 100 created, 24 running, 75 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 1 runningButNotReady \r\nJun 15 08:56:45.516: INFO: Density Pods: 100 out of 100 created, 31 running, 68 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 1 runningButNotReady \r\nJun 15 08:56:55.516: INFO: Density Pods: 100 out of 100 created, 38 running, 61 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 1 runningButNotReady \r\nJun 15 08:57:05.516: INFO: Density Pods: 100 out of 100 created, 69 running, 29 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 2 runningButNotReady \r\nJun 15 08:57:15.516: INFO: Density Pods: 100 out of 100 created, 100 running, 0 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady \r\n\r\nIn fact on this run, I had containers fail to start:\r\nJun 15 08:57:17.164: INFO: Unexpected error occurred: 2 containers failed which is more than allowed 0\r\npanic: \r\nYour test failed.\r\n"}}
{"actor":{"display_login":"rrati"}}