{"action":"created","issue":{"url":"https://api.github.com/repos/cockroachdb/cockroach/issues/22424","repository_url":"https://api.github.com/repos/cockroachdb/cockroach","labels_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/22424/labels{/name}","comments_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/22424/comments","events_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/22424/events","html_url":"https://github.com/cockroachdb/cockroach/issues/22424","id":294849349,"number":22424,"title":"QPS Drops to Zero during rolling upgrade","user":{"login":"nstewart","id":894257,"avatar_url":"https://avatars0.githubusercontent.com/u/894257?v=4","gravatar_id":"","url":"https://api.github.com/users/nstewart","html_url":"https://github.com/nstewart","followers_url":"https://api.github.com/users/nstewart/followers","following_url":"https://api.github.com/users/nstewart/following{/other_user}","gists_url":"https://api.github.com/users/nstewart/gists{/gist_id}","starred_url":"https://api.github.com/users/nstewart/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nstewart/subscriptions","organizations_url":"https://api.github.com/users/nstewart/orgs","repos_url":"https://api.github.com/users/nstewart/repos","events_url":"https://api.github.com/users/nstewart/events{/privacy}","received_events_url":"https://api.github.com/users/nstewart/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-02-06T17:38:11Z","updated_at":"2018-02-06T17:39:46Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"Running kv workload on a 6 node k8s cluster (5 cockroachDB nodes). I triggered a rolling update from 1.1.5 to v2.0-alpha.20180129 and saw QPS temporarily dropped to zero\r\n\r\n```\r\n   6m53s        0          861.8          626.5      2.5   3892.3   4026.5\r\n   6m54s        0          560.9          626.3    226.5    335.5    352.3\r\n   6m55s        0           25.0          624.9    436.2    436.2    436.2\r\n   6m56s        0            0.0          623.4      0.0      0.0      0.0\r\n   6m57s        0            0.0          621.9      0.0      0.0      0.0\r\n   6m58s        0            0.0          620.4      0.0      0.0      0.0\r\n   6m59s        0            0.0          618.9      0.0      0.0      0.0\r\n    7m0s        0            0.0          617.4      0.0      0.0      0.0\r\n_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p95(ms)__p99(ms)_pMax(ms)\r\n    7m1s        0            0.0          616.0      0.0      0.0      0.0\r\n    7m2s        0            0.0          614.5      0.0      0.0      0.0\r\n    7m3s        0            0.0          613.1      0.0      0.0      0.0\r\n    7m4s        0            0.0          611.6      0.0      0.0      0.0\r\n    7m5s        0            0.0          610.2      0.0      0.0      0.0\r\n    7m6s        0            0.0          608.8      0.0      0.0      0.0\r\n    7m7s        0            0.0          607.3      0.0      0.0      0.0\r\n    7m8s        0            0.0          605.9      0.0      0.0      0.0\r\n    7m9s        0            0.0          604.5      0.0      0.0      0.0\r\n   7m10s        0            0.0          603.1      0.0      0.0      0.0\r\n   7m11s        0            0.0          601.7      0.0      0.0      0.0\r\n   7m12s        0            0.0          600.3      0.0      0.0      0.0\r\n```\r\n\r\nI also saw `pq: server is not accepting clients` once, which incremented the error count to 1. \r\n\r\n![image](https://user-images.githubusercontent.com/894257/35874154-c672d982-0b39-11e8-8563-efae42bd19fe.png)\r\n\r\nI'm using the k8s internal load balancer, which checks /health on nodes every 5 seconds. \r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/cockroachdb/cockroach/issues/comments/363503447","html_url":"https://github.com/cockroachdb/cockroach/issues/22424#issuecomment-363503447","issue_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/22424","id":363503447,"user":{"login":"nstewart","id":894257,"avatar_url":"https://avatars0.githubusercontent.com/u/894257?v=4","gravatar_id":"","url":"https://api.github.com/users/nstewart","html_url":"https://github.com/nstewart","followers_url":"https://api.github.com/users/nstewart/followers","following_url":"https://api.github.com/users/nstewart/following{/other_user}","gists_url":"https://api.github.com/users/nstewart/gists{/gist_id}","starred_url":"https://api.github.com/users/nstewart/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nstewart/subscriptions","organizations_url":"https://api.github.com/users/nstewart/orgs","repos_url":"https://api.github.com/users/nstewart/repos","events_url":"https://api.github.com/users/nstewart/events{/privacy}","received_events_url":"https://api.github.com/users/nstewart/received_events","type":"User","site_admin":false},"created_at":"2018-02-06T17:39:46Z","updated_at":"2018-02-06T17:39:46Z","author_association":"CONTRIBUTOR","body":"note from @bdarnell re: the server not accepting clients\r\n\r\n> our health endpoint may need to be updated to support our draining state\r\n> or we need to augment the draining protocol to wait long enough for the LB's health checks to occur before going into this draining state"}}
{"actor":{"display_login":"nstewart"}}