{"action":"created","issue":{"url":"https://api.github.com/repos/cockroachdb/cockroach/issues/32760","repository_url":"https://api.github.com/repos/cockroachdb/cockroach","labels_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/32760/labels{/name}","comments_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/32760/comments","events_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/32760/events","html_url":"https://github.com/cockroachdb/cockroach/issues/32760","id":386430414,"node_id":"MDU6SXNzdWUzODY0MzA0MTQ=","number":32760,"title":"storage: TestRemovePlaceholderRace failed under stress","user":{"login":"cockroach-teamcity","id":20559710,"node_id":"MDQ6VXNlcjIwNTU5NzEw","avatar_url":"https://avatars0.githubusercontent.com/u/20559710?v=4","gravatar_id":"","url":"https://api.github.com/users/cockroach-teamcity","html_url":"https://github.com/cockroach-teamcity","followers_url":"https://api.github.com/users/cockroach-teamcity/followers","following_url":"https://api.github.com/users/cockroach-teamcity/following{/other_user}","gists_url":"https://api.github.com/users/cockroach-teamcity/gists{/gist_id}","starred_url":"https://api.github.com/users/cockroach-teamcity/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cockroach-teamcity/subscriptions","organizations_url":"https://api.github.com/users/cockroach-teamcity/orgs","repos_url":"https://api.github.com/users/cockroach-teamcity/repos","events_url":"https://api.github.com/users/cockroach-teamcity/events{/privacy}","received_events_url":"https://api.github.com/users/cockroach-teamcity/received_events","type":"User","site_admin":false},"labels":[{"id":191587409,"node_id":"MDU6TGFiZWwxOTE1ODc0MDk=","url":"https://api.github.com/repos/cockroachdb/cockroach/labels/C-test-failure","name":"C-test-failure","color":"c5def5","default":false},{"id":335217860,"node_id":"MDU6TGFiZWwzMzUyMTc4NjA=","url":"https://api.github.com/repos/cockroachdb/cockroach/labels/O-robot","name":"O-robot","color":"666ad1","default":false}],"state":"open","locked":false,"assignee":{"login":"petermattis","id":6201335,"node_id":"MDQ6VXNlcjYyMDEzMzU=","avatar_url":"https://avatars0.githubusercontent.com/u/6201335?v=4","gravatar_id":"","url":"https://api.github.com/users/petermattis","html_url":"https://github.com/petermattis","followers_url":"https://api.github.com/users/petermattis/followers","following_url":"https://api.github.com/users/petermattis/following{/other_user}","gists_url":"https://api.github.com/users/petermattis/gists{/gist_id}","starred_url":"https://api.github.com/users/petermattis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/petermattis/subscriptions","organizations_url":"https://api.github.com/users/petermattis/orgs","repos_url":"https://api.github.com/users/petermattis/repos","events_url":"https://api.github.com/users/petermattis/events{/privacy}","received_events_url":"https://api.github.com/users/petermattis/received_events","type":"User","site_admin":false},"assignees":[{"login":"petermattis","id":6201335,"node_id":"MDQ6VXNlcjYyMDEzMzU=","avatar_url":"https://avatars0.githubusercontent.com/u/6201335?v=4","gravatar_id":"","url":"https://api.github.com/users/petermattis","html_url":"https://github.com/petermattis","followers_url":"https://api.github.com/users/petermattis/followers","following_url":"https://api.github.com/users/petermattis/following{/other_user}","gists_url":"https://api.github.com/users/petermattis/gists{/gist_id}","starred_url":"https://api.github.com/users/petermattis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/petermattis/subscriptions","organizations_url":"https://api.github.com/users/petermattis/orgs","repos_url":"https://api.github.com/users/petermattis/repos","events_url":"https://api.github.com/users/petermattis/events{/privacy}","received_events_url":"https://api.github.com/users/petermattis/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/cockroachdb/cockroach/milestones/17","html_url":"https://github.com/cockroachdb/cockroach/milestone/17","labels_url":"https://api.github.com/repos/cockroachdb/cockroach/milestones/17/labels","id":3492386,"node_id":"MDk6TWlsZXN0b25lMzQ5MjM4Ng==","number":17,"title":"2.2","description":"","creator":{"login":"petermattis","id":6201335,"node_id":"MDQ6VXNlcjYyMDEzMzU=","avatar_url":"https://avatars0.githubusercontent.com/u/6201335?v=4","gravatar_id":"","url":"https://api.github.com/users/petermattis","html_url":"https://github.com/petermattis","followers_url":"https://api.github.com/users/petermattis/followers","following_url":"https://api.github.com/users/petermattis/following{/other_user}","gists_url":"https://api.github.com/users/petermattis/gists{/gist_id}","starred_url":"https://api.github.com/users/petermattis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/petermattis/subscriptions","organizations_url":"https://api.github.com/users/petermattis/orgs","repos_url":"https://api.github.com/users/petermattis/repos","events_url":"https://api.github.com/users/petermattis/events{/privacy}","received_events_url":"https://api.github.com/users/petermattis/received_events","type":"User","site_admin":false},"open_issues":195,"closed_issues":732,"state":"open","created_at":"2018-07-12T18:50:47Z","updated_at":"2018-12-03T21:33:52Z","due_on":null,"closed_at":null},"comments":2,"created_at":"2018-12-01T06:46:22Z","updated_at":"2018-12-04T07:07:17Z","closed_at":null,"author_association":"MEMBER","body":"SHA: https://github.com/cockroachdb/cockroach/commits/ce35979e2d8d7fdbffa6543fb646fadbdf442ab3\n\nParameters:\n```\nTAGS=\nGOFLAGS=\n```\n\nTo repro, try:\n\n```\n# Don't forget to check out a clean suitable branch and experiment with the\n# stress invocation until the desired results present themselves. For example,\n# using stress instead of stressrace and passing the '-p' stressflag which\n# controls concurrency.\n./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh\ncd ~/go/src/github.com/cockroachdb/cockroach && \\\nstdbuf -oL -eL \\\nmake stressrace TESTS=TestRemovePlaceholderRace PKG=github.com/cockroachdb/cockroach/pkg/storage TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log\n```\n\nFailed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1034772&tab=buildLog\n\n```\nW181201 06:38:26.651610 70312 storage/node_liveness.go:454  [hb] failed node liveness heartbeat: context deadline exceeded\nI181201 06:38:26.673227 70541 storage/node_liveness.go:790  [hb] retrying liveness update after storage.errRetryLiveness: result is ambiguous (context deadline exceeded)\nW181201 06:38:26.673278 70541 storage/node_liveness.go:454  [hb] failed node liveness heartbeat: context deadline exceeded\nW181201 06:38:27.106152 70312 storage/node_liveness.go:454  [hb] failed node liveness heartbeat: aborted while in command queue: context deadline exceeded\nW181201 06:38:27.124176 70441 storage/node_liveness.go:454  [hb] failed node liveness heartbeat: aborted while in command queue: context deadline exceeded\nW181201 06:38:27.125327 70541 storage/node_liveness.go:454  [hb] failed node liveness heartbeat: aborted while in command queue: context deadline exceeded\nW181201 06:38:51.867806 70411 storage/store_rebalancer.go:226  [s1,store-rebalancer] StorePool missing descriptor for local store\nW181201 06:39:02.331301 70523 storage/store_rebalancer.go:226  [s2,store-rebalancer] StorePool missing descriptor for local store\nW181201 06:39:13.862573 70619 storage/store_rebalancer.go:226  [s3,store-rebalancer] StorePool missing descriptor for local store\nW181201 06:39:51.478881 70523 storage/store_rebalancer.go:226  [s2,store-rebalancer] StorePool missing descriptor for local store\nW181201 06:39:55.162162 70411 storage/store_rebalancer.go:226  [s1,store-rebalancer] StorePool missing descriptor for local store\nW181201 06:40:26.724292 70619 storage/store_rebalancer.go:226  [s3,store-rebalancer] StorePool missing descriptor for local store\nW181201 06:40:41.747581 70523 storage/store_rebalancer.go:226  [s2,store-rebalancer] StorePool missing descriptor for local store\nW181201 06:41:03.335600 70411 storage/store_rebalancer.go:226  [s1,store-rebalancer] StorePool missing descriptor for local store\nW181201 06:41:32.025214 70619 storage/store_rebalancer.go:226  [s3,store-rebalancer] StorePool missing descriptor for local store\nW181201 06:41:39.129535 70523 storage/store_rebalancer.go:226  [s2,store-rebalancer] StorePool missing descriptor for local store\nW181201 06:42:05.482910 70411 storage/store_rebalancer.go:226  [s1,store-rebalancer] StorePool missing descriptor for local store\nW181201 06:42:35.586197 70619 storage/store_rebalancer.go:226  [s3,store-rebalancer] StorePool missing descriptor for local store\nW181201 06:42:48.243636 70660 gossip/gossip.go:1506  [n3] first range unavailable; trying remaining resolvers\nI181201 06:42:50.605207 70660 gossip/gossip.go:1515  [n3] node has connected to cluster via gossip\nW181201 06:42:52.351591 70523 storage/store_rebalancer.go:226  [s2,store-rebalancer] StorePool missing descriptor for local store\nW181201 06:43:12.677699 70411 storage/store_rebalancer.go:226  [s1,store-rebalancer] StorePool missing descriptor for local store\nW181201 06:43:30.375823 70619 storage/store_rebalancer.go:226  [s3,store-rebalancer] StorePool missing descriptor for local store\nW181201 06:43:49.534961 70523 storage/store_rebalancer.go:226  [s2,store-rebalancer] StorePool missing descriptor for local store\nW181201 06:44:14.802130 70411 storage/store_rebalancer.go:226  [s1,store-rebalancer] StorePool missing descriptor for local store\nW181201 06:44:29.150158 70619 storage/store_rebalancer.go:226  [s3,store-rebalancer] StorePool missing descriptor for local store\nW181201 06:44:55.151724 70523 storage/store_rebalancer.go:226  [s2,store-rebalancer] StorePool missing descriptor for local store\nW181201 06:45:00.426294 70411 storage/store_rebalancer.go:226  [s1,store-rebalancer] StorePool missing descriptor for local store\nW181201 06:45:21.851914 70619 storage/store_rebalancer.go:226  [s3,store-rebalancer] StorePool missing descriptor for local store\nW181201 06:45:49.794140 70411 storage/store_rebalancer.go:226  [s1,store-rebalancer] StorePool missing descriptor for local store\nW181201 06:45:54.754144 70523 storage/store_rebalancer.go:226  [s2,store-rebalancer] StorePool missing descriptor for local store\n\n\nERROR: signal: killed\n\n44 runs completed, 1 failures, over 20m8s\ncontext canceled\n\n```"},"comment":{"url":"https://api.github.com/repos/cockroachdb/cockroach/issues/comments/443994491","html_url":"https://github.com/cockroachdb/cockroach/issues/32760#issuecomment-443994491","issue_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/32760","id":443994491,"node_id":"MDEyOklzc3VlQ29tbWVudDQ0Mzk5NDQ5MQ==","user":{"login":"cockroach-teamcity","id":20559710,"node_id":"MDQ6VXNlcjIwNTU5NzEw","avatar_url":"https://avatars0.githubusercontent.com/u/20559710?v=4","gravatar_id":"","url":"https://api.github.com/users/cockroach-teamcity","html_url":"https://github.com/cockroach-teamcity","followers_url":"https://api.github.com/users/cockroach-teamcity/followers","following_url":"https://api.github.com/users/cockroach-teamcity/following{/other_user}","gists_url":"https://api.github.com/users/cockroach-teamcity/gists{/gist_id}","starred_url":"https://api.github.com/users/cockroach-teamcity/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cockroach-teamcity/subscriptions","organizations_url":"https://api.github.com/users/cockroach-teamcity/orgs","repos_url":"https://api.github.com/users/cockroach-teamcity/repos","events_url":"https://api.github.com/users/cockroach-teamcity/events{/privacy}","received_events_url":"https://api.github.com/users/cockroach-teamcity/received_events","type":"User","site_admin":false},"created_at":"2018-12-04T07:07:17Z","updated_at":"2018-12-04T07:07:17Z","author_association":"MEMBER","body":"SHA: https://github.com/cockroachdb/cockroach/commits/e6cb0c5c329617b560eee37527248171b5e06382\n\nParameters:\n```\nTAGS=\nGOFLAGS=\n```\n\nTo repro, try:\n\n```\n# Don't forget to check out a clean suitable branch and experiment with the\n# stress invocation until the desired results present themselves. For example,\n# using stress instead of stressrace and passing the '-p' stressflag which\n# controls concurrency.\n./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh\ncd ~/go/src/github.com/cockroachdb/cockroach && \\\nstdbuf -oL -eL \\\nmake stressrace TESTS=TestRemovePlaceholderRace PKG=github.com/cockroachdb/cockroach/pkg/storage TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log\n```\n\nFailed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1038695&tab=buildLog\n\n```\nW181204 07:05:35.017832 70563 storage/node_liveness.go:454  [hb] failed node liveness heartbeat: context deadline exceeded\nW181204 07:05:35.052201 70645 storage/store_rebalancer.go:226  [s2,store-rebalancer] StorePool missing descriptor for local store\nW181204 07:05:35.052308 70692 gossip/gossip.go:1504  [n2] first range unavailable; resolvers exhausted\nW181204 07:05:35.099990 70540 storage/store_rebalancer.go:226  [s1,store-rebalancer] StorePool missing descriptor for local store\nW181204 07:05:35.100039 70540 storage/store_rebalancer.go:226  [s1,store-rebalancer] StorePool missing descriptor for local store\nW181204 07:05:35.100047 70773 storage/store_rebalancer.go:226  [s3,store-rebalancer] StorePool missing descriptor for local store\nW181204 07:05:35.100112 70653 gossip/gossip.go:1506  [n3] first range unavailable; trying remaining resolvers\nW181204 07:05:35.100579 70655 storage/node_liveness.go:518  [hb] slow heartbeat took 69.1s\nW181204 07:05:35.100642 70655 storage/node_liveness.go:454  [hb] failed node liveness heartbeat: context deadline exceeded\nW181204 07:05:35.102026 70666 storage/node_liveness.go:518  [hb] slow heartbeat took 293.5s\nW181204 07:05:35.102078 70666 storage/node_liveness.go:454  [hb] failed node liveness heartbeat: context deadline exceeded\nW181204 07:05:35.176839 70773 storage/store_rebalancer.go:226  [s3,store-rebalancer] StorePool missing descriptor for local store\nW181204 07:05:35.772163 70666 storage/node_liveness.go:454  [hb] failed node liveness heartbeat: aborted before command queue: context deadline exceeded\nI181204 07:05:35.773229 70563 storage/node_liveness.go:790  [hb] retrying liveness update after storage.errRetryLiveness: result is ambiguous (context deadline exceeded)\nW181204 07:05:35.773285 70563 storage/node_liveness.go:454  [hb] failed node liveness heartbeat: context deadline exceeded\nI181204 07:05:35.774328 70655 storage/node_liveness.go:790  [hb] retrying liveness update after storage.errRetryLiveness: result is ambiguous (context deadline exceeded)\nW181204 07:05:35.774376 70655 storage/node_liveness.go:454  [hb] failed node liveness heartbeat: context deadline exceeded\nI181204 07:05:36.032415 2449837 gossip/client.go:129  [n3] started gossip client to 127.0.0.1:37247\nW181204 07:05:36.032907 70467 storage/store.go:3748  [s1] handle raft ready: 0.9s [processed=0]\nI181204 07:05:36.033550 2450712 gossip/client.go:129  [n2] started gossip client to 127.0.0.1:36917\nI181204 07:05:36.421118 2449837 gossip/client.go:134  [n3] closing client to n2 (127.0.0.1:37247): stopping outgoing client to n2 (127.0.0.1:37247); already have incoming\nW181204 07:05:36.421643 70563 storage/node_liveness.go:454  [hb] failed node liveness heartbeat: aborted while in command queue: context deadline exceeded\nI181204 07:05:36.421657 2450664 gossip/client.go:129  [n3] started gossip client to 127.0.0.1:39015\nI181204 07:05:36.421827 2450772 gossip/client.go:129  [n2] started gossip client to 127.0.0.1:39015\nW181204 07:05:36.422042 70655 storage/node_liveness.go:454  [hb] failed node liveness heartbeat: aborted while in command queue: context deadline exceeded\nI181204 07:05:36.423076 70666 storage/node_liveness.go:790  [hb] retrying liveness update after storage.errRetryLiveness: result is ambiguous (context deadline exceeded)\nW181204 07:05:36.423118 70666 storage/node_liveness.go:454  [hb] failed node liveness heartbeat: context deadline exceeded\nI181204 07:05:36.904918 70653 gossip/gossip.go:1515  [n3] node has connected to cluster via gossip\nI181204 07:05:36.906118 70692 gossip/gossip.go:1515  [n2] node has connected to cluster via gossip\nW181204 07:06:35.206626 70773 storage/store_rebalancer.go:226  [s3,store-rebalancer] StorePool missing descriptor for local store\nW181204 07:06:39.788621 70540 storage/store_rebalancer.go:226  [s1,store-rebalancer] StorePool missing descriptor for local store\nW181204 07:06:42.426125 70645 storage/store_rebalancer.go:226  [s2,store-rebalancer] StorePool missing descriptor for local store\n\n\nERROR: signal: killed\n\n82 runs completed, 1 failures, over 42m13s\ncontext canceled\n\n```"}}
{"actor":{"display_login":"cockroach-teamcity"}}