{"action":"created","issue":{"url":"https://api.github.com/repos/cockroachdb/cockroach/issues/27497","repository_url":"https://api.github.com/repos/cockroachdb/cockroach","labels_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/27497/labels{/name}","comments_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/27497/comments","events_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/27497/events","html_url":"https://github.com/cockroachdb/cockroach/issues/27497","id":340896345,"node_id":"MDU6SXNzdWUzNDA4OTYzNDU=","number":27497,"title":"*: grpc error during replica rebalancing (fails TestShowTraceReplica, TestReplicateQueueDownReplicate)","user":{"login":"cockroach-teamcity","id":20559710,"node_id":"MDQ6VXNlcjIwNTU5NzEw","avatar_url":"https://avatars0.githubusercontent.com/u/20559710?v=4","gravatar_id":"","url":"https://api.github.com/users/cockroach-teamcity","html_url":"https://github.com/cockroach-teamcity","followers_url":"https://api.github.com/users/cockroach-teamcity/followers","following_url":"https://api.github.com/users/cockroach-teamcity/following{/other_user}","gists_url":"https://api.github.com/users/cockroach-teamcity/gists{/gist_id}","starred_url":"https://api.github.com/users/cockroach-teamcity/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cockroach-teamcity/subscriptions","organizations_url":"https://api.github.com/users/cockroach-teamcity/orgs","repos_url":"https://api.github.com/users/cockroach-teamcity/repos","events_url":"https://api.github.com/users/cockroach-teamcity/events{/privacy}","received_events_url":"https://api.github.com/users/cockroach-teamcity/received_events","type":"User","site_admin":false},"labels":[{"id":191587409,"node_id":"MDU6TGFiZWwxOTE1ODc0MDk=","url":"https://api.github.com/repos/cockroachdb/cockroach/labels/C-test-failure","name":"C-test-failure","color":"c5def5","default":false},{"id":335217860,"node_id":"MDU6TGFiZWwzMzUyMTc4NjA=","url":"https://api.github.com/repos/cockroachdb/cockroach/labels/O-robot","name":"O-robot","color":"666ad1","default":false}],"state":"open","locked":false,"assignee":{"login":"petermattis","id":6201335,"node_id":"MDQ6VXNlcjYyMDEzMzU=","avatar_url":"https://avatars0.githubusercontent.com/u/6201335?v=4","gravatar_id":"","url":"https://api.github.com/users/petermattis","html_url":"https://github.com/petermattis","followers_url":"https://api.github.com/users/petermattis/followers","following_url":"https://api.github.com/users/petermattis/following{/other_user}","gists_url":"https://api.github.com/users/petermattis/gists{/gist_id}","starred_url":"https://api.github.com/users/petermattis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/petermattis/subscriptions","organizations_url":"https://api.github.com/users/petermattis/orgs","repos_url":"https://api.github.com/users/petermattis/repos","events_url":"https://api.github.com/users/petermattis/events{/privacy}","received_events_url":"https://api.github.com/users/petermattis/received_events","type":"User","site_admin":false},"assignees":[{"login":"petermattis","id":6201335,"node_id":"MDQ6VXNlcjYyMDEzMzU=","avatar_url":"https://avatars0.githubusercontent.com/u/6201335?v=4","gravatar_id":"","url":"https://api.github.com/users/petermattis","html_url":"https://github.com/petermattis","followers_url":"https://api.github.com/users/petermattis/followers","following_url":"https://api.github.com/users/petermattis/following{/other_user}","gists_url":"https://api.github.com/users/petermattis/gists{/gist_id}","starred_url":"https://api.github.com/users/petermattis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/petermattis/subscriptions","organizations_url":"https://api.github.com/users/petermattis/orgs","repos_url":"https://api.github.com/users/petermattis/repos","events_url":"https://api.github.com/users/petermattis/events{/privacy}","received_events_url":"https://api.github.com/users/petermattis/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/cockroachdb/cockroach/milestones/13","html_url":"https://github.com/cockroachdb/cockroach/milestone/13","labels_url":"https://api.github.com/repos/cockroachdb/cockroach/milestones/13/labels","id":2745639,"node_id":"MDk6TWlsZXN0b25lMjc0NTYzOQ==","number":13,"title":"2.1","description":"","creator":{"login":"knz","id":642886,"node_id":"MDQ6VXNlcjY0Mjg4Ng==","avatar_url":"https://avatars3.githubusercontent.com/u/642886?v=4","gravatar_id":"","url":"https://api.github.com/users/knz","html_url":"https://github.com/knz","followers_url":"https://api.github.com/users/knz/followers","following_url":"https://api.github.com/users/knz/following{/other_user}","gists_url":"https://api.github.com/users/knz/gists{/gist_id}","starred_url":"https://api.github.com/users/knz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/knz/subscriptions","organizations_url":"https://api.github.com/users/knz/orgs","repos_url":"https://api.github.com/users/knz/repos","events_url":"https://api.github.com/users/knz/events{/privacy}","received_events_url":"https://api.github.com/users/knz/received_events","type":"User","site_admin":false},"open_issues":578,"closed_issues":837,"state":"open","created_at":"2017-09-05T15:30:49Z","updated_at":"2018-07-13T08:23:40Z","due_on":null,"closed_at":null},"comments":1,"created_at":"2018-07-13T06:25:15Z","updated_at":"2018-07-13T14:44:27Z","closed_at":null,"author_association":"MEMBER","body":"SHA: https://github.com/cockroachdb/cockroach/commits/7a388b610d1e91d8fa3f62c29a7ad39f17b0cc60\n\nParameters:\n```\nTAGS=\nGOFLAGS=-race\n```\n\nFailed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=776684&tab=buildLog\n\n```\n=== RUN   TestShowTraceReplica/DELETE_FROM_d.t2\nI180713 06:24:10.128895 162404 storage/replica_command.go:736  [replicate,n3,s3,r26/3:/{Table/55-Max}] change replicas (REMOVE_REPLICA (n1,s1):1): read existing descriptor r26:/{Table/55-Max} [(n1,s1):1, (n3,s3):3, next=4]\nI180713 06:24:10.135850 162404 storage/replica.go:3548  [n3,s3,r26/3:/{Table/55-Max}] proposing REMOVE_REPLICA((n1,s1):1): updated=[(n3,s3):3] next=4\nI180713 06:24:10.137696 158759 storage/store.go:3557  [n1,s1,r26/1:/{Table/55-Max}] added to replica GC queue (peer suggestion)\nI180713 06:24:10.137777 162410 storage/store.go:2503  [replicaGC,n1,s1,r26/1:/{Table/55-Max}] removing replica\nI180713 06:24:10.138022 162410 storage/replica.go:835  [replicaGC,n1,s1,r26/1:/{Table/55-Max}] removed 8 (0+8) keys in 0ms [clear=0ms commit=0ms]\nI180713 06:24:10.158506 162415 storage/replica_command.go:736  [replicate,n4,s4,r12/2:/Table/1{5-6}] change replicas (REMOVE_REPLICA (n1,s1):1): read existing descriptor r12:/Table/1{5-6} [(n1,s1):1, (n4,s4):2, next=3]\nI180713 06:24:10.161976 162415 storage/replica.go:3548  [n4,s4,r12/2:/Table/1{5-6}] proposing REMOVE_REPLICA((n1,s1):1): updated=[(n4,s4):2] next=3\nI180713 06:24:10.164677 162437 storage/store.go:2503  [replicaGC,n1,s1,r12/1:/Table/1{5-6}] removing replica\nI180713 06:24:10.164834 158377 storage/store.go:3557  [n1,s1,r12/1:/Table/1{5-6}] added to replica GC queue (peer suggestion)\nI180713 06:24:10.164858 162437 storage/replica.go:835  [replicaGC,n1,s1,r12/1:/Table/1{5-6}] removed 8 (0+8) keys in 0ms [clear=0ms commit=0ms]\nI180713 06:24:11.694563 162441 storage/replica_range_lease.go:554  [replicate,n1,s1,r25/1:/Table/5{4-5}] transferring lease to s4\nI180713 06:24:11.695490 162441 storage/replica_range_lease.go:617  [replicate,n1,s1,r25/1:/Table/5{4-5}] done transferring lease to s4: <nil>\nI180713 06:24:11.695696 158116 storage/replica_proposal.go:203  [n4,s4,r25/2:/Table/5{4-5}] new range lease repl=(n4,s4):2 seq=7 start=1531463051.694574684,0 epo=1 pro=1531463051.694578117,0 following repl=(n1,s1):1 seq=6 start=1531463044.156017688,0 epo=1 pro=1531463044.156022555,0\nI180713 06:24:12.213439 157147 storage/compactor/compactor.go:370  [n1,s1,compactor] processing compaction #1-8/8 (/Table/12-/Max) for 48 KiB (reasons: size=false used=true avail=false)\nI180713 06:24:12.215600 157147 storage/compactor/compactor.go:386  [n1,s1,compactor] processed compaction #1-8/8 (/Table/12-/Max) for 48 KiB in 0.0s\nI180713 06:24:13.677933 157169 server/status/runtime.go:255  [n1] runtime stats: 240 MiB RSS, 1144 goroutines, 39 MiB/45 MiB/112 MiB GO alloc/idle/total, 37 MiB/79 MiB CGO alloc/total, 265.70cgo/sec, 0.04/0.01 %(u/s)time, 0.00 %gc (2x)\nI180713 06:24:13.963979 157558 server/status/runtime.go:255  [n2] runtime stats: 239 MiB RSS, 1144 goroutines, 45 MiB/40 MiB/112 MiB GO alloc/idle/total, 40 MiB/80 MiB CGO alloc/total, 263.12cgo/sec, 0.04/0.01 %(u/s)time, 0.00 %gc (1x)\nI180713 06:24:14.051908 157813 server/status/runtime.go:255  [n3] runtime stats: 239 MiB RSS, 1144 goroutines, 50 MiB/35 MiB/112 MiB GO alloc/idle/total, 40 MiB/80 MiB CGO alloc/total, 263.10cgo/sec, 0.04/0.01 %(u/s)time, 0.00 %gc (1x)\nI180713 06:24:14.123886 158030 server/status/runtime.go:255  [n4] runtime stats: 239 MiB RSS, 1144 goroutines, 30 MiB/51 MiB/112 MiB GO alloc/idle/total, 40 MiB/81 MiB CGO alloc/total, 263.19cgo/sec, 0.04/0.01 %(u/s)time, 0.00 %gc (2x)\nI180713 06:24:20.163192 162586 storage/replica_range_lease.go:554  [replicate,n4,s4,r25/2:/Table/5{4-5}] transferring lease to s1\nI180713 06:24:20.164168 162586 storage/replica_range_lease.go:617  [replicate,n4,s4,r25/2:/Table/5{4-5}] done transferring lease to s1: <nil>\nI180713 06:24:20.164607 157105 storage/replica_proposal.go:203  [n1,s1,r25/1:/Table/5{4-5}] new range lease repl=(n1,s1):1 seq=8 start=1531463060.163212894,0 epo=1 pro=1531463060.163216279,0 following repl=(n4,s4):2 seq=7 start=1531463051.694574684,0 epo=1 pro=1531463051.694578117,0\nI180713 06:24:20.697657 162479 storage/replica_range_lease.go:554  [replicate,n1,s1,r25/1:/Table/5{4-5}] transferring lease to s4\nI180713 06:24:20.698993 162479 storage/replica_range_lease.go:617  [replicate,n1,s1,r25/1:/Table/5{4-5}] done transferring lease to s4: <nil>\nI180713 06:24:20.699696 158085 storage/replica_proposal.go:203  [n4,s4,r25/2:/Table/5{4-5}] new range lease repl=(n4,s4):2 seq=9 start=1531463060.697673403,0 epo=1 pro=1531463060.697676329,0 following repl=(n1,s1):1 seq=8 start=1531463060.163212894,0 epo=1 pro=1531463060.163216279,0\nI180713 06:24:23.677552 157166 gossip/gossip.go:509  [n1] gossip status (ok, 4 nodes)\ngossip client (0/3 cur/max conns)\ngossip server (3/3 cur/max conns, infos 4061/159 sent/received, bytes 2824973B/59432B sent/received)\n  2: 127.0.0.1:42827 (1m0s)\n  3: 127.0.0.1:46247 (1m0s)\n  4: 127.0.0.1:34649 (1m0s)\nI180713 06:24:23.677780 157169 server/status/runtime.go:255  [n1] runtime stats: 243 MiB RSS, 1144 goroutines, 42 MiB/41 MiB/112 MiB GO alloc/idle/total, 56 MiB/97 MiB CGO alloc/total, 79.20cgo/sec, 0.03/0.00 %(u/s)time, 0.00 %gc (1x)\nI180713 06:24:23.968832 157558 server/status/runtime.go:255  [n2] runtime stats: 243 MiB RSS, 1144 goroutines, 48 MiB/36 MiB/112 MiB GO alloc/idle/total, 56 MiB/97 MiB CGO alloc/total, 83.26cgo/sec, 0.03/0.00 %(u/s)time, 0.00 %gc (1x)\nI180713 06:24:23.972062 157555 gossip/gossip.go:509  [n2] gossip status (ok, 4 nodes)\ngossip client (1/3 cur/max conns)\n  1: 127.0.0.1:35815 (1m0s: infos 6/3198 sent/received, bytes 957B/2225492B sent/received)\ngossip server (0/3 cur/max conns, infos 6/3198 sent/received, bytes 957B/2225492B sent/received)\nI180713 06:24:24.054885 157813 server/status/runtime.go:255  [n3] runtime stats: 243 MiB RSS, 1156 goroutines, 54 MiB/30 MiB/112 MiB GO alloc/idle/total, 56 MiB/97 MiB CGO alloc/total, 86.77cgo/sec, 0.03/0.00 %(u/s)time, 0.00 %gc (1x)\nI180713 06:24:24.055090 157810 gossip/gossip.go:509  [n3] gossip status (ok, 4 nodes)\ngossip client (1/3 cur/max conns)\n  1: 127.0.0.1:35815 (1m0s: infos 73/447 sent/received, bytes 13865B/406140B sent/received)\ngossip server (0/3 cur/max conns, infos 73/447 sent/received, bytes 13865B/406140B sent/received)\nI180713 06:24:24.123378 158027 gossip/gossip.go:509  [n4] gossip status (ok, 4 nodes)\ngossip client (1/3 cur/max conns)\n  1: 127.0.0.1:35815 (1m0s: infos 80/463 sent/received, bytes 44610B/209269B sent/received)\ngossip server (0/3 cur/max conns, infos 80/463 sent/received, bytes 44610B/209269B sent/received)\nI180713 06:24:24.123819 158030 server/status/runtime.go:255  [n4] runtime stats: 243 MiB RSS, 1154 goroutines, 34 MiB/47 MiB/112 MiB GO alloc/idle/total, 56 MiB/97 MiB CGO alloc/total, 90.20cgo/sec, 0.03/0.00 %(u/s)time, 0.00 %gc (1x)\nI180713 06:24:33.677902 157169 server/status/runtime.go:255  [n1] runtime stats: 246 MiB RSS, 1154 goroutines, 44 MiB/39 MiB/112 MiB GO alloc/idle/total, 56 MiB/97 MiB CGO alloc/total, 76.00cgo/sec, 0.03/0.00 %(u/s)time, 0.00 %gc (1x)\nI180713 06:24:33.963898 157558 server/status/runtime.go:255  [n2] runtime stats: 246 MiB RSS, 1154 goroutines, 49 MiB/34 MiB/112 MiB GO alloc/idle/total, 56 MiB/97 MiB CGO alloc/total, 72.54cgo/sec, 0.03/0.00 %(u/s)time, 0.00 %gc (1x)\nI180713 06:24:34.051823 157813 server/status/runtime.go:255  [n3] runtime stats: 246 MiB RSS, 1154 goroutines, 55 MiB/29 MiB/112 MiB GO alloc/idle/total, 56 MiB/97 MiB CGO alloc/total, 69.02cgo/sec, 0.03/0.00 %(u/s)time, 0.00 %gc (1x)\nI180713 06:24:34.123511 158030 server/status/runtime.go:255  [n4] runtime stats: 248 MiB RSS, 1154 goroutines, 33 MiB/49 MiB/112 MiB GO alloc/idle/total, 56 MiB/97 MiB CGO alloc/total, 65.50cgo/sec, 0.03/0.00 %(u/s)time, 0.00 %gc (1x)\nI180713 06:24:41.176138 163062 storage/replica_range_lease.go:554  [replicate,n4,s4,r25/2:/Table/5{4-5}] transferring lease to s1\nI180713 06:24:41.177204 163062 storage/replica_range_lease.go:617  [replicate,n4,s4,r25/2:/Table/5{4-5}] done transferring lease to s1: <nil>\nI180713 06:24:41.177740 157121 storage/replica_proposal.go:203  [n1,s1,r25/1:/Table/5{4-5}] new range lease repl=(n1,s1):1 seq=10 start=1531463081.176155553,0 epo=1 pro=1531463081.176159917,0 following repl=(n4,s4):2 seq=9 start=1531463060.697673403,0 epo=1 pro=1531463060.697676329,0\nI180713 06:24:42.724036 163081 storage/replica_range_lease.go:554  [replicate,n1,s1,r25/1:/Table/5{4-5}] transferring lease to s4\nI180713 06:24:42.725527 163081 storage/replica_range_lease.go:617  [replicate,n1,s1,r25/1:/Table/5{4-5}] done transferring lease to s4: <nil>\nI180713 06:24:42.727090 158133 storage/replica_proposal.go:203  [n4,s4,r25/2:/Table/5{4-5}] new range lease repl=(n4,s4):2 seq=11 start=1531463082.724051871,0 epo=1 pro=1531463082.724056744,0 following repl=(n1,s1):1 seq=10 start=1531463081.176155553,0 epo=1 pro=1531463081.176159917,0\nI180713 06:24:43.677903 157169 server/status/runtime.go:255  [n1] runtime stats: 248 MiB RSS, 1154 goroutines, 43 MiB/40 MiB/112 MiB GO alloc/idle/total, 56 MiB/97 MiB CGO alloc/total, 80.40cgo/sec, 0.03/0.00 %(u/s)time, 0.00 %gc (1x)\nI180713 06:24:43.972650 157558 server/status/runtime.go:255  [n2] runtime stats: 248 MiB RSS, 1154 goroutines, 48 MiB/35 MiB/112 MiB GO alloc/idle/total, 56 MiB/97 MiB CGO alloc/total, 79.37cgo/sec, 0.03/0.00 %(u/s)time, 0.00 %gc (1x)\nI180713 06:24:44.051925 157813 server/status/runtime.go:255  [n3] runtime stats: 248 MiB RSS, 1154 goroutines, 54 MiB/30 MiB/112 MiB GO alloc/idle/total, 56 MiB/97 MiB CGO alloc/total, 80.40cgo/sec, 0.03/0.00 %(u/s)time, 0.00 %gc (1x)\nI180713 06:24:44.123667 158030 server/status/runtime.go:255  [n4] runtime stats: 248 MiB RSS, 1154 goroutines, 31 MiB/50 MiB/112 MiB GO alloc/idle/total, 56 MiB/97 MiB CGO alloc/total, 80.40cgo/sec, 0.02/0.00 %(u/s)time, 0.00 %gc (1x)\nI180713 06:24:53.678466 157169 server/status/runtime.go:255  [n1] runtime stats: 248 MiB RSS, 1154 goroutines, 41 MiB/43 MiB/112 MiB GO alloc/idle/total, 56 MiB/97 MiB CGO alloc/total, 68.00cgo/sec, 0.02/0.00 %(u/s)time, 0.00 %gc (1x)\nI180713 06:24:53.966745 157558 server/status/runtime.go:255  [n2] runtime stats: 248 MiB RSS, 1154 goroutines, 47 MiB/37 MiB/112 MiB GO alloc/idle/total, 56 MiB/97 MiB CGO alloc/total, 71.91cgo/sec, 0.02/0.00 %(u/s)time, 0.00 %gc (1x)\nI180713 06:24:54.052020 157813 server/status/runtime.go:255  [n3] runtime stats: 248 MiB RSS, 1154 goroutines, 53 MiB/32 MiB/112 MiB GO alloc/idle/total, 56 MiB/97 MiB CGO alloc/total, 75.90cgo/sec, 0.02/0.00 %(u/s)time, 0.00 %gc (1x)\nI180713 06:24:54.123530 158030 server/status/runtime.go:255  [n4] runtime stats: 248 MiB RSS, 1154 goroutines, 32 MiB/49 MiB/112 MiB GO alloc/idle/total, 56 MiB/97 MiB CGO alloc/total, 83.90cgo/sec, 0.02/0.01 %(u/s)time, 0.00 %gc (1x)\n    --- FAIL: TestShowTraceReplica/DELETE_FROM_d.t2 (45.30s)\n    \tshow_trace_replica_test.go:101: condition failed to evaluate within 45s: DELETE FROM d.t2: got [[4 4] [4 4]] expected [[2 2] [2 2]]\n    \t\tgoroutine 162273 [running]:\n    \t\truntime/debug.Stack(0xa7a358200, 0xc423447f78, 0x2933da0)\n    \t\t\t/usr/local/go/src/runtime/debug/stack.go:24 +0xa7\n    \t\tgithub.com/cockroachdb/cockroach/pkg/testutils.SucceedsSoon(0x2980a20, 0xc425a62000, 0xc42328df78)\n    \t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/testutils/soon.go:38 +0x7d\n    \t\tgithub.com/cockroachdb/cockroach/pkg/sql_test.TestShowTraceReplica.func2(0xc425a62000)\n    \t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/sql/show_trace_replica_test.go:101 +0x64\n    \t\ttesting.tRunner(0xc425a62000, 0xc4203ec020)\n    \t\t\t/usr/local/go/src/testing/testing.go:777 +0xd0\n    \t\tcreated by testing.(*T).Run\n    \t\t\t/usr/local/go/src/testing/testing.go:824 +0x2e0\n\n```"},"comment":{"url":"https://api.github.com/repos/cockroachdb/cockroach/issues/comments/404854545","html_url":"https://github.com/cockroachdb/cockroach/issues/27497#issuecomment-404854545","issue_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/27497","id":404854545,"node_id":"MDEyOklzc3VlQ29tbWVudDQwNDg1NDU0NQ==","user":{"login":"petermattis","id":6201335,"node_id":"MDQ6VXNlcjYyMDEzMzU=","avatar_url":"https://avatars0.githubusercontent.com/u/6201335?v=4","gravatar_id":"","url":"https://api.github.com/users/petermattis","html_url":"https://github.com/petermattis","followers_url":"https://api.github.com/users/petermattis/followers","following_url":"https://api.github.com/users/petermattis/following{/other_user}","gists_url":"https://api.github.com/users/petermattis/gists{/gist_id}","starred_url":"https://api.github.com/users/petermattis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/petermattis/subscriptions","organizations_url":"https://api.github.com/users/petermattis/orgs","repos_url":"https://api.github.com/users/petermattis/repos","events_url":"https://api.github.com/users/petermattis/events{/privacy}","received_events_url":"https://api.github.com/users/petermattis/received_events","type":"User","site_admin":false},"created_at":"2018-07-13T14:44:27Z","updated_at":"2018-07-13T14:44:27Z","author_association":"CONTRIBUTOR","body":"I'm not able to reproduce any of these failures."}}
{"actor":{"display_login":"petermattis"}}