{"action":"created","issue":{"url":"https://api.github.com/repos/cockroachdb/cockroach/issues/9029","repository_url":"https://api.github.com/repos/cockroachdb/cockroach","labels_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/9029/labels{/name}","comments_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/9029/comments","events_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/9029/events","html_url":"https://github.com/cockroachdb/cockroach/issues/9029","id":174559195,"number":9029,"title":"acceptance: Raft panic in TestRebalance_3To5Small","user":{"login":"cuongdo","id":1564582,"avatar_url":"https://avatars.githubusercontent.com/u/1564582?v=3","gravatar_id":"","url":"https://api.github.com/users/cuongdo","html_url":"https://github.com/cuongdo","followers_url":"https://api.github.com/users/cuongdo/followers","following_url":"https://api.github.com/users/cuongdo/following{/other_user}","gists_url":"https://api.github.com/users/cuongdo/gists{/gist_id}","starred_url":"https://api.github.com/users/cuongdo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cuongdo/subscriptions","organizations_url":"https://api.github.com/users/cuongdo/orgs","repos_url":"https://api.github.com/users/cuongdo/repos","events_url":"https://api.github.com/users/cuongdo/events{/privacy}","received_events_url":"https://api.github.com/users/cuongdo/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/cockroachdb/cockroach/labels/stability","name":"stability","color":"b60205"},{"url":"https://api.github.com/repos/cockroachdb/cockroach/labels/test-failure","name":"test-failure","color":"e11d21"}],"state":"open","locked":false,"assignee":{"login":"bdarnell","id":160562,"avatar_url":"https://avatars.githubusercontent.com/u/160562?v=3","gravatar_id":"","url":"https://api.github.com/users/bdarnell","html_url":"https://github.com/bdarnell","followers_url":"https://api.github.com/users/bdarnell/followers","following_url":"https://api.github.com/users/bdarnell/following{/other_user}","gists_url":"https://api.github.com/users/bdarnell/gists{/gist_id}","starred_url":"https://api.github.com/users/bdarnell/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bdarnell/subscriptions","organizations_url":"https://api.github.com/users/bdarnell/orgs","repos_url":"https://api.github.com/users/bdarnell/repos","events_url":"https://api.github.com/users/bdarnell/events{/privacy}","received_events_url":"https://api.github.com/users/bdarnell/received_events","type":"User","site_admin":false},"assignees":[{"login":"bdarnell","id":160562,"avatar_url":"https://avatars.githubusercontent.com/u/160562?v=3","gravatar_id":"","url":"https://api.github.com/users/bdarnell","html_url":"https://github.com/bdarnell","followers_url":"https://api.github.com/users/bdarnell/followers","following_url":"https://api.github.com/users/bdarnell/following{/other_user}","gists_url":"https://api.github.com/users/bdarnell/gists{/gist_id}","starred_url":"https://api.github.com/users/bdarnell/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bdarnell/subscriptions","organizations_url":"https://api.github.com/users/bdarnell/orgs","repos_url":"https://api.github.com/users/bdarnell/repos","events_url":"https://api.github.com/users/bdarnell/events{/privacy}","received_events_url":"https://api.github.com/users/bdarnell/received_events","type":"User","site_admin":false}],"milestone":null,"comments":17,"created_at":"2016-09-01T15:48:13Z","updated_at":"2016-09-29T00:33:39Z","closed_at":null,"body":"SHA: 24f6cdcb1b7373f1b667678172b5342bc43495b6\r\n\r\nThe test creates 3 nodes and restores 10GiB of archived store data to each node. After the nodes have their data, we add two new nodes to the cluster, which quickly results in the following panic on the first (original) node:\r\n\r\n```\r\nE160901 15:29:14.295604 565 raft/raft.go:1012  store=1:1 305{/Table/51/1/1214430205347752988/\"7282335d-1674-4545-b952-b564d217ec56\"/2382-/Table/51/1/1247684140858310777/\"e9a91de4-9dda-40a4-9d66-58f413f51946\"/1147}: 1 state.commit 347 is out of range [342, 345]\r\npanic: store=1:1 305{/Table/51/1/1214430205347752988/\"7282335d-1674-4545-b952-b564d217ec56\"/2382-/Table/51/1/1247684140858310777/\"e9a91de4-9dda-40a4-9d66-58f413f51946\"/1147}: 1 state.commit 347 is out of range [342, 345] [recovered]\r\n       \tpanic: store=1:1 305{/Table/51/1/1214430205347752988/\"7282335d-1674-4545-b952-b564d217ec56\"/2382-/Table/51/1/1247684140858310777/\"e9a91de4-9dda-40a4-9d66-58f413f51946\"/1147}: 1 state.commit 347 is out of range [342, 345]\r\n\r\ngoroutine 565 [running]:\r\npanic(0x1559060, 0xc4222c1e40)\r\n       \t/usr/local/go/src/runtime/panic.go:500 +0x1a1\r\ngithub.com/cockroachdb/cockroach/util/stop.(*Stopper).Recover(0xc42042f3b0)\r\n       \t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:176 +0x6e\r\npanic(0x1559060, 0xc4222c1e40)\r\n       \t/usr/local/go/src/runtime/panic.go:458 +0x243\r\ngithub.com/cockroachdb/cockroach/storage.(*raftLogger).Panicf(0xc4222c1a70, 0x171d4ec, 0x2b, 0xc42056e8c0, 0x4, 0x4)\r\n       \t/go/src/github.com/cockroachdb/cockroach/storage/raft.go:121 +0x204\r\ngithub.com/coreos/etcd/raft.(*raft).loadState(0xc420410c30, 0xd, 0x1, 0x15b, 0x0, 0x0, 0x0)\r\n       \t/go/src/github.com/coreos/etcd/raft/raft.go:1012 +0x1db\r\ngithub.com/coreos/etcd/raft.newRaft(0xc421de57c0, 0xc42015c040)\r\n       \t/go/src/github.com/coreos/etcd/raft/raft.go:252 +0xd1c\r\ngithub.com/coreos/etcd/raft.NewRawNode(0xc421de57c0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)\r\n       \t/go/src/github.com/coreos/etcd/raft/rawnode.go:76 +0x71\r\ngithub.com/cockroachdb/cockroach/storage.(*Replica).withRaftGroupLocked(0xc4202e7200, 0xc421de5ad8, 0x0, 0x0)\r\n       \t/go/src/github.com/cockroachdb/cockroach/storage/replica.go:376 +0x235\r\ngithub.com/cockroachdb/cockroach/storage.defaultProposeRaftCommandLocked(0xc4202e7200, 0xc421e320c0, 0x0, 0x0)\r\n       \t/go/src/github.com/cockroachdb/cockroach/storage/replica.go:1578 +0x1fa\r\ngithub.com/cockroachdb/cockroach/storage.(*Replica).proposePendingCmdLocked(0xc4202e7200, 0xc421e320c0, 0xc420014268, 0xc4222c1a38)\r\n       \t/go/src/github.com/cockroachdb/cockroach/storage/replica.go:1530 +0x77\r\ngithub.com/cockroachdb/cockroach/storage.(*Replica).proposeRaftCommand(0xc4202e7200, 0x7f7cd005bc70, 0xc420014268, 0x14703ca43ff361ab, 0x0, 0x0, 0x0, 0x131, 0x0, 0x0, ...)\r\n       \t/go/src/github.com/cockroachdb/cockroach/storage/replica.go:1509 +0x1f0\r\ngithub.com/cockroachdb/cockroach/storage.(*pendingLeaseRequest).InitOrJoinRequest.func1()\r\n       \t/go/src/github.com/cockroachdb/cockroach/storage/replica_range_lease.go:133 +0x1a9\r\ngithub.com/cockroachdb/cockroach/util/stop.(*Stopper).RunAsyncTask.func1(0xc42042f3b0, 0x1d772b3, 0x1e, 0xa8, 0xc420e911a0)\r\n       \t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:231 +0x95\r\ncreated by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunAsyncTask\r\n       \t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:232 +0xbc\r\n```\r\n\r\nTo repro:\r\n```\r\nmake test \\\r\n  TESTTIMEOUT=48h \\\r\n  PKG=./acceptance \\\r\n  TESTS=Rebalance_3To5Small \\\r\n  TESTFLAGS='-v -remote -key-name google_compute_engine -cwd terraform -tf.keep-cluster=failed'\r\n```\r\n\r\nFull log here:\r\n\r\n[cockroach.stderr.txt](https://github.com/cockroachdb/cockroach/files/450113/cockroach.stderr.txt)\r\n"},"comment":{"url":"https://api.github.com/repos/cockroachdb/cockroach/issues/comments/250340283","html_url":"https://github.com/cockroachdb/cockroach/issues/9029#issuecomment-250340283","issue_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/9029","id":250340283,"user":{"login":"petermattis","id":6201335,"avatar_url":"https://avatars.githubusercontent.com/u/6201335?v=3","gravatar_id":"","url":"https://api.github.com/users/petermattis","html_url":"https://github.com/petermattis","followers_url":"https://api.github.com/users/petermattis/followers","following_url":"https://api.github.com/users/petermattis/following{/other_user}","gists_url":"https://api.github.com/users/petermattis/gists{/gist_id}","starred_url":"https://api.github.com/users/petermattis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/petermattis/subscriptions","organizations_url":"https://api.github.com/users/petermattis/orgs","repos_url":"https://api.github.com/users/petermattis/repos","events_url":"https://api.github.com/users/petermattis/events{/privacy}","received_events_url":"https://api.github.com/users/petermattis/received_events","type":"User","site_admin":false},"created_at":"2016-09-29T00:33:39Z","updated_at":"2016-09-29T00:33:39Z","body":"Heh, I can't get 24f6cdcb1b7373f1b667678172b5342bc43495b6 to finish the test using a local 5-node cluster. The cockroach processes very quickly blow up to multiple gigabytes in size. \r\n\r\nLater: same runaway memory growth with a5ce7dd749642b60b15b9693b008af968c3f7553 (the commit that bumped us to RocksDB 4.9)."}}
{"actor":{"display_login":"petermattis"}}