{"action":"created","issue":{"url":"https://api.github.com/repos/cockroachdb/cockroach/issues/6991","repository_url":"https://api.github.com/repos/cockroachdb/cockroach","labels_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/6991/labels{/name}","comments_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/6991/comments","events_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/6991/events","html_url":"https://github.com/cockroachdb/cockroach/issues/6991","id":157961848,"number":6991,"title":"stability: resurrecting registration cluster","user":{"login":"mberhault","id":5901480,"avatar_url":"https://avatars.githubusercontent.com/u/5901480?v=3","gravatar_id":"","url":"https://api.github.com/users/mberhault","html_url":"https://github.com/mberhault","followers_url":"https://api.github.com/users/mberhault/followers","following_url":"https://api.github.com/users/mberhault/following{/other_user}","gists_url":"https://api.github.com/users/mberhault/gists{/gist_id}","starred_url":"https://api.github.com/users/mberhault/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mberhault/subscriptions","organizations_url":"https://api.github.com/users/mberhault/orgs","repos_url":"https://api.github.com/users/mberhault/repos","events_url":"https://api.github.com/users/mberhault/events{/privacy}","received_events_url":"https://api.github.com/users/mberhault/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":20,"created_at":"2016-06-01T17:21:43Z","updated_at":"2016-07-01T15:16:23Z","closed_at":null,"body":"@bdarnell: I'll be keeping track of actions and results here.\r\n\r\nQuick summary:\r\nthe registration cluster is falling over repeatedly due to large snapshot sizes. Specifically, recipients of range 1 snapshots OOM during `applySnapshot`. \r\n\r\neg, on node 2 `ec2-52-91-3-164.compute-1.amazonaws.com`:\r\n```\r\nI160601 16:24:14.957105 storage/replica_raftstorage.go:610  received snapshot for range 1 at index 6818966. encoded size=1204695315, 14475 KV pairs, 384994 log entries\r\n...\r\nI160601 16:24:25.479455 /go/src/google.golang.org/grpc/clientconn.go:499  grpc: Conn.resetTransport failed to create client transport: connection error: desc = \"transport: dial tcp 172.31.14.204:26257: getsockopt: connection refused\"; Reconnecting to \"ip-172-31-14-204:26257\"\r\nterminate called after throwing an instance of 'std::bad_alloc'\r\n  what():  std::bad_alloc\r\nSIGABRT: abort\r\nPC=0x7f78fae4acc9 m=7\r\nsignal arrived during cgo execution\r\n\r\ngoroutine 86 [syscall, locked to thread]:\r\nruntime.cgocall(0x11b2d80, 0xc822780308, 0x7f7800000000)\r\n        /usr/local/go/src/runtime/cgocall.go:123 +0x11b fp=0xc8227802b0 sp=0xc822780280\r\ngithub.com/cockroachdb/cockroach/storage/engine._Cfunc_DBApplyBatchRepr(0x7f78ee825b90, 0xc917dce000, 0x208f, 0x0, 0x0)\r\n        ??:0 +0x53 fp=0xc822780308 sp=0xc8227802b0\r\ngithub.com/cockroachdb/cockroach/storage/engine.dbApplyBatchRepr(0x7f78ee825b90, 0xc917dce000, 0x208f, 0x4000000, 0x0, 0x0)\r\n        /go/src/github.com/cockroachdb/cockroach/storage/engine/rocksdb.go:990 +0x138 fp=0xc8227803a0 sp=0xc822780308\r\ngithub.com/cockroachdb/cockroach/storage/engine.(*rocksDBBatch).ApplyBatchRepr(0xc822348000, 0xc917dce000, 0x208f, 0x4000000, 0x0, 0x0)\r\n        /go/src/github.com/cockroachdb/cockroach/storage/engine/rocksdb.go:578 +0x4f fp=0xc8227803d8 sp=0xc8227803a0\r\ngithub.com/cockroachdb/cockroach/storage/engine.(*rocksDBBatch).flushMutations(0xc822348000)\r\n        /go/src/github.com/cockroachdb/cockroach/storage/engine/rocksdb.go:672 +0x146 fp=0xc822780468 sp=0xc8227803d8\r\ngithub.com/cockroachdb/cockroach/storage/engine.(*rocksDBBatchIterator).Seek(0xc822348030, 0xc934324b80, 0x10, 0x20, 0x0, 0x0)\r\n        /go/src/github.com/cockroachdb/cockroach/storage/engine/rocksdb.go:498 +0x25 fp=0xc8227804a0 sp=0xc822780468\r\ngithub.com/cockroachdb/cockroach/storage/engine.mvccGetMetadata(0x7f78fbaca080, 0xc822348030, 0xc934324b80, 0x10, 0x20, 0x0, 0x0, 0xc938fc6000, 0xc938fc6000, 0x11, ...)\r\n        /go/src/github.com/cockroachdb/cockroach/storage/engine/mvcc.go:684 +0xcf fp=0xc8227805d8 sp=0xc8227804a0\r\ngithub.com/cockroachdb/cockroach/storage/engine.mvccPutInternal(0x7f78fba528b0, 0xc82000ef48, 0x7f78fbaca120, 0xc822348000, 0x7f78fbaca080, 0xc822348030, 0x0, 0xc934324b80, 0x10, 0x20, ...)\r\n        /go/src/github.com/cockroachdb/cockroach/storage/engine/mvcc.go:1020 +0x1f9 fp=0xc822780b20 sp=0xc8227805d8\r\ngithub.com/cockroachdb/cockroach/storage/engine.mvccPutUsingIter(0x7f78fba528b0, 0xc82000ef48, 0x7f78fbaca120, 0xc822348000, 0x7f78fbaca080, 0xc822348030, 0x0, 0xc934324b80, 0x10, 0x20, ...)\r\n        /go/src/github.com/cockroachdb/cockroach/storage/engine/mvcc.go:988 +0x1bb fp=0xc822780bf8 sp=0xc822780b20\r\ngithub.com/cockroachdb/cockroach/storage/engine.MVCCPut(0x7f78fba528b0, 0xc82000ef48, 0x7f78fbac9f98, 0xc822348000, 0x0, 0xc934324b80, 0x10, 0x20, 0x0, 0xc800000000, ...)\r\n        /go/src/github.com/cockroachdb/cockroach/storage/engine/mvcc.go:926 +0x1bb fp=0xc822780cc8 sp=0xc822780bf8\r\ngithub.com/cockroachdb/cockroach/storage/engine.MVCCPutProto(0x7f78fba528b0, 0xc82000ef48, 0x7f78fbac9f98, 0xc822348000, 0x0, 0xc934324b80, 0x10, 0x20, 0x0, 0x0, ...)\r\n        /go/src/github.com/cockroachdb/cockroach/storage/engine/mvcc.go:549 +0x1c5 fp=0xc822780d98 sp=0xc822780cc8\r\ngithub.com/cockroachdb/cockroach/storage.(*Replica).append(0xc8237fd9e0, 0x7f78fbac9f98, 0xc822348000, 0x0, 0xc8f810e000, 0x5dfe2, 0x5dfe2, 0x1626, 0x0, 0x0)\r\n        /go/src/github.com/cockroachdb/cockroach/storage/replica_raftstorage.go:547 +0x20f fp=0xc822780ed8 sp=0xc822780d98\r\ngithub.com/cockroachdb/cockroach/storage.(*Replica).applySnapshot(0xc8237fd9e0, 0x7f78fbac9f08, 0xc822348000, 0xc88f6b6000, 0x47ce3113, 0x47ce4000, 0xc821b06cc0, 0x4, 0x4, 0x0, ...)\r\n        /go/src/github.com/cockroachdb/cockroach/storage/replica_raftstorage.go:670 +0xc06 fp=0xc8227812f8 sp=0xc822780ed8\r\ngithub.com/cockroachdb/cockroach/storage.(*Replica).handleRaftReady(0xc8237fd9e0, 0x0, 0x0)\r\n        /go/src/github.com/cockroachdb/cockroach/storage/replica.go:1425 +0x29e fp=0xc822781bd0 sp=0xc8227812f8\r\ngithub.com/cockroachdb/cockroach/storage.(*Store).processRaft.func1()\r\n        /go/src/github.com/cockroachdb/cockroach/storage/store.go:2055 +0x35d fp=0xc822781f60 sp=0xc822781bd0\r\ngithub.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8202efce0, 0xc82206b240)\r\n        /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:139 +0x52 fp=0xc822781f80 sp=0xc822781f60\r\nruntime.goexit()\r\n        /usr/local/go/src/runtime/asm_amd64.s:1998 +0x1 fp=0xc822781f88 sp=0xc822781f80\r\ncreated by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker\r\n        /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:140 +0x62\r\n\r\n```\r\n\r\nThere is no corresponding \"applied snapshot for range 1\" message, and the stack trace does list an `applySnapshot` entry. Can't confirm from the trace that it is for that range (the range ID is not one of the simple arguments), but it most likely is. Similar pattern appeared multiple times.\r\n\r\nI will perform the following to try to resurrect the cluster:\r\n* stop all nodes\r\n* backup all rocksdb data\r\n* change the default zone config to have only two replicas"},"comment":{"url":"https://api.github.com/repos/cockroachdb/cockroach/issues/comments/229973533","html_url":"https://github.com/cockroachdb/cockroach/issues/6991#issuecomment-229973533","issue_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/6991","id":229973533,"user":{"login":"petermattis","id":6201335,"avatar_url":"https://avatars.githubusercontent.com/u/6201335?v=3","gravatar_id":"","url":"https://api.github.com/users/petermattis","html_url":"https://github.com/petermattis","followers_url":"https://api.github.com/users/petermattis/followers","following_url":"https://api.github.com/users/petermattis/following{/other_user}","gists_url":"https://api.github.com/users/petermattis/gists{/gist_id}","starred_url":"https://api.github.com/users/petermattis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/petermattis/subscriptions","organizations_url":"https://api.github.com/users/petermattis/orgs","repos_url":"https://api.github.com/users/petermattis/repos","events_url":"https://api.github.com/users/petermattis/events{/privacy}","received_events_url":"https://api.github.com/users/petermattis/received_events","type":"User","site_admin":false},"created_at":"2016-07-01T15:16:23Z","updated_at":"2016-07-01T15:16:23Z","body":"If you're running insecure you can do: `ssh -L 8080:localhost:8080 <some-machine>`. I did this earlier today without difficulty."}}
{"actor":{"display_login":"petermattis"}}