{"action":"created","issue":{"url":"https://api.github.com/repos/cockroachdb/cockroach/issues/5821","repository_url":"https://api.github.com/repos/cockroachdb/cockroach","labels_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/5821/labels{/name}","comments_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/5821/comments","events_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/5821/events","html_url":"https://github.com/cockroachdb/cockroach/issues/5821","id":145393101,"number":5821,"title":"stability: replica is invalid: NodeID must not be zero","user":{"login":"mberhault","id":5901480,"avatar_url":"https://avatars.githubusercontent.com/u/5901480?v=3","gravatar_id":"","url":"https://api.github.com/users/mberhault","html_url":"https://github.com/mberhault","followers_url":"https://api.github.com/users/mberhault/followers","following_url":"https://api.github.com/users/mberhault/following{/other_user}","gists_url":"https://api.github.com/users/mberhault/gists{/gist_id}","starred_url":"https://api.github.com/users/mberhault/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mberhault/subscriptions","organizations_url":"https://api.github.com/users/mberhault/orgs","repos_url":"https://api.github.com/users/mberhault/repos","events_url":"https://api.github.com/users/mberhault/events{/privacy}","received_events_url":"https://api.github.com/users/mberhault/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2016-04-02T16:45:31Z","updated_at":"2016-04-04T19:30:47Z","closed_at":"2016-04-04T19:30:47Z","body":"Scenario: \r\n* beta cluster running for >1d with 4 nodes at sha b329564\r\n* load was photos + block_writer\r\n* added a 5th node running sha 636e1b9. joined cluster and received load.\r\n* clients starting failing with \"timestamp in the future\" problems.\r\n```\r\n2016/04/02 16:12:41 error running blockwriter 6a27d68d-5b48-4067-8138-5888648e6678: pq: storage/store.go:1541: rejecting command with timestamp in the future: 1459613560699437844 (255.85043ms ahead)\r\n```\r\n* forced ntp sync on all nodes. errors still happening.\r\n* started upgrading the first 4 nodes to sha 636e1b9\r\n\r\nFor a few minutes after being upgraded, nodes 1, 2, and 3 (but not 0 and 4) were logging:\r\n```\r\nE160402 16:24:34.083927 storage/queue.go:446  [replicate] range=83 [/Table/55/1/3155670939066101284/\r\n\"172ac2db-5c6e-46ef-87d6-8052593aacf0\"/183120-/Table/55/1/3227186889529930184/\"172ac2db-5c6e-46ef-87\r\nd6-8052593aacf0\"/67210): error: storage/replica_command.go:2211: change replicas of 83 failed: roach\r\npb/metadata.go:136: replica 3 is invalid: roachpb/metadata.go:153: NodeID must not be zero\r\nE160402 16:24:39.076993 storage/queue.go:446  [replicate] range=40 [/Table/55/1/3443414669130073203/\r\n\"172ac2db-5c6e-46ef-87d6-8052593aacf0\"/18400-/Table/55/1/3513651914126681548/\"4bc7b6af-0f79-43bd-a42\r\n1-5b721c21391a\"/181179): error: storage/replica_command.go:2211: change replicas of 40 failed: roach\r\npb/metadata.go:136: replica 3 is invalid: roachpb/metadata.go:153: NodeID must not be zero\r\n```\r\n\r\nIt's been a few minutes since the last one.\r\nFull logs attached: may be best to search for NodeID must not be zero\r\n[node0.log.txt.gz](https://github.com/cockroachdb/cockroach/files/200911/node0.log.txt.gz)\r\n[node1.log.txt.gz](https://github.com/cockroachdb/cockroach/files/200912/node1.log.txt.gz)\r\n[node2.log.txt.gz](https://github.com/cockroachdb/cockroach/files/200915/node2.log.txt.gz)\r\n[node3.log.txt.gz](https://github.com/cockroachdb/cockroach/files/200914/node3.log.txt.gz)\r\n[node4.log.txt.gz](https://github.com/cockroachdb/cockroach/files/200913/node4.log.txt.gz)\r\n"},"comment":{"url":"https://api.github.com/repos/cockroachdb/cockroach/issues/comments/205459954","html_url":"https://github.com/cockroachdb/cockroach/issues/5821#issuecomment-205459954","issue_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/5821","id":205459954,"user":{"login":"mberhault","id":5901480,"avatar_url":"https://avatars.githubusercontent.com/u/5901480?v=3","gravatar_id":"","url":"https://api.github.com/users/mberhault","html_url":"https://github.com/mberhault","followers_url":"https://api.github.com/users/mberhault/followers","following_url":"https://api.github.com/users/mberhault/following{/other_user}","gists_url":"https://api.github.com/users/mberhault/gists{/gist_id}","starred_url":"https://api.github.com/users/mberhault/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mberhault/subscriptions","organizations_url":"https://api.github.com/users/mberhault/orgs","repos_url":"https://api.github.com/users/mberhault/repos","events_url":"https://api.github.com/users/mberhault/events{/privacy}","received_events_url":"https://api.github.com/users/mberhault/received_events","type":"User","site_admin":false},"created_at":"2016-04-04T19:30:47Z","updated_at":"2016-04-04T19:30:47Z","body":"dup of #5844 "}}