{"action":"created","issue":{"url":"https://api.github.com/repos/cockroachdb/cockroach/issues/4687","repository_url":"https://api.github.com/repos/cockroachdb/cockroach","labels_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/4687/labels{/name}","comments_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/4687/comments","events_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/4687/events","html_url":"https://github.com/cockroachdb/cockroach/issues/4687","id":136691823,"number":4687,"title":"Consistency anomaly detected by a Jepsen bank transfer test","user":{"login":"knz","id":642886,"avatar_url":"https://avatars.githubusercontent.com/u/642886?v=3","gravatar_id":"","url":"https://api.github.com/users/knz","html_url":"https://github.com/knz","followers_url":"https://api.github.com/users/knz/followers","following_url":"https://api.github.com/users/knz/following{/other_user}","gists_url":"https://api.github.com/users/knz/gists{/gist_id}","starred_url":"https://api.github.com/users/knz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/knz/subscriptions","organizations_url":"https://api.github.com/users/knz/orgs","repos_url":"https://api.github.com/users/knz/repos","events_url":"https://api.github.com/users/knz/events{/privacy}","received_events_url":"https://api.github.com/users/knz/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2016-02-26T13:03:17Z","updated_at":"2016-02-26T13:22:40Z","closed_at":null,"body":"The bank test is a transfer test that should preserve the total sum. The anomaly is that the sum is not preserved.\r\n\r\nThe test scenario, as encoded in https://github.com/cockroachdb/jepsen/blob/master/cockroachdb/src/jepsen/cockroach.clj#L839:\r\n\r\n```sql\r\ncreate table accounts (id int not null primary key, balance bigint not null)\r\ninsert into accounts values (0, 10)\r\ninsert into accounts values (1, 10)\r\ninsert into accounts values (2, 10)\r\ninsert into accounts values (3, 10)\r\n-- in a loop: transfer(rand, rand, rand) where:\r\n   transfer(From, To, Amount) =\r\n      begin\r\n         b1 := (select balance from accounts where id = From) - Amount\r\n         b2 := (select balance from accounts where id = To) + Amount\r\n         if b1 >= 0 and b2 >= 0 then\r\n            update accounts set balance = b1 where id = From\r\n            update accounts set balance = b2 where id = To\r\n         endif\r\n      commit\r\n```\r\n\r\nAnd then also `select balance from accounts` from time to time to\r\ncheck the intermediate states of the table. That's how the inconsistency is detected.\r\n\r\nThe errors/traces:\r\nhttp://52.91.194.28/cgi-bin/display.py?path=cockroachdb-bank%2F20160226T122420.000Z%2Fresults.edn\r\nhttp://52.91.194.28/cockroachdb-bank/20160226T122420.000Z/history.txt\r\n\r\nhttp://52.91.194.28/cgi-bin/display.py?path=cockroachdb-bank%2F20160226T121719.000Z%2Fresults.edn\r\nhttp://52.91.194.28/cockroachdb-bank/20160226T121719.000Z/history.txt\r\n\r\nhttp://52.91.194.28/cgi-bin/display.py?path=cockroachdb-bank%2F20160226T121608.000Z%2Fresults.edn\r\nhttp://52.91.194.28/cockroachdb-bank/20160226T121608.000Z/history.txt\r\n\r\nIn all 3 cases the state becomes inconsistent at about the time a network partition is resolved (`:nemesis :info :stop \"fully connected\"`)\r\n\r\nNothing special in the server logs (all available from the UI in the rightmost column at http://52.91.194.28/)\r\n\r\nPerhaps not relevant, but I also noticed the following oddity on one node:\r\n\r\n```\r\nW160226 12:24:45.486123 storage/replica.go:1774 could not GC completed transaction: context deadline exceeded\r\nW160226 12:24:47.699457 storage/replica.go:1873 unable to resolve local intents; range 0: replica <nil> not leader; leader is <nil>\r\n```\r\nSource: http://52.91.194.28/cgi-bin/display.py?path=cockroachdb-bank%2F20160226T122420.000Z%2Fn1l%2Fcockroach.stderr\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/cockroachdb/cockroach/issues/comments/189270543","html_url":"https://github.com/cockroachdb/cockroach/issues/4687#issuecomment-189270543","issue_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/4687","id":189270543,"user":{"login":"tschottdorf","id":5076964,"avatar_url":"https://avatars.githubusercontent.com/u/5076964?v=3","gravatar_id":"","url":"https://api.github.com/users/tschottdorf","html_url":"https://github.com/tschottdorf","followers_url":"https://api.github.com/users/tschottdorf/followers","following_url":"https://api.github.com/users/tschottdorf/following{/other_user}","gists_url":"https://api.github.com/users/tschottdorf/gists{/gist_id}","starred_url":"https://api.github.com/users/tschottdorf/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tschottdorf/subscriptions","organizations_url":"https://api.github.com/users/tschottdorf/orgs","repos_url":"https://api.github.com/users/tschottdorf/repos","events_url":"https://api.github.com/users/tschottdorf/events{/privacy}","received_events_url":"https://api.github.com/users/tschottdorf/received_events","type":"User","site_admin":false},"created_at":"2016-02-26T13:22:40Z","updated_at":"2016-02-26T13:22:40Z","body":"Against which revision? "}}