{"action":"created","issue":{"url":"https://api.github.com/repos/scylladb/scylla/issues/3557","repository_url":"https://api.github.com/repos/scylladb/scylla","labels_url":"https://api.github.com/repos/scylladb/scylla/issues/3557/labels{/name}","comments_url":"https://api.github.com/repos/scylladb/scylla/issues/3557/comments","events_url":"https://api.github.com/repos/scylladb/scylla/issues/3557/events","html_url":"https://github.com/scylladb/scylla/issues/3557","id":336590243,"node_id":"MDU6SXNzdWUzMzY1OTAyNDM=","number":3557,"title":"Consistency issues","user":{"login":"jbdalido","id":2461655,"node_id":"MDQ6VXNlcjI0NjE2NTU=","avatar_url":"https://avatars1.githubusercontent.com/u/2461655?v=4","gravatar_id":"","url":"https://api.github.com/users/jbdalido","html_url":"https://github.com/jbdalido","followers_url":"https://api.github.com/users/jbdalido/followers","following_url":"https://api.github.com/users/jbdalido/following{/other_user}","gists_url":"https://api.github.com/users/jbdalido/gists{/gist_id}","starred_url":"https://api.github.com/users/jbdalido/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jbdalido/subscriptions","organizations_url":"https://api.github.com/users/jbdalido/orgs","repos_url":"https://api.github.com/users/jbdalido/repos","events_url":"https://api.github.com/users/jbdalido/events{/privacy}","received_events_url":"https://api.github.com/users/jbdalido/received_events","type":"User","site_admin":false},"labels":[{"id":160570956,"node_id":"MDU6TGFiZWwxNjA1NzA5NTY=","url":"https://api.github.com/repos/scylladb/scylla/labels/bug","name":"bug","color":"eb6420","default":true}],"state":"open","locked":false,"assignee":{"login":"nyh","id":584227,"node_id":"MDQ6VXNlcjU4NDIyNw==","avatar_url":"https://avatars3.githubusercontent.com/u/584227?v=4","gravatar_id":"","url":"https://api.github.com/users/nyh","html_url":"https://github.com/nyh","followers_url":"https://api.github.com/users/nyh/followers","following_url":"https://api.github.com/users/nyh/following{/other_user}","gists_url":"https://api.github.com/users/nyh/gists{/gist_id}","starred_url":"https://api.github.com/users/nyh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nyh/subscriptions","organizations_url":"https://api.github.com/users/nyh/orgs","repos_url":"https://api.github.com/users/nyh/repos","events_url":"https://api.github.com/users/nyh/events{/privacy}","received_events_url":"https://api.github.com/users/nyh/received_events","type":"User","site_admin":false},"assignees":[{"login":"nyh","id":584227,"node_id":"MDQ6VXNlcjU4NDIyNw==","avatar_url":"https://avatars3.githubusercontent.com/u/584227?v=4","gravatar_id":"","url":"https://api.github.com/users/nyh","html_url":"https://github.com/nyh","followers_url":"https://api.github.com/users/nyh/followers","following_url":"https://api.github.com/users/nyh/following{/other_user}","gists_url":"https://api.github.com/users/nyh/gists{/gist_id}","starred_url":"https://api.github.com/users/nyh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nyh/subscriptions","organizations_url":"https://api.github.com/users/nyh/orgs","repos_url":"https://api.github.com/users/nyh/repos","events_url":"https://api.github.com/users/nyh/events{/privacy}","received_events_url":"https://api.github.com/users/nyh/received_events","type":"User","site_admin":false}],"milestone":null,"comments":23,"created_at":"2018-06-28T11:56:41Z","updated_at":"2018-07-03T19:52:24Z","closed_at":null,"author_association":"NONE","body":"Hey everyone, we've upgraded a dual-dc cluster in 2.1.5 from 2.0.4 and we have huge consistency issues. Wwe already had the issue with 2.0.4 and we tried upgrading to mitigate without success,\r\n\r\nIt's critical as it's a full production cluster\r\n\r\n### Scenarii:\r\n\r\n1 - full upgrade + repair -pr on each nodes on every clusters\r\n2 - selecting a row in local_quorum on dc1: row is present\r\n3 - selecting the same row in local_quorum on dc2:  row is not present\r\n4 - selecting the same row in quorum : row is present\r\n5 - re run repair -pr on every node\r\n6 - return to 1 with another row same result\r\n7 - script a full crawl to read every row in ALL to have the data repaired\r\n8 - counting the rows in local quorum on each DC, 30k differences between the two, still missing rows checked manually.\r\n\r\n### Detail of the script:\r\nChecking every row, and for each uuid checking that:\r\n-  friend A is friend with B  == one row \r\n- friend B is friend with A == one row. \r\n\r\nThe result we have is 90k row on cluster B does not have its counterpart, and only 58k on the production DC. Something interesting, the data is supposedly missing from the cluster we are writing on (We use DC-2 in local_quorum write and reads) and we have more data in DC-1.\r\n\r\nI've talked with @tgrabiec on Slack and we tried to clear the cache of the replicas on a missing row. After clearing the cache the data was back.\r\n\r\nWe've mitigated the issue for now by disabling all query made in local_* to quorum to ensure we have the data but we're not even sure it's working. Clusters are not overloaded, we have no issues whatsoever with timeouts or reads unavailable. The only problem we have is as soon as the cache builds up (a few days) we have every single nodes outputting  the same error:\r\n\r\n```batchlog_manager - Exception in batch replay: exceptions::read_timeout_exception (Operation timed out - received only 0 responses.)```\r\n\r\nWhen this issue arise, our max reactor load per node is at 100/ once we restart the entire cluster, max is down to 20/25 steady until the cache builds up, then same issue again, we have this since 1.7.\r\n\r\n### Details\r\n\r\n*Installation details*\r\nScylla version (or git commit hash):\r\nCluster size: 14\r\nOS (RHEL/CentOS/Ubuntu/AWS AMI): Ubuntu\r\n\r\n*Hardware details (for performance issues)*        \r\nPlatform (physical/VM/cloud instance type/docker): gcloud / 10 cpu 70Ggb RAM\r\nDisks: (SSD/HDD, count): 2 local SSD (1 commitlog, 1 data )\r\n"},"comment":{"url":"https://api.github.com/repos/scylladb/scylla/issues/comments/402273724","html_url":"https://github.com/scylladb/scylla/issues/3557#issuecomment-402273724","issue_url":"https://api.github.com/repos/scylladb/scylla/issues/3557","id":402273724,"node_id":"MDEyOklzc3VlQ29tbWVudDQwMjI3MzcyNA==","user":{"login":"nyh","id":584227,"node_id":"MDQ6VXNlcjU4NDIyNw==","avatar_url":"https://avatars3.githubusercontent.com/u/584227?v=4","gravatar_id":"","url":"https://api.github.com/users/nyh","html_url":"https://github.com/nyh","followers_url":"https://api.github.com/users/nyh/followers","following_url":"https://api.github.com/users/nyh/following{/other_user}","gists_url":"https://api.github.com/users/nyh/gists{/gist_id}","starred_url":"https://api.github.com/users/nyh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nyh/subscriptions","organizations_url":"https://api.github.com/users/nyh/orgs","repos_url":"https://api.github.com/users/nyh/repos","events_url":"https://api.github.com/users/nyh/events{/privacy}","received_events_url":"https://api.github.com/users/nyh/received_events","type":"User","site_admin":false},"created_at":"2018-07-03T19:52:24Z","updated_at":"2018-07-03T19:52:24Z","author_association":"CONTRIBUTOR","body":"@tgrabiec I dont know why, but I understood previously that repair was run regularly (just with \"-pr\"), more frequently than GC grace period. If that is not the case then this is the quintessential reason for resurrection of deleted data, and I don't think there's any question why data was resurrected :-( Resurrection will happen not just with \"-pr\" repair - it can even happen without a repair at all: It's enough that you have two replicas, do an insert and delete, but one of the replicas missed the delete. Some time later, compaction in the first replica leaves only the delete, with only an insert in the second replica. 10 days (GC period) after the deletion, the deletion marker (\"tombstone\") is removed, and we are left with only an insertion in one of the replicas. At this point, the deleted data was already resurrected (a read which happens to consult this replica will return it, and cause an automatic read-repair of this entry). A full repair (with or without -pr) will just make this more obvious.\r\n\r\nI think your example above is still very interesting, if you are curious how can it be that a repair without \"-pr\" didn't resurrect deleted data, but with \"-pr\" it did. I think it is ok in this case that \"-pr\" repair didn't solve the problem (which was inevitable) but the fact the non-\"-pr\" repair did solve it indicates maybe we could have done the \"-pr\" repair differently and by chance, also solve this case. What we could, and probably should (what do you think?), have done differently is to NOT do tombstone gc at all in the repair code's compaction. In your example, node C should not have compacted A and B's data to \"nothing\" but rather should have left the tombstone (which still exists - by chance - in node B).\r\n\r\nNote that all of the above relies on chance. As soon as node B also compacts its sstables its older-than-GC tombstone will also be removed, and the data on node A will be resurrected, no matter what kind of repair you run or don't run afterwards."}}
{"actor":{"display_login":"nyh"}}