{"action":"created","issue":{"url":"https://api.github.com/repos/cockroachdb/cockroach/issues/5518","repository_url":"https://api.github.com/repos/cockroachdb/cockroach","labels_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/5518/labels{/name}","comments_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/5518/comments","events_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/5518/events","html_url":"https://github.com/cockroachdb/cockroach/issues/5518","id":143075260,"number":5518,"title":"storage: Separate out consistency checks over floating point data","user":{"login":"vivekmenezes","id":11774212,"avatar_url":"https://avatars.githubusercontent.com/u/11774212?v=3","gravatar_id":"","url":"https://api.github.com/users/vivekmenezes","html_url":"https://github.com/vivekmenezes","followers_url":"https://api.github.com/users/vivekmenezes/followers","following_url":"https://api.github.com/users/vivekmenezes/following{/other_user}","gists_url":"https://api.github.com/users/vivekmenezes/gists{/gist_id}","starred_url":"https://api.github.com/users/vivekmenezes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vivekmenezes/subscriptions","organizations_url":"https://api.github.com/users/vivekmenezes/orgs","repos_url":"https://api.github.com/users/vivekmenezes/repos","events_url":"https://api.github.com/users/vivekmenezes/events{/privacy}","received_events_url":"https://api.github.com/users/vivekmenezes/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"vivekmenezes","id":11774212,"avatar_url":"https://avatars.githubusercontent.com/u/11774212?v=3","gravatar_id":"","url":"https://api.github.com/users/vivekmenezes","html_url":"https://github.com/vivekmenezes","followers_url":"https://api.github.com/users/vivekmenezes/followers","following_url":"https://api.github.com/users/vivekmenezes/following{/other_user}","gists_url":"https://api.github.com/users/vivekmenezes/gists{/gist_id}","starred_url":"https://api.github.com/users/vivekmenezes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vivekmenezes/subscriptions","organizations_url":"https://api.github.com/users/vivekmenezes/orgs","repos_url":"https://api.github.com/users/vivekmenezes/repos","events_url":"https://api.github.com/users/vivekmenezes/events{/privacy}","received_events_url":"https://api.github.com/users/vivekmenezes/received_events","type":"User","site_admin":false},"milestone":null,"comments":26,"created_at":"2016-03-23T20:44:13Z","updated_at":"2016-03-26T01:32:51Z","closed_at":null,"body":"We run periodic consistency checks over all the replicated data. Timeseries data which is cockroach internal, is replicated consistently, but merged independently often outside of raft on each replica. While the consistency checker forces a merge on each replica at a snapshot, thereby making it consistent before computing a checksum on the data, the data itself contains floating points, with the merge operator running floating point math like adding up all the timeseries values. We'd like to continue running consistency checks on this data so we can catch consistency problems on homogenous clusters, while accepting consistency differences on heterogenous one.\r\n\r\nOptions:\r\n1. Separate out the checksum over timeseries data so that it doesn't interfere with the range checksum (preferred).\r\n2. Convert the floating point values into integers and run integer math on them. Not recommended because we also really don't care about this data being that accurate."},"comment":{"url":"https://api.github.com/repos/cockroachdb/cockroach/issues/comments/201650888","html_url":"https://github.com/cockroachdb/cockroach/issues/5518#issuecomment-201650888","issue_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/5518","id":201650888,"user":{"login":"vivekmenezes","id":11774212,"avatar_url":"https://avatars.githubusercontent.com/u/11774212?v=3","gravatar_id":"","url":"https://api.github.com/users/vivekmenezes","html_url":"https://github.com/vivekmenezes","followers_url":"https://api.github.com/users/vivekmenezes/followers","following_url":"https://api.github.com/users/vivekmenezes/following{/other_user}","gists_url":"https://api.github.com/users/vivekmenezes/gists{/gist_id}","starred_url":"https://api.github.com/users/vivekmenezes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vivekmenezes/subscriptions","organizations_url":"https://api.github.com/users/vivekmenezes/orgs","repos_url":"https://api.github.com/users/vivekmenezes/repos","events_url":"https://api.github.com/users/vivekmenezes/events{/privacy}","received_events_url":"https://api.github.com/users/vivekmenezes/received_events","type":"User","site_admin":false},"created_at":"2016-03-26T01:32:51Z","updated_at":"2016-03-26T01:32:51Z","body":"You are referring to the approach of not doing the addition at all and just merging the values into single proto per offset. That reduces the risk considerably because I feel a lot more comfortable with the representation being stable. And the order is guaranteed via raft.\r\n\r\nLuckily we can change `optional sum` to `repeated sum ` whenever we want, so there is no hurry to do that as far as the protobuf freeze is concerned. We might as well make this change now. We can also rename sum to value and get rid of count. It might actually be a savings given that we are eliminating count.\r\n\r\nI can do this over the weekend if you like as long as there is consensus that we will merge it."}}