{"action":"created","issue":{"url":"https://api.github.com/repos/weaveworks/cortex/issues/607","repository_url":"https://api.github.com/repos/weaveworks/cortex","labels_url":"https://api.github.com/repos/weaveworks/cortex/issues/607/labels{/name}","comments_url":"https://api.github.com/repos/weaveworks/cortex/issues/607/comments","events_url":"https://api.github.com/repos/weaveworks/cortex/issues/607/events","html_url":"https://github.com/weaveworks/cortex/issues/607","id":275714201,"node_id":"MDU6SXNzdWUyNzU3MTQyMDE=","number":607,"title":"Reduce duplication when writing","user":{"login":"bboreham","id":8125524,"node_id":"MDQ6VXNlcjgxMjU1MjQ=","avatar_url":"https://avatars2.githubusercontent.com/u/8125524?v=4","gravatar_id":"","url":"https://api.github.com/users/bboreham","html_url":"https://github.com/bboreham","followers_url":"https://api.github.com/users/bboreham/followers","following_url":"https://api.github.com/users/bboreham/following{/other_user}","gists_url":"https://api.github.com/users/bboreham/gists{/gist_id}","starred_url":"https://api.github.com/users/bboreham/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bboreham/subscriptions","organizations_url":"https://api.github.com/users/bboreham/orgs","repos_url":"https://api.github.com/users/bboreham/repos","events_url":"https://api.github.com/users/bboreham/events{/privacy}","received_events_url":"https://api.github.com/users/bboreham/received_events","type":"User","site_admin":false},"labels":[{"id":619941414,"node_id":"MDU6TGFiZWw2MTk5NDE0MTQ=","url":"https://api.github.com/repos/weaveworks/cortex/labels/querier/optimisation","name":"querier/optimisation","color":"fef2c0","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":8,"created_at":"2017-11-21T13:37:01Z","updated_at":"2018-07-16T10:01:29Z","closed_at":null,"author_association":"MEMBER","body":"We replicate writes to N (typically 3) ingesters to reduce data loss from random failures of machine or process. However this means in stable operation each metric is written to store three times, along with index entries for each label (at least in the DynamoDB back-end).  Say 10 index entries per chunk, for a total of 33 writes.  Nearly one third of these writes are pointless and, worse, they slow down queries because all those entries get re-fetched and finally deduplicated.\r\n\r\nOne alternative idea:\r\nWhen an ingester is about to write a chunk, query the index to see if an overlapping chunk already exists. If so, subtract the overlapping part and merge the remainder back into the chunks being accumulated.  This would replace the 33 writes with 11 writes and 2 reads.\r\n\r\nThis idea doesn't require any particular consistency from the store - if it under-reports we'll just store some pointless chunks as before."},"comment":{"url":"https://api.github.com/repos/weaveworks/cortex/issues/comments/405200302","html_url":"https://github.com/weaveworks/cortex/issues/607#issuecomment-405200302","issue_url":"https://api.github.com/repos/weaveworks/cortex/issues/607","id":405200302,"node_id":"MDEyOklzc3VlQ29tbWVudDQwNTIwMDMwMg==","user":{"login":"tomwilkie","id":444037,"node_id":"MDQ6VXNlcjQ0NDAzNw==","avatar_url":"https://avatars1.githubusercontent.com/u/444037?v=4","gravatar_id":"","url":"https://api.github.com/users/tomwilkie","html_url":"https://github.com/tomwilkie","followers_url":"https://api.github.com/users/tomwilkie/followers","following_url":"https://api.github.com/users/tomwilkie/following{/other_user}","gists_url":"https://api.github.com/users/tomwilkie/gists{/gist_id}","starred_url":"https://api.github.com/users/tomwilkie/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tomwilkie/subscriptions","organizations_url":"https://api.github.com/users/tomwilkie/orgs","repos_url":"https://api.github.com/users/tomwilkie/repos","events_url":"https://api.github.com/users/tomwilkie/events{/privacy}","received_events_url":"https://api.github.com/users/tomwilkie/received_events","type":"User","site_admin":false},"created_at":"2018-07-16T10:01:29Z","updated_at":"2018-07-16T10:01:29Z","author_association":"COLLABORATOR","body":"#875 now writes out `(label:value) -> []seriesID` and `seriesID -> []chunkID` indexes, which reduces the size of the index by 6-12x over the `(label:value) -> []chunkID` indexes of old - 3x due to replication and 4x due to average chunk size of 6hrs vs 24hrs index window.\r\n\r\nWe don't dedupe chunks yet, but this should massively reduce size of index."}}
{"actor":{"display_login":"tomwilkie"}}