{"action":"created","issue":{"url":"https://api.github.com/repos/improbable-eng/thanos/issues/282","repository_url":"https://api.github.com/repos/improbable-eng/thanos","labels_url":"https://api.github.com/repos/improbable-eng/thanos/issues/282/labels{/name}","comments_url":"https://api.github.com/repos/improbable-eng/thanos/issues/282/comments","events_url":"https://api.github.com/repos/improbable-eng/thanos/issues/282/events","html_url":"https://github.com/improbable-eng/thanos/pull/282","id":313508222,"number":282,"title":"compact: Use sync-delay only for fresh blocks. Refactored halt, retry logic.","user":{"login":"Bplotka","id":6950331,"avatar_url":"https://avatars2.githubusercontent.com/u/6950331?v=4","gravatar_id":"","url":"https://api.github.com/users/Bplotka","html_url":"https://github.com/Bplotka","followers_url":"https://api.github.com/users/Bplotka/followers","following_url":"https://api.github.com/users/Bplotka/following{/other_user}","gists_url":"https://api.github.com/users/Bplotka/gists{/gist_id}","starred_url":"https://api.github.com/users/Bplotka/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Bplotka/subscriptions","organizations_url":"https://api.github.com/users/Bplotka/orgs","repos_url":"https://api.github.com/users/Bplotka/repos","events_url":"https://api.github.com/users/Bplotka/events{/privacy}","received_events_url":"https://api.github.com/users/Bplotka/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2018-04-11T22:20:19Z","updated_at":"2018-04-13T06:00:49Z","closed_at":null,"author_association":"COLLABORATOR","pull_request":{"url":"https://api.github.com/repos/improbable-eng/thanos/pulls/282","html_url":"https://github.com/improbable-eng/thanos/pull/282","diff_url":"https://github.com/improbable-eng/thanos/pull/282.diff","patch_url":"https://github.com/improbable-eng/thanos/pull/282.patch"},"body":"Currently, we have a similar issue as with prom 2.2.0 compaction.\r\n\r\n< Critical fix >\r\n\r\nFor newly uploaded compacted blocks we need to assume strong consistency for object storage.\r\nIf not, we need to address it somehow with file locks or something like that. Basically, compaction is not safe to be planned without whole overview. (except fresh blocks - these can be safely excluded)\r\n\r\nGCS has strong write-read consistency: https://cloud.google.com/storage/docs/consistency\r\nS3 has weird consistency, but seems to be write-read for newly created objects:  https://codeburst.io/quick-explanation-of-the-s3-consistency-model-6c9f325e3f82\r\n\r\nSigned-off-by: Bartek Plotka <bwplotka@gmail.com>"},"comment":{"url":"https://api.github.com/repos/improbable-eng/thanos/issues/comments/381032530","html_url":"https://github.com/improbable-eng/thanos/pull/282#issuecomment-381032530","issue_url":"https://api.github.com/repos/improbable-eng/thanos/issues/282","id":381032530,"user":{"login":"fabxc","id":4948210,"avatar_url":"https://avatars2.githubusercontent.com/u/4948210?v=4","gravatar_id":"","url":"https://api.github.com/users/fabxc","html_url":"https://github.com/fabxc","followers_url":"https://api.github.com/users/fabxc/followers","following_url":"https://api.github.com/users/fabxc/following{/other_user}","gists_url":"https://api.github.com/users/fabxc/gists{/gist_id}","starred_url":"https://api.github.com/users/fabxc/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fabxc/subscriptions","organizations_url":"https://api.github.com/users/fabxc/orgs","repos_url":"https://api.github.com/users/fabxc/repos","events_url":"https://api.github.com/users/fabxc/events{/privacy}","received_events_url":"https://api.github.com/users/fabxc/received_events","type":"User","site_admin":false},"created_at":"2018-04-13T06:00:49Z","updated_at":"2018-04-13T06:00:49Z","author_association":"COLLABORATOR","body":"> I don't know how we missed that, but with sync-delay we introduced exactly same issue. We are ignoring newest blocks., so even these newly compacted for older time ranges. That resulted in compaction wrongly assuming that the block is not there, thus creating one without the recently created and when all of them are synced (after delay) we have overlapping blocks.\r\n\r\nI don't follow. The compactor will only compact a certain range, if there's data at the start and end of it. So until we see the rightmost, fresh block, we would not attempt compacting the larger range it includes.  \r\nThis should not trigger things as you described here. Am I missing something?"}}
{"actor":{"display_login":"fabxc"}}