{"action":"created","issue":{"url":"https://api.github.com/repos/weaveworks/cortex/issues/730","repository_url":"https://api.github.com/repos/weaveworks/cortex","labels_url":"https://api.github.com/repos/weaveworks/cortex/issues/730/labels{/name}","comments_url":"https://api.github.com/repos/weaveworks/cortex/issues/730/comments","events_url":"https://api.github.com/repos/weaveworks/cortex/issues/730/events","html_url":"https://github.com/weaveworks/cortex/issues/730","id":301704041,"number":730,"title":"Stop early-return after majority success in distributor","user":{"login":"bboreham","id":8125524,"avatar_url":"https://avatars2.githubusercontent.com/u/8125524?v=4","gravatar_id":"","url":"https://api.github.com/users/bboreham","html_url":"https://github.com/bboreham","followers_url":"https://api.github.com/users/bboreham/followers","following_url":"https://api.github.com/users/bboreham/following{/other_user}","gists_url":"https://api.github.com/users/bboreham/gists{/gist_id}","starred_url":"https://api.github.com/users/bboreham/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bboreham/subscriptions","organizations_url":"https://api.github.com/users/bboreham/orgs","repos_url":"https://api.github.com/users/bboreham/repos","events_url":"https://api.github.com/users/bboreham/events{/privacy}","received_events_url":"https://api.github.com/users/bboreham/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-03-02T09:30:27Z","updated_at":"2018-03-02T09:40:51Z","closed_at":null,"author_association":"OWNER","body":"Suppose we have a replication factor of 3, then the writing process is:\r\n- Distributor replicates the data three times and fires up three goroutines to deliver the data.\r\n- Once two of the calls have returned from ingesters, distributor returns success.\r\n- Context is cancelled (in distributor, because the call ended; in ruler, to avoid leaking timers)\r\n- Third call may or may not have updated its ingester.\r\n\r\nI already think this causes #670; the idea that each replica may be missing some samples makes me more queasy.\r\n\r\nIn a typical `Push()` call containing 100 samples, the distributor will be writing to all ingesters (in different combinations for each metric), which will make the missing samples issue rare. And until #673 there was no timeout on ruler updates, so that context was never cancelled.\r\n\r\nLet's just stop it - wait for all calls to return, or a timeout."},"comment":{"url":"https://api.github.com/repos/weaveworks/cortex/issues/comments/369872729","html_url":"https://github.com/weaveworks/cortex/issues/730#issuecomment-369872729","issue_url":"https://api.github.com/repos/weaveworks/cortex/issues/730","id":369872729,"user":{"login":"rade","id":109109,"avatar_url":"https://avatars3.githubusercontent.com/u/109109?v=4","gravatar_id":"","url":"https://api.github.com/users/rade","html_url":"https://github.com/rade","followers_url":"https://api.github.com/users/rade/followers","following_url":"https://api.github.com/users/rade/following{/other_user}","gists_url":"https://api.github.com/users/rade/gists{/gist_id}","starred_url":"https://api.github.com/users/rade/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rade/subscriptions","organizations_url":"https://api.github.com/users/rade/orgs","repos_url":"https://api.github.com/users/rade/repos","events_url":"https://api.github.com/users/rade/events{/privacy}","received_events_url":"https://api.github.com/users/rade/received_events","type":"User","site_admin":false},"created_at":"2018-03-02T09:40:51Z","updated_at":"2018-03-02T09:40:51Z","author_association":"OWNER","body":"> Let's just stop it - wait for all calls to return, or a timeout.\r\n\r\nDoesn't that mean one misbehaving ingester - being really slow or never responding - would slow down everything?"}}
{"actor":{"display_login":"rade"}}