{"action":"created","issue":{"url":"https://api.github.com/repos/Shopify/semian/issues/188","repository_url":"https://api.github.com/repos/Shopify/semian","labels_url":"https://api.github.com/repos/Shopify/semian/issues/188/labels{/name}","comments_url":"https://api.github.com/repos/Shopify/semian/issues/188/comments","events_url":"https://api.github.com/repos/Shopify/semian/issues/188/events","html_url":"https://github.com/Shopify/semian/pull/188","id":345707010,"node_id":"MDExOlB1bGxSZXF1ZXN0MjA0NzY1NDEw","number":188,"title":"circuit: introduce half_open_resource_timeout","user":{"login":"sirupsen","id":97400,"node_id":"MDQ6VXNlcjk3NDAw","avatar_url":"https://avatars3.githubusercontent.com/u/97400?v=4","gravatar_id":"","url":"https://api.github.com/users/sirupsen","html_url":"https://github.com/sirupsen","followers_url":"https://api.github.com/users/sirupsen/followers","following_url":"https://api.github.com/users/sirupsen/following{/other_user}","gists_url":"https://api.github.com/users/sirupsen/gists{/gist_id}","starred_url":"https://api.github.com/users/sirupsen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sirupsen/subscriptions","organizations_url":"https://api.github.com/users/sirupsen/orgs","repos_url":"https://api.github.com/users/sirupsen/repos","events_url":"https://api.github.com/users/sirupsen/events{/privacy}","received_events_url":"https://api.github.com/users/sirupsen/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2018-07-30T10:50:17Z","updated_at":"2018-07-31T09:46:42Z","closed_at":"2018-07-31T08:30:27Z","author_association":"MEMBER","pull_request":{"url":"https://api.github.com/repos/Shopify/semian/pulls/188","html_url":"https://github.com/Shopify/semian/pull/188","diff_url":"https://github.com/Shopify/semian/pull/188.diff","patch_url":"https://github.com/Shopify/semian/pull/188.patch"},"body":"When a circuit is busy (for those at Shopify, see https://github.com/Shopify/shopify/pull/164168) you may reduce a substantial amount of capacity if you have large timeouts (see https://github.com/Shopify/semian/pull/145). One way to circumvent this is to have very large `error_threshold` timeouts, as a multiple of your timeout. E.g. if your resource timeout is 30s, then an error_timeout of 30s will make sure you have 50% of capacity available, and 60s will make sure you have 25% available (assuming that the circuits trigger equally over the timeout period, which is not an unreasonable assumption in production). However, this has obvious drawbacks, making recovering from incidents take longer for the affected resource.\r\n\r\nInstead, we can reduce the resource timeout in the `half_open?` state to achieve something similar. Normally, timeouts are to allow outlier queries to go through—in the face of an incident, we can eat that (we'd want to kill those anyway) and reduce the timeout dramatically while `half_open?` on the circuit.\r\n\r\nThe ugly thing about this, is that it couples the circuit with the resource. However, I think it makes more sense for the circuit to know about the resource, than for the resource to know about the circuit—that's why I chose to implement it this way.\r\n\r\nThis will need to be implemented per adapter, by implementing `with_resource_timeout`.\r\n\r\ncc @Shopify/servcomm "},"comment":{"url":"https://api.github.com/repos/Shopify/semian/issues/comments/409162268","html_url":"https://github.com/Shopify/semian/pull/188#issuecomment-409162268","issue_url":"https://api.github.com/repos/Shopify/semian/issues/188","id":409162268,"node_id":"MDEyOklzc3VlQ29tbWVudDQwOTE2MjI2OA==","user":{"login":"kirs","id":522155,"node_id":"MDQ6VXNlcjUyMjE1NQ==","avatar_url":"https://avatars3.githubusercontent.com/u/522155?v=4","gravatar_id":"","url":"https://api.github.com/users/kirs","html_url":"https://github.com/kirs","followers_url":"https://api.github.com/users/kirs/followers","following_url":"https://api.github.com/users/kirs/following{/other_user}","gists_url":"https://api.github.com/users/kirs/gists{/gist_id}","starred_url":"https://api.github.com/users/kirs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kirs/subscriptions","organizations_url":"https://api.github.com/users/kirs/orgs","repos_url":"https://api.github.com/users/kirs/repos","events_url":"https://api.github.com/users/kirs/events{/privacy}","received_events_url":"https://api.github.com/users/kirs/received_events","type":"User","site_admin":false},"created_at":"2018-07-31T09:46:42Z","updated_at":"2018-07-31T09:46:42Z","author_association":"CONTRIBUTOR","body":"@sirupsen have you looked into how hard it would be to make other adapters (e.g. Redis) to support the same `with_resource_timeout` trick?"}}
{"actor":{"display_login":"kirs"}}