{"action":"created","issue":{"url":"https://api.github.com/repos/Metaswitch/ellis/issues/175","repository_url":"https://api.github.com/repos/Metaswitch/ellis","labels_url":"https://api.github.com/repos/Metaswitch/ellis/issues/175/labels{/name}","comments_url":"https://api.github.com/repos/Metaswitch/ellis/issues/175/comments","events_url":"https://api.github.com/repos/Metaswitch/ellis/issues/175/events","html_url":"https://github.com/Metaswitch/ellis/issues/175","id":157684296,"number":175,"title":"Ellis doesn't retry on an upstream 5xx error","user":{"login":"rkd-msw","id":4363492,"avatar_url":"https://avatars.githubusercontent.com/u/4363492?v=3","gravatar_id":"","url":"https://api.github.com/users/rkd-msw","html_url":"https://github.com/rkd-msw","followers_url":"https://api.github.com/users/rkd-msw/followers","following_url":"https://api.github.com/users/rkd-msw/following{/other_user}","gists_url":"https://api.github.com/users/rkd-msw/gists{/gist_id}","starred_url":"https://api.github.com/users/rkd-msw/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rkd-msw/subscriptions","organizations_url":"https://api.github.com/users/rkd-msw/orgs","repos_url":"https://api.github.com/users/rkd-msw/repos","events_url":"https://api.github.com/users/rkd-msw/events{/privacy}","received_events_url":"https://api.github.com/users/rkd-msw/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/Metaswitch/ellis/labels/low-priority","name":"low-priority","color":"fbca04"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2016-05-31T14:22:56Z","updated_at":"2016-06-01T09:15:45Z","closed_at":null,"body":"<!--\r\nThis page is for reporting issues with Project Clearwater. If you have a question, rather than a bug report, the mailing list at clearwater@lists.projectclearwater.org is a better place for it.\r\n\r\nTo give us the best chance of fixing the problem, we've suggested some information to give - please follow these guidelines if possible.\r\n\r\nDon't forget that you can attach logs and screenshots to Github issues - this may help us debug a problem.\r\n-->\r\n\r\n#### Symptoms\r\n<!-- What exactly went wrong? (If possible, copy and paste the exact output you saw) -->\r\n\r\nI ran the live tests while clearwater-queue-manager was restarting the deployment's processes. A group of the tests failed with a 502 Bad Gateway error while creating numbers in Ellis.\r\n\r\nOn investigation, it looks like:\r\n\r\n- clearwater-queue-manager restarted homestead-prov\r\n- nginx noticed the failed connection and started returning 502 Bad Gateway\r\n- when Ellis got that response, it failed the request, rather than trying the other (working) Homestead\r\n\r\n#### Impact\r\n<!-- What is this preventing you from doing? Does this stop Clearwater working, or stop some calls being processed? -->\r\n\r\nUsers using Ellis don't get the full advantages of homestead-prov's redundancy.\r\n\r\nYou have to wait until clearwater-queue-manager has finished updating a deployment before using Ellis.\r\n\r\n#### Release and environment \r\n<!-- For example: the \"Doom\" release, whether you're on VMware or OpenStack, etc. -->\r\n\r\nFarfetch'd release (release-99 tag), EC2 dogfood system (with 2 Homesteads).\r\n\r\n#### Steps to reproduce\r\n\r\nUpdate shared config.\r\n\r\nRepeatedly run the live tests."},"comment":{"url":"https://api.github.com/repos/Metaswitch/ellis/issues/comments/222937539","html_url":"https://github.com/Metaswitch/ellis/issues/175#issuecomment-222937539","issue_url":"https://api.github.com/repos/Metaswitch/ellis/issues/175","id":222937539,"user":{"login":"rkd-msw","id":4363492,"avatar_url":"https://avatars.githubusercontent.com/u/4363492?v=3","gravatar_id":"","url":"https://api.github.com/users/rkd-msw","html_url":"https://github.com/rkd-msw","followers_url":"https://api.github.com/users/rkd-msw/followers","following_url":"https://api.github.com/users/rkd-msw/following{/other_user}","gists_url":"https://api.github.com/users/rkd-msw/gists{/gist_id}","starred_url":"https://api.github.com/users/rkd-msw/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rkd-msw/subscriptions","organizations_url":"https://api.github.com/users/rkd-msw/orgs","repos_url":"https://api.github.com/users/rkd-msw/repos","events_url":"https://api.github.com/users/rkd-msw/events{/privacy}","received_events_url":"https://api.github.com/users/rkd-msw/received_events","type":"User","site_admin":false},"created_at":"2016-06-01T09:15:45Z","updated_at":"2016-06-01T09:15:45Z","body":"Actually, we do retry on 503 errors from Homestead (https://github.com/Metaswitch/ellis/blob/dev/src/metaswitch/ellis/remote/homestead.py#L278)."}}
{"actor":{"display_login":"rkd-msw"}}