{"action":"opened","issue":{"url":"https://api.github.com/repos/Metaswitch/sprout/issues/1399","repository_url":"https://api.github.com/repos/Metaswitch/sprout","labels_url":"https://api.github.com/repos/Metaswitch/sprout/issues/1399/labels{/name}","comments_url":"https://api.github.com/repos/Metaswitch/sprout/issues/1399/comments","events_url":"https://api.github.com/repos/Metaswitch/sprout/issues/1399/events","html_url":"https://github.com/Metaswitch/sprout/issues/1399","id":148203760,"number":1399,"title":"Sprout rejects calls with 503, reports that it is due to overload control in SAS","user":{"login":"johadalin","id":14074163,"avatar_url":"https://avatars.githubusercontent.com/u/14074163?v=3","gravatar_id":"","url":"https://api.github.com/users/johadalin","html_url":"https://github.com/johadalin","followers_url":"https://api.github.com/users/johadalin/followers","following_url":"https://api.github.com/users/johadalin/following{/other_user}","gists_url":"https://api.github.com/users/johadalin/gists{/gist_id}","starred_url":"https://api.github.com/users/johadalin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/johadalin/subscriptions","organizations_url":"https://api.github.com/users/johadalin/orgs","repos_url":"https://api.github.com/users/johadalin/repos","events_url":"https://api.github.com/users/johadalin/events{/privacy}","received_events_url":"https://api.github.com/users/johadalin/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/Metaswitch/sprout/labels/bug","name":"bug","color":"fc2929"},{"url":"https://api.github.com/repos/Metaswitch/sprout/labels/critical","name":"critical","color":"ff0000"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2016-04-13T22:43:47Z","updated_at":"2016-04-13T22:43:47Z","closed_at":null,"body":"<!--\r\nThis page is for reporting issues with Project Clearwater. If you have a question, rather than a bug report, the mailing list at clearwater@lists.projectclearwater.org is a better place for it.\r\n\r\nTo give us the best chance of fixing the problem, we've suggested some information to give - please follow these guidelines if possible.\r\n\r\nDon't forget that you can attach logs and screenshots to Github issues - this may help us debug a problem.\r\n-->\r\n\r\n#### Symptoms\r\n<!-- What exactly went wrong? (If possible, copy and paste the exact output you saw) -->\r\nThe system is CCv9 latest build, on openstack, though this has also been seen on VMware.\r\n\r\nRunning load at ~20 calls per second, occasional calls are rejected with a SIP 503 Service unavailable. The SAS trace for such a call states that this: \"SIP request rejected due to overload control\"\r\n\r\nIn the Sprout logs at this time we see the log text:\r\n`13-04-2016 21:13:43.787 UTC Warning scscfsproutlet.cpp:1769: No charging role in Route header, assume originating\r\n13-04-2016 21:13:43.788 UTC Error dnscachedresolver.cpp:567: Failed to retrieve record for os2.richlab.datcon.co.uk: DNS server returned answer with no data\r\n13-04-2016 21:13:43.788 UTC Error dnscachedresolver.cpp:567: Failed to retrieve record for _sip._udp.os2.richlab.datcon.co.uk: Domain name not found`\r\n\r\nUnfortunately we did not have a pcap running between sprout and the DNS at this point.\r\n\r\n\r\n#### Impact\r\n<!-- What is this preventing you from doing? Does this stop Clearwater working, or stop some calls being processed? -->\r\nThis issue causes the call attempt to fail, and so will initially be marked as critical.\r\nIt is however very intermittent, and the cause is unclear at the moment, so we may eventually wish to downgrade it if we can't reliably reproduce it to gather the correct diagnostics, or if the number of calls dropped is not significant enough.\r\n\r\n#### Diagnostics\r\nAttaching the logs from Sprouts 1 and 2, and putting them with the SAS trace in the diags space. This is from the latest SAS so will need to be viewed on an equivalent version (9.2.10). The Richardson OS2 (openstack 2) deployment has a SAS of this version if needed. \r\nMore diags should be gained when we see this again.\r\n\r\n[sp-1-access_20160413T210000Z.txt](https://github.com/Metaswitch/sprout/files/218090/sp-1-access_20160413T210000Z.txt)\r\n[sp-1-sprout_20160413T210000Z.txt](https://github.com/Metaswitch/sprout/files/218089/sp-1-sprout_20160413T210000Z.txt)\r\n[sp-2-access_20160413T210000Z.txt](https://github.com/Metaswitch/sprout/files/218092/sp-2-access_20160413T210000Z.txt)\r\n[sp-2-sprout_20160413T210000Z.txt](https://github.com/Metaswitch/sprout/files/218091/sp-2-sprout_20160413T210000Z.txt)\r\n\r\n"}}