{"action":"created","issue":{"url":"https://api.github.com/repos/mozilla/TUID/issues/60","repository_url":"https://api.github.com/repos/mozilla/TUID","labels_url":"https://api.github.com/repos/mozilla/TUID/issues/60/labels{/name}","comments_url":"https://api.github.com/repos/mozilla/TUID/issues/60/comments","events_url":"https://api.github.com/repos/mozilla/TUID/issues/60/events","html_url":"https://github.com/mozilla/TUID/issues/60","id":340886033,"node_id":"MDU6SXNzdWUzNDA4ODYwMzM=","number":60,"title":"HTTP requests to json-annotate caused high load on hg.mozilla.org","user":{"login":"indygreg","id":342993,"node_id":"MDQ6VXNlcjM0Mjk5Mw==","avatar_url":"https://avatars2.githubusercontent.com/u/342993?v=4","gravatar_id":"","url":"https://api.github.com/users/indygreg","html_url":"https://github.com/indygreg","followers_url":"https://api.github.com/users/indygreg/followers","following_url":"https://api.github.com/users/indygreg/following{/other_user}","gists_url":"https://api.github.com/users/indygreg/gists{/gist_id}","starred_url":"https://api.github.com/users/indygreg/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/indygreg/subscriptions","organizations_url":"https://api.github.com/users/indygreg/orgs","repos_url":"https://api.github.com/users/indygreg/repos","events_url":"https://api.github.com/users/indygreg/events{/privacy}","received_events_url":"https://api.github.com/users/indygreg/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-07-13T05:27:18Z","updated_at":"2018-07-13T16:22:15Z","closed_at":null,"author_association":"MEMBER","body":"We received automated alerts about high load on hg.mozilla.org around 0351 UTC (~80 minutes ago).\r\n\r\nThe onset of the server load correlates to an influx of HTTP requests with the UA ``https://github.com/mozilla/TUID``. Many requests were for URLs under ``/mozilla-central/json-annotate/ccdb64ade35f``. But later it branched out into other changesets.\r\n\r\nThe request volume was high enough to exhaust all server capacity and result in a tree closure.\r\n\r\nThere are instructions to ban the UA / IP if another high volume episode occurs.\r\n\r\nSo, what should be done about things?\r\n\r\nThe highest priority is to dial back the concurrency of HTTP requests made to hg.mozilla.org. My logs show at *least* 9 concurrent HTTP requests hitting the same server. We're probably looking at at least 30-40 concurrent requests across the entire cluster. That's **way too high**. Please limit concurrent HTTP requests to the service to <=10 requests.\r\n\r\nDepending on how many queries need to run, we should also look into offloading this computation so we don't need to burn CPU cycles on hg.mozilla.org. Calculating file annotations is a **very** expensive computation. It can take >10s of CPU per file. Issuing thousands of requests to compute file annotations is a very easy way to exhaust hg.mozilla.org of CPU capacity.\r\n\r\nFile annotations can be performed against a local clone. e.g. by running `hg annotate -T json` to get machine readable output. We could potentially offload bulk querying of file annotations to Taskcluster tasks. Or special infrastructure could be stood up to maintain a persistent clone to facilitate ad-hoc, low-latency querying.\r\n\r\nThere are also other possibilities. e.g. the fastannotate extension (https://bitbucket.org/facebook/hg-experimental/src/6d4d0eb3b82d0043fdf00157ce92e11fbb7b3a7c/hgext3rd/fastannotate/?at=default) stores annotate data in such a way to facilitate faster querying. The data can likely even be exchanged between hosts. We used to have the extension enabled on some repos on hg.mozilla.org. But we disabled it after it broke after a Mercurial upgrade. https://bugzilla.mozilla.org/show_bug.cgi?id=1427019. We may want to get that running again...\r\n\r\nCC @klibby"},"comment":{"url":"https://api.github.com/repos/mozilla/TUID/issues/comments/404882592","html_url":"https://github.com/mozilla/TUID/issues/60#issuecomment-404882592","issue_url":"https://api.github.com/repos/mozilla/TUID/issues/60","id":404882592,"node_id":"MDEyOklzc3VlQ29tbWVudDQwNDg4MjU5Mg==","user":{"login":"gmierz","id":10966989,"node_id":"MDQ6VXNlcjEwOTY2OTg5","avatar_url":"https://avatars3.githubusercontent.com/u/10966989?v=4","gravatar_id":"","url":"https://api.github.com/users/gmierz","html_url":"https://github.com/gmierz","followers_url":"https://api.github.com/users/gmierz/followers","following_url":"https://api.github.com/users/gmierz/following{/other_user}","gists_url":"https://api.github.com/users/gmierz/gists{/gist_id}","starred_url":"https://api.github.com/users/gmierz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gmierz/subscriptions","organizations_url":"https://api.github.com/users/gmierz/orgs","repos_url":"https://api.github.com/users/gmierz/repos","events_url":"https://api.github.com/users/gmierz/events{/privacy}","received_events_url":"https://api.github.com/users/gmierz/received_events","type":"User","site_admin":false},"created_at":"2018-07-13T16:22:15Z","updated_at":"2018-07-13T16:22:15Z","author_association":"COLLABORATOR","body":"Thanks for letting us know @indygreg. The service is shutdown until we resolve this.\r\n\r\nAnd thank you for all the solutions you've outlined, but I don't think we need to add any special tooling for this because we will only be requesting the annotations once for all files in the near future.\r\n\r\nBecause of that, I've started implementing a restriction so that no more than 10 annotations are ever being requested concurrently. Once I've tested it and made sure it does what is expected of it, I'll mention it here and restart it. Here's the start: https://github.com/mozilla/TUID/commit/41fb49e3ef1403d05cdc35e26fef2659d6cc7a3c"}}
{"actor":{"display_login":"gmierz"}}