{"action":"created","issue":{"url":"https://api.github.com/repos/beer-garden/beer-garden/issues/176","repository_url":"https://api.github.com/repos/beer-garden/beer-garden","labels_url":"https://api.github.com/repos/beer-garden/beer-garden/issues/176/labels{/name}","comments_url":"https://api.github.com/repos/beer-garden/beer-garden/issues/176/comments","events_url":"https://api.github.com/repos/beer-garden/beer-garden/issues/176/events","html_url":"https://github.com/beer-garden/beer-garden/issues/176","id":357905547,"node_id":"MDU6SXNzdWUzNTc5MDU1NDc=","number":176,"title":"Slowness on underpowered systems","user":{"login":"hazmat345","id":7344489,"node_id":"MDQ6VXNlcjczNDQ0ODk=","avatar_url":"https://avatars2.githubusercontent.com/u/7344489?v=4","gravatar_id":"","url":"https://api.github.com/users/hazmat345","html_url":"https://github.com/hazmat345","followers_url":"https://api.github.com/users/hazmat345/followers","following_url":"https://api.github.com/users/hazmat345/following{/other_user}","gists_url":"https://api.github.com/users/hazmat345/gists{/gist_id}","starred_url":"https://api.github.com/users/hazmat345/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hazmat345/subscriptions","organizations_url":"https://api.github.com/users/hazmat345/orgs","repos_url":"https://api.github.com/users/hazmat345/repos","events_url":"https://api.github.com/users/hazmat345/events{/privacy}","received_events_url":"https://api.github.com/users/hazmat345/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-09-07T03:23:31Z","updated_at":"2018-09-07T03:36:29Z","closed_at":null,"author_association":"MEMBER","body":"BLUF: I think we should punt on this and advise users to schedule jobs on the order of minutes, not seconds.\r\n\r\nThe current executor is basically a threadpool. That's not super great to run alongside an IO loop, but it's worse when you consider that our tasks are also calling out to the server in the same process. Just to generate a request this needs to happen (take this with a large salt cube. This is basically a guess):\r\n\r\n- Start state. All quiet.\r\n- Scheduler is scheduled (ha) in the IO loop\r\n- There's a task that needs to be run. It's submitted to the executor\r\n- ... wait for context switch ...\r\n- Worker thread is switched in. Request is created.\r\n- ... wait for context switch ...\r\n- IO loop switched in. Request is received, start processing.\r\n- Save request to Mongo (blocks IO loop)\r\n- Finish processing\r\n- ... wait for context switch ...\r\n- Sweet, request is successfully created. Worker is done and can go back in the pool.\r\n\r\nAlso, if the worker thread ever gets switched back in while the IO loop is processing / saving the new request that's essentially wasted slices since the worker is unable to actually do anything. I'm guessing that happens a bit since request creation is taking ~10s of milliseconds.\r\n\r\nSo the amount of time that the server can process anything else (e.g. the static content) is pretty reduced.\r\n\r\napscheduler will run the job in the IO loop instead if the job function is a native coroutine. Since we are running legacy Python that's not an option, so to test / start to fix this I created a new executor that will also work on `tornado.gen.coroutine` decorated functions.\r\n\r\n... which completely locked the whole thing, as expected. Since we aren't using an async HTTP client we deadlock the first time we make_request.\r\n\r\nSo if we could swap in an async version of Requests I think we could dramatically improve our performance here, but that's going to be ... not trivial.\r\n\r\nWe should also run the external stress testing scripts to see if the same amount of slowness occurs with the same amount of requests."},"comment":{"url":"https://api.github.com/repos/beer-garden/beer-garden/issues/comments/419312262","html_url":"https://github.com/beer-garden/beer-garden/issues/176#issuecomment-419312262","issue_url":"https://api.github.com/repos/beer-garden/beer-garden/issues/176","id":419312262,"node_id":"MDEyOklzc3VlQ29tbWVudDQxOTMxMjI2Mg==","user":{"login":"hazmat345","id":7344489,"node_id":"MDQ6VXNlcjczNDQ0ODk=","avatar_url":"https://avatars2.githubusercontent.com/u/7344489?v=4","gravatar_id":"","url":"https://api.github.com/users/hazmat345","html_url":"https://github.com/hazmat345","followers_url":"https://api.github.com/users/hazmat345/followers","following_url":"https://api.github.com/users/hazmat345/following{/other_user}","gists_url":"https://api.github.com/users/hazmat345/gists{/gist_id}","starred_url":"https://api.github.com/users/hazmat345/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hazmat345/subscriptions","organizations_url":"https://api.github.com/users/hazmat345/orgs","repos_url":"https://api.github.com/users/hazmat345/repos","events_url":"https://api.github.com/users/hazmat345/events{/privacy}","received_events_url":"https://api.github.com/users/hazmat345/received_events","type":"User","site_admin":false},"created_at":"2018-09-07T03:36:29Z","updated_at":"2018-09-07T03:36:29Z","author_association":"MEMBER","body":"So I think we should punt until 3.0 when we can create a request from within Beer-garden without needing to go through the REST endpoints. "}}
{"actor":{"display_login":"hazmat345"}}