{"action":"created","issue":{"url":"https://api.github.com/repos/openaustralia/morph/issues/1177","repository_url":"https://api.github.com/repos/openaustralia/morph","labels_url":"https://api.github.com/repos/openaustralia/morph/issues/1177/labels{/name}","comments_url":"https://api.github.com/repos/openaustralia/morph/issues/1177/comments","events_url":"https://api.github.com/repos/openaustralia/morph/issues/1177/events","html_url":"https://github.com/openaustralia/morph/issues/1177","id":300445424,"number":1177,"title":"Passenger queue fills up","user":{"login":"mlandauer","id":2221,"avatar_url":"https://avatars2.githubusercontent.com/u/2221?v=4","gravatar_id":"","url":"https://api.github.com/users/mlandauer","html_url":"https://github.com/mlandauer","followers_url":"https://api.github.com/users/mlandauer/followers","following_url":"https://api.github.com/users/mlandauer/following{/other_user}","gists_url":"https://api.github.com/users/mlandauer/gists{/gist_id}","starred_url":"https://api.github.com/users/mlandauer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mlandauer/subscriptions","organizations_url":"https://api.github.com/users/mlandauer/orgs","repos_url":"https://api.github.com/users/mlandauer/repos","events_url":"https://api.github.com/users/mlandauer/events{/privacy}","received_events_url":"https://api.github.com/users/mlandauer/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"mlandauer","id":2221,"avatar_url":"https://avatars2.githubusercontent.com/u/2221?v=4","gravatar_id":"","url":"https://api.github.com/users/mlandauer","html_url":"https://github.com/mlandauer","followers_url":"https://api.github.com/users/mlandauer/followers","following_url":"https://api.github.com/users/mlandauer/following{/other_user}","gists_url":"https://api.github.com/users/mlandauer/gists{/gist_id}","starred_url":"https://api.github.com/users/mlandauer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mlandauer/subscriptions","organizations_url":"https://api.github.com/users/mlandauer/orgs","repos_url":"https://api.github.com/users/mlandauer/repos","events_url":"https://api.github.com/users/mlandauer/events{/privacy}","received_events_url":"https://api.github.com/users/mlandauer/received_events","type":"User","site_admin":false},"assignees":[{"login":"mlandauer","id":2221,"avatar_url":"https://avatars2.githubusercontent.com/u/2221?v=4","gravatar_id":"","url":"https://api.github.com/users/mlandauer","html_url":"https://github.com/mlandauer","followers_url":"https://api.github.com/users/mlandauer/followers","following_url":"https://api.github.com/users/mlandauer/following{/other_user}","gists_url":"https://api.github.com/users/mlandauer/gists{/gist_id}","starred_url":"https://api.github.com/users/mlandauer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mlandauer/subscriptions","organizations_url":"https://api.github.com/users/mlandauer/orgs","repos_url":"https://api.github.com/users/mlandauer/repos","events_url":"https://api.github.com/users/mlandauer/events{/privacy}","received_events_url":"https://api.github.com/users/mlandauer/received_events","type":"User","site_admin":false}],"milestone":null,"comments":2,"created_at":"2018-02-27T00:12:30Z","updated_at":"2018-03-09T00:52:16Z","closed_at":null,"author_association":"OWNER","body":"We've had this a couple of times in the last week where the websites stops being accessible and it just shows \"This website is under heavy load (queue full)\".\r\n\r\nRunning `passenger-status` confirms this:\r\n```\r\nVersion : 5.1.12\r\nDate    : 2018-02-27 11:08:04 +1100\r\nInstance: 6K1CeA0Z (nginx/1.12.2 Phusion_Passenger/5.1.12)\r\n\r\n----------- General information -----------\r\nMax pool size : 6\r\nApp groups    : 1\r\nProcesses     : 6\r\nRequests in top-level queue : 0\r\n\r\n----------- Application groups -----------\r\n/var/www/current/public:\r\n  App root: /var/www/current\r\n  Requests in queue: 100\r\n  * PID: 16361   Sessions: 1       Processed: 40951   Uptime: 18h 53m 12s\r\n    CPU: 3%      Memory  : 1511M   Last used: 15h 8m\r\n  * PID: 16400   Sessions: 1       Processed: 77536   Uptime: 18h 53m 11s\r\n    CPU: 3%      Memory  : 1537M   Last used: 11h 7m\r\n  * PID: 16441   Sessions: 1       Processed: 27030   Uptime: 18h 53m 11s\r\n    CPU: 1%      Memory  : 1144M   Last used: 11h 5m\r\n  * PID: 2546    Sessions: 1       Processed: 99757   Uptime: 15h 39m 8s\r\n    CPU: 6%      Memory  : 1123M   Last used: 2h 39m 5\r\n  * PID: 29117   Sessions: 1       Processed: 22178   Uptime: 11h 6m 57s\r\n    CPU: 3%      Memory  : 1210M   Last used: 2h 30m 7\r\n  * PID: 16433   Sessions: 1       Processed: 5864    Uptime: 11h 4m 57s\r\n    CPU: 1%      Memory  : 1073M   Last used: 2h 38m 5\r\n```\r\n\r\nAfter restarting nginx:\r\n```\r\n*** Cleaning stale instance directory /tmp/passenger.4QJRk5y\r\nVersion : 5.1.12\r\nDate    : 2018-02-27 11:11:04 +1100\r\nInstance: kIhYiAbS (nginx/1.12.2 Phusion_Passenger/5.1.12)\r\n\r\n----------- General information -----------\r\nMax pool size : 6\r\nApp groups    : 1\r\nProcesses     : 5\r\nRequests in top-level queue : 0\r\n\r\n----------- Application groups -----------\r\n/var/www/current/public:\r\n  App root: /var/www/current\r\n  Requests in queue: 0\r\n  * PID: 25859   Sessions: 0       Processed: 146     Uptime: 2m 8s\r\n    CPU: 13%     Memory  : 152M    Last used: 0s ago\r\n  * PID: 25869   Sessions: 0       Processed: 40      Uptime: 2m 8s\r\n    CPU: 4%      Memory  : 138M    Last used: 3s ago\r\n  * PID: 25882   Sessions: 0       Processed: 12      Uptime: 2m 7s\r\n    CPU: 1%      Memory  : 125M    Last used: 5s ago\r\n  * PID: 25894   Sessions: 0       Processed: 2       Uptime: 2m 7s\r\n    CPU: 0%      Memory  : 67M     Last used: 7s ago\r\n  * PID: 25905   Sessions: 0       Processed: 0       Uptime: 2m 7s\r\n    CPU: 0%      Memory  : 26M     Last used: 2m 7s ago\r\n```\r\n\r\nIt would be a good if we could monitor the 'requests in queue' to see what event is causing this to happen. Is it just really really high load, or are there some slow processes which cause everything to get backed up?"},"comment":{"url":"https://api.github.com/repos/openaustralia/morph/issues/comments/371676484","html_url":"https://github.com/openaustralia/morph/issues/1177#issuecomment-371676484","issue_url":"https://api.github.com/repos/openaustralia/morph/issues/1177","id":371676484,"user":{"login":"mlandauer","id":2221,"avatar_url":"https://avatars2.githubusercontent.com/u/2221?v=4","gravatar_id":"","url":"https://api.github.com/users/mlandauer","html_url":"https://github.com/mlandauer","followers_url":"https://api.github.com/users/mlandauer/followers","following_url":"https://api.github.com/users/mlandauer/following{/other_user}","gists_url":"https://api.github.com/users/mlandauer/gists{/gist_id}","starred_url":"https://api.github.com/users/mlandauer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mlandauer/subscriptions","organizations_url":"https://api.github.com/users/mlandauer/orgs","repos_url":"https://api.github.com/users/mlandauer/repos","events_url":"https://api.github.com/users/mlandauer/events{/privacy}","received_events_url":"https://api.github.com/users/mlandauer/received_events","type":"User","site_admin":false},"created_at":"2018-03-09T00:52:16Z","updated_at":"2018-03-09T00:52:16Z","author_association":"OWNER","body":"When you try to do this on local development version (where the server is running from `bundle exec foreman start`) everything works fine.\r\n\r\nSo I had to go through the effort of building a local development server using vagrant to test out the full setup as used in production.\r\n\r\nWhen I did that I could reproduce the api query hang on an error. I just did a query using invalid sql."}}
{"actor":{"display_login":"mlandauer"}}