{"action":"opened","number":307,"pull_request":{"url":"https://api.github.com/repos/uktrade/export-opportunities/pulls/307","id":210696892,"node_id":"MDExOlB1bGxSZXF1ZXN0MjEwNjk2ODky","html_url":"https://github.com/uktrade/export-opportunities/pull/307","diff_url":"https://github.com/uktrade/export-opportunities/pull/307.diff","patch_url":"https://github.com/uktrade/export-opportunities/pull/307.patch","issue_url":"https://api.github.com/repos/uktrade/export-opportunities/issues/307","number":307,"state":"open","locked":false,"title":"(feat) Decrease thread and process limits","user":{"login":"michalc","id":13877,"node_id":"MDQ6VXNlcjEzODc3","avatar_url":"https://avatars1.githubusercontent.com/u/13877?v=4","gravatar_id":"","url":"https://api.github.com/users/michalc","html_url":"https://github.com/michalc","followers_url":"https://api.github.com/users/michalc/followers","following_url":"https://api.github.com/users/michalc/following{/other_user}","gists_url":"https://api.github.com/users/michalc/gists{/gist_id}","starred_url":"https://api.github.com/users/michalc/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/michalc/subscriptions","organizations_url":"https://api.github.com/users/michalc/orgs","repos_url":"https://api.github.com/users/michalc/repos","events_url":"https://api.github.com/users/michalc/events{/privacy}","received_events_url":"https://api.github.com/users/michalc/received_events","type":"User","site_admin":false},"body":"From testing on staging, these (and similar) changes reduced frequency of 502s\r\nand booted workers. It did not completely eliminate them however.\r\n\r\n----\r\n\r\nOn production, there are 900 threads, spread over 4 instances, 8 processes per\r\ninstance, 30 threads per process, 8G memory per instance. If each thread used\r\nas much memory as it could, at most 33mb per thread. Some brief heap analysis\r\nlocally appeared to show that starting a thread takes ~30mb of memory, so this\r\nappears to be too low.\r\n\r\nAlso, there are documented 500 max connections to the database being used, so\r\neven if all of these threads could be used at once, many would fail.\r\n\r\nThere are also (apparently) max of 4 cores on each instance: so in terms of\r\nparallel processing, there can never be more than 4 instructions being executed\r\nin parallel at any given time on any instance.\r\n\r\nSo, to determine better values for threads and process, say we examine two\r\n“extreme” applications...\r\n\r\nCPU bound application, where the GIL effect would be strong\r\n- Number of process should be ~the number of CPUs, and 1 thread per process.\r\n  Other options needless use more memory, increase context switching, or\r\n  blocking on the GIL.\r\n\r\nIO bound application, where the GIL effect would be negligable\r\n- Should have 1 process, higher number of threads, up to whatever limit IO can\r\n  be done in parallel without slowdown. Other options (i.e. high number of\r\n  processes, fewer threads), would needlessly use more memory (each process has\r\n  its own copy of all the Ruby code etc..).\r\n\r\nConsidering the above, even an application of a mixture of IO and CPU, if there\r\nmaximum of 4 cores available, there isn't any case 8 processes would be better\r\nthan 4 processes.\r\n\r\nIf there are 4 processes per instance (down from the 8 now), then we must\r\ndecide how many threads per process (noting there are 30 now). To never exhaust\r\ndatabase connections, which is documented to be 500, if we have between 1 and 8\r\napplication instances, and in the “worst” case of 8 instances\r\n\r\n- DB connections per instance:  500/8 = 62.5 per instance\r\n- 4 processes per instances gives 62.5 / 4 = 15.625 threads per process\r\n\r\nMax concurrent requests that could be served by the cluster, even if it was\r\ndropped to a single instance, would still be 15 * 4 = 60. At 4 instances (which\r\nis how many there are currently), this would be 240. Quickly increasing load\r\nwould still be able to be handled without failure (at least, no greater chance\r\nof failure than now), since they would be queued by Puma.\r\n\r\nHowever, from the available data, but it looks like there is a rough max of 5\r\nconcurrent requests at any given time. Even 1 instance running gives a lot of\r\nroom to handle incoming requests without them queueing up.\r\n\r\nMemory-wise, if each thread used as much as it could, then memory per thread\r\nwould be 8GB/(4*15) = 133mb, which feels much safer than the 33mb now.\r\n\r\n- Recommendation: reduce processes per instance from 8 to 4, threads per\r\n  process from 30 to 15.","created_at":"2018-08-24T10:09:08Z","updated_at":"2018-08-24T10:09:08Z","closed_at":null,"merged_at":null,"merge_commit_sha":null,"assignee":null,"assignees":[],"requested_reviewers":[],"requested_teams":[],"labels":[],"milestone":null,"commits_url":"https://api.github.com/repos/uktrade/export-opportunities/pulls/307/commits","review_comments_url":"https://api.github.com/repos/uktrade/export-opportunities/pulls/307/comments","review_comment_url":"https://api.github.com/repos/uktrade/export-opportunities/pulls/comments{/number}","comments_url":"https://api.github.com/repos/uktrade/export-opportunities/issues/307/comments","statuses_url":"https://api.github.com/repos/uktrade/export-opportunities/statuses/909514a33fbdf4f10922c95b7791b7e996c91c0e","head":{"label":"uktrade:feature/decrease-thread-process-limits","ref":"feature/decrease-thread-process-limits","sha":"909514a33fbdf4f10922c95b7791b7e996c91c0e","user":{"login":"uktrade","id":15107524,"node_id":"MDEyOk9yZ2FuaXphdGlvbjE1MTA3NTI0","avatar_url":"https://avatars2.githubusercontent.com/u/15107524?v=4","gravatar_id":"","url":"https://api.github.com/users/uktrade","html_url":"https://github.com/uktrade","followers_url":"https://api.github.com/users/uktrade/followers","following_url":"https://api.github.com/users/uktrade/following{/other_user}","gists_url":"https://api.github.com/users/uktrade/gists{/gist_id}","starred_url":"https://api.github.com/users/uktrade/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/uktrade/subscriptions","organizations_url":"https://api.github.com/users/uktrade/orgs","repos_url":"https://api.github.com/users/uktrade/repos","events_url":"https://api.github.com/users/uktrade/events{/privacy}","received_events_url":"https://api.github.com/users/uktrade/received_events","type":"Organization","site_admin":false},"repo":{"id":88037370,"node_id":"MDEwOlJlcG9zaXRvcnk4ODAzNzM3MA==","name":"export-opportunities","full_name":"uktrade/export-opportunities","owner":{"login":"uktrade","id":15107524,"node_id":"MDEyOk9yZ2FuaXphdGlvbjE1MTA3NTI0","avatar_url":"https://avatars2.githubusercontent.com/u/15107524?v=4","gravatar_id":"","url":"https://api.github.com/users/uktrade","html_url":"https://github.com/uktrade","followers_url":"https://api.github.com/users/uktrade/followers","following_url":"https://api.github.com/users/uktrade/following{/other_user}","gists_url":"https://api.github.com/users/uktrade/gists{/gist_id}","starred_url":"https://api.github.com/users/uktrade/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/uktrade/subscriptions","organizations_url":"https://api.github.com/users/uktrade/orgs","repos_url":"https://api.github.com/users/uktrade/repos","events_url":"https://api.github.com/users/uktrade/events{/privacy}","received_events_url":"https://api.github.com/users/uktrade/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/uktrade/export-opportunities","description":"UK - Great.gov - Export Opportunities - Find and apply for overseas opportunities from businesses  looking for products or services like yours.","fork":false,"url":"https://api.github.com/repos/uktrade/export-opportunities","forks_url":"https://api.github.com/repos/uktrade/export-opportunities/forks","keys_url":"https://api.github.com/repos/uktrade/export-opportunities/keys{/key_id}","collaborators_url":"https://api.github.com/repos/uktrade/export-opportunities/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/uktrade/export-opportunities/teams","hooks_url":"https://api.github.com/repos/uktrade/export-opportunities/hooks","issue_events_url":"https://api.github.com/repos/uktrade/export-opportunities/issues/events{/number}","events_url":"https://api.github.com/repos/uktrade/export-opportunities/events","assignees_url":"https://api.github.com/repos/uktrade/export-opportunities/assignees{/user}","branches_url":"https://api.github.com/repos/uktrade/export-opportunities/branches{/branch}","tags_url":"https://api.github.com/repos/uktrade/export-opportunities/tags","blobs_url":"https://api.github.com/repos/uktrade/export-opportunities/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/uktrade/export-opportunities/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/uktrade/export-opportunities/git/refs{/sha}","trees_url":"https://api.github.com/repos/uktrade/export-opportunities/git/trees{/sha}","statuses_url":"https://api.github.com/repos/uktrade/export-opportunities/statuses/{sha}","languages_url":"https://api.github.com/repos/uktrade/export-opportunities/languages","stargazers_url":"https://api.github.com/repos/uktrade/export-opportunities/stargazers","contributors_url":"https://api.github.com/repos/uktrade/export-opportunities/contributors","subscribers_url":"https://api.github.com/repos/uktrade/export-opportunities/subscribers","subscription_url":"https://api.github.com/repos/uktrade/export-opportunities/subscription","commits_url":"https://api.github.com/repos/uktrade/export-opportunities/commits{/sha}","git_commits_url":"https://api.github.com/repos/uktrade/export-opportunities/git/commits{/sha}","comments_url":"https://api.github.com/repos/uktrade/export-opportunities/comments{/number}","issue_comment_url":"https://api.github.com/repos/uktrade/export-opportunities/issues/comments{/number}","contents_url":"https://api.github.com/repos/uktrade/export-opportunities/contents/{+path}","compare_url":"https://api.github.com/repos/uktrade/export-opportunities/compare/{base}...{head}","merges_url":"https://api.github.com/repos/uktrade/export-opportunities/merges","archive_url":"https://api.github.com/repos/uktrade/export-opportunities/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/uktrade/export-opportunities/downloads","issues_url":"https://api.github.com/repos/uktrade/export-opportunities/issues{/number}","pulls_url":"https://api.github.com/repos/uktrade/export-opportunities/pulls{/number}","milestones_url":"https://api.github.com/repos/uktrade/export-opportunities/milestones{/number}","notifications_url":"https://api.github.com/repos/uktrade/export-opportunities/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/uktrade/export-opportunities/labels{/name}","releases_url":"https://api.github.com/repos/uktrade/export-opportunities/releases{/id}","deployments_url":"https://api.github.com/repos/uktrade/export-opportunities/deployments","created_at":"2017-04-12T10:00:41Z","updated_at":"2018-08-22T15:06:53Z","pushed_at":"2018-08-24T10:08:44Z","git_url":"git://github.com/uktrade/export-opportunities.git","ssh_url":"git@github.com:uktrade/export-opportunities.git","clone_url":"https://github.com/uktrade/export-opportunities.git","svn_url":"https://github.com/uktrade/export-opportunities","homepage":"https://opportunities.export.great.gov.uk/","size":120297,"stargazers_count":5,"watchers_count":5,"language":"Ruby","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":1,"mirror_url":null,"archived":false,"open_issues_count":1,"license":{"key":"mit","name":"MIT License","spdx_id":"MIT","url":"https://api.github.com/licenses/mit","node_id":"MDc6TGljZW5zZTEz"},"forks":1,"open_issues":1,"watchers":5,"default_branch":"develop"}},"base":{"label":"uktrade:develop","ref":"develop","sha":"856a3c3c291bc7608ccf1c1a97fc021803fceb55","user":{"login":"uktrade","id":15107524,"node_id":"MDEyOk9yZ2FuaXphdGlvbjE1MTA3NTI0","avatar_url":"https://avatars2.githubusercontent.com/u/15107524?v=4","gravatar_id":"","url":"https://api.github.com/users/uktrade","html_url":"https://github.com/uktrade","followers_url":"https://api.github.com/users/uktrade/followers","following_url":"https://api.github.com/users/uktrade/following{/other_user}","gists_url":"https://api.github.com/users/uktrade/gists{/gist_id}","starred_url":"https://api.github.com/users/uktrade/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/uktrade/subscriptions","organizations_url":"https://api.github.com/users/uktrade/orgs","repos_url":"https://api.github.com/users/uktrade/repos","events_url":"https://api.github.com/users/uktrade/events{/privacy}","received_events_url":"https://api.github.com/users/uktrade/received_events","type":"Organization","site_admin":false},"repo":{"id":88037370,"node_id":"MDEwOlJlcG9zaXRvcnk4ODAzNzM3MA==","name":"export-opportunities","full_name":"uktrade/export-opportunities","owner":{"login":"uktrade","id":15107524,"node_id":"MDEyOk9yZ2FuaXphdGlvbjE1MTA3NTI0","avatar_url":"https://avatars2.githubusercontent.com/u/15107524?v=4","gravatar_id":"","url":"https://api.github.com/users/uktrade","html_url":"https://github.com/uktrade","followers_url":"https://api.github.com/users/uktrade/followers","following_url":"https://api.github.com/users/uktrade/following{/other_user}","gists_url":"https://api.github.com/users/uktrade/gists{/gist_id}","starred_url":"https://api.github.com/users/uktrade/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/uktrade/subscriptions","organizations_url":"https://api.github.com/users/uktrade/orgs","repos_url":"https://api.github.com/users/uktrade/repos","events_url":"https://api.github.com/users/uktrade/events{/privacy}","received_events_url":"https://api.github.com/users/uktrade/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/uktrade/export-opportunities","description":"UK - Great.gov - Export Opportunities - Find and apply for overseas opportunities from businesses  looking for products or services like yours.","fork":false,"url":"https://api.github.com/repos/uktrade/export-opportunities","forks_url":"https://api.github.com/repos/uktrade/export-opportunities/forks","keys_url":"https://api.github.com/repos/uktrade/export-opportunities/keys{/key_id}","collaborators_url":"https://api.github.com/repos/uktrade/export-opportunities/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/uktrade/export-opportunities/teams","hooks_url":"https://api.github.com/repos/uktrade/export-opportunities/hooks","issue_events_url":"https://api.github.com/repos/uktrade/export-opportunities/issues/events{/number}","events_url":"https://api.github.com/repos/uktrade/export-opportunities/events","assignees_url":"https://api.github.com/repos/uktrade/export-opportunities/assignees{/user}","branches_url":"https://api.github.com/repos/uktrade/export-opportunities/branches{/branch}","tags_url":"https://api.github.com/repos/uktrade/export-opportunities/tags","blobs_url":"https://api.github.com/repos/uktrade/export-opportunities/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/uktrade/export-opportunities/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/uktrade/export-opportunities/git/refs{/sha}","trees_url":"https://api.github.com/repos/uktrade/export-opportunities/git/trees{/sha}","statuses_url":"https://api.github.com/repos/uktrade/export-opportunities/statuses/{sha}","languages_url":"https://api.github.com/repos/uktrade/export-opportunities/languages","stargazers_url":"https://api.github.com/repos/uktrade/export-opportunities/stargazers","contributors_url":"https://api.github.com/repos/uktrade/export-opportunities/contributors","subscribers_url":"https://api.github.com/repos/uktrade/export-opportunities/subscribers","subscription_url":"https://api.github.com/repos/uktrade/export-opportunities/subscription","commits_url":"https://api.github.com/repos/uktrade/export-opportunities/commits{/sha}","git_commits_url":"https://api.github.com/repos/uktrade/export-opportunities/git/commits{/sha}","comments_url":"https://api.github.com/repos/uktrade/export-opportunities/comments{/number}","issue_comment_url":"https://api.github.com/repos/uktrade/export-opportunities/issues/comments{/number}","contents_url":"https://api.github.com/repos/uktrade/export-opportunities/contents/{+path}","compare_url":"https://api.github.com/repos/uktrade/export-opportunities/compare/{base}...{head}","merges_url":"https://api.github.com/repos/uktrade/export-opportunities/merges","archive_url":"https://api.github.com/repos/uktrade/export-opportunities/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/uktrade/export-opportunities/downloads","issues_url":"https://api.github.com/repos/uktrade/export-opportunities/issues{/number}","pulls_url":"https://api.github.com/repos/uktrade/export-opportunities/pulls{/number}","milestones_url":"https://api.github.com/repos/uktrade/export-opportunities/milestones{/number}","notifications_url":"https://api.github.com/repos/uktrade/export-opportunities/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/uktrade/export-opportunities/labels{/name}","releases_url":"https://api.github.com/repos/uktrade/export-opportunities/releases{/id}","deployments_url":"https://api.github.com/repos/uktrade/export-opportunities/deployments","created_at":"2017-04-12T10:00:41Z","updated_at":"2018-08-22T15:06:53Z","pushed_at":"2018-08-24T10:08:44Z","git_url":"git://github.com/uktrade/export-opportunities.git","ssh_url":"git@github.com:uktrade/export-opportunities.git","clone_url":"https://github.com/uktrade/export-opportunities.git","svn_url":"https://github.com/uktrade/export-opportunities","homepage":"https://opportunities.export.great.gov.uk/","size":120297,"stargazers_count":5,"watchers_count":5,"language":"Ruby","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":1,"mirror_url":null,"archived":false,"open_issues_count":1,"license":{"key":"mit","name":"MIT License","spdx_id":"MIT","url":"https://api.github.com/licenses/mit","node_id":"MDc6TGljZW5zZTEz"},"forks":1,"open_issues":1,"watchers":5,"default_branch":"develop"}},"_links":{"self":{"href":"https://api.github.com/repos/uktrade/export-opportunities/pulls/307"},"html":{"href":"https://github.com/uktrade/export-opportunities/pull/307"},"issue":{"href":"https://api.github.com/repos/uktrade/export-opportunities/issues/307"},"comments":{"href":"https://api.github.com/repos/uktrade/export-opportunities/issues/307/comments"},"review_comments":{"href":"https://api.github.com/repos/uktrade/export-opportunities/pulls/307/comments"},"review_comment":{"href":"https://api.github.com/repos/uktrade/export-opportunities/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/uktrade/export-opportunities/pulls/307/commits"},"statuses":{"href":"https://api.github.com/repos/uktrade/export-opportunities/statuses/909514a33fbdf4f10922c95b7791b7e996c91c0e"}},"author_association":"COLLABORATOR","merged":false,"mergeable":null,"rebaseable":null,"mergeable_state":"unknown","merged_by":null,"comments":0,"review_comments":0,"maintainer_can_modify":false,"commits":1,"additions":2,"deletions":2,"changed_files":1}}
{"actor":{"display_login":"michalc"}}