{"action":"created","issue":{"url":"https://api.github.com/repos/alphagov/govuk-puppet/issues/7713","repository_url":"https://api.github.com/repos/alphagov/govuk-puppet","labels_url":"https://api.github.com/repos/alphagov/govuk-puppet/issues/7713/labels{/name}","comments_url":"https://api.github.com/repos/alphagov/govuk-puppet/issues/7713/comments","events_url":"https://api.github.com/repos/alphagov/govuk-puppet/issues/7713/events","html_url":"https://github.com/alphagov/govuk-puppet/pull/7713","id":335814498,"node_id":"MDExOlB1bGxSZXF1ZXN0MTk3NDE4NzY5","number":7713,"title":"Mongodb - Maximum open files","user":{"login":"suthagarht","id":16442954,"node_id":"MDQ6VXNlcjE2NDQyOTU0","avatar_url":"https://avatars3.githubusercontent.com/u/16442954?v=4","gravatar_id":"","url":"https://api.github.com/users/suthagarht","html_url":"https://github.com/suthagarht","followers_url":"https://api.github.com/users/suthagarht/followers","following_url":"https://api.github.com/users/suthagarht/following{/other_user}","gists_url":"https://api.github.com/users/suthagarht/gists{/gist_id}","starred_url":"https://api.github.com/users/suthagarht/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/suthagarht/subscriptions","organizations_url":"https://api.github.com/users/suthagarht/orgs","repos_url":"https://api.github.com/users/suthagarht/repos","events_url":"https://api.github.com/users/suthagarht/events{/privacy}","received_events_url":"https://api.github.com/users/suthagarht/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2018-06-26T13:16:36Z","updated_at":"2018-06-26T13:48:28Z","closed_at":"2018-06-26T13:24:55Z","author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/alphagov/govuk-puppet/pulls/7713","html_url":"https://github.com/alphagov/govuk-puppet/pull/7713","diff_url":"https://github.com/alphagov/govuk-puppet/pull/7713.diff","patch_url":"https://github.com/alphagov/govuk-puppet/pull/7713.patch"},"body":"- We have experienced two recorded occasions where the mongo cluster\r\nfailed.\r\n\r\n- During the investigation , we found-out that the number of openfiles\r\nexceeded 3000. However, the current limit is set at 2000.\r\n\r\n- This change will increase the acceptable number of openfiles to 4096.\r\n\r\n`root@api-mongo-3:~# ps -C mongod -o pid --no-headers\r\n 5297`\r\n\r\n`root@api-mongo-3:~# cat /proc/5297/limits\r\nLimit                     Soft Limit           Hard Limit           Units\r\n......\r\nMax open files            20000                20000                files\r\n......\r\nMax realtime timeout      unlimited            unlimited            us`\r\n\r\nhttps://govuk.zendesk.com/agent/tickets/2782624\r\n\r\n\r\n![screen shot 2018-06-26 at 14 18 15](https://user-images.githubusercontent.com/16442954/41914718-e3ba238a-794b-11e8-9cef-9c573f7950a2.png)\r\n\r\nSolo: @suthagarht"},"comment":{"url":"https://api.github.com/repos/alphagov/govuk-puppet/issues/comments/400315229","html_url":"https://github.com/alphagov/govuk-puppet/pull/7713#issuecomment-400315229","issue_url":"https://api.github.com/repos/alphagov/govuk-puppet/issues/7713","id":400315229,"node_id":"MDEyOklzc3VlQ29tbWVudDQwMDMxNTIyOQ==","user":{"login":"suthagarht","id":16442954,"node_id":"MDQ6VXNlcjE2NDQyOTU0","avatar_url":"https://avatars3.githubusercontent.com/u/16442954?v=4","gravatar_id":"","url":"https://api.github.com/users/suthagarht","html_url":"https://github.com/suthagarht","followers_url":"https://api.github.com/users/suthagarht/followers","following_url":"https://api.github.com/users/suthagarht/following{/other_user}","gists_url":"https://api.github.com/users/suthagarht/gists{/gist_id}","starred_url":"https://api.github.com/users/suthagarht/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/suthagarht/subscriptions","organizations_url":"https://api.github.com/users/suthagarht/orgs","repos_url":"https://api.github.com/users/suthagarht/repos","events_url":"https://api.github.com/users/suthagarht/events{/privacy}","received_events_url":"https://api.github.com/users/suthagarht/received_events","type":"User","site_admin":false},"created_at":"2018-06-26T13:48:28Z","updated_at":"2018-06-26T13:48:28Z","author_association":"CONTRIBUTOR","body":"hi @barrucadu , That is a very good question. This particular incident times relate to the time when we do a mongo backup to AWS S3.\r\n\r\nIn theory, this limit should apply to the designated user. This also means that each process forked by the user will have the limit inherited. \r\n\r\nFor example, if **mongodb** started two processes, each of these processes can open 2000 files. In that occasion, we should see 4000 open files. \r\n\r\nWe can see a high rate of forking around the time in question.\r\n\r\n![screen shot 2018-06-26 at 14 41 55](https://user-images.githubusercontent.com/16442954/41916208-752b049e-794f-11e8-8012-dd5b4bcc816a.png)\r\n\r\nHowever, this is purely based on informed assumptions on the actual reason why the number of open files went up-to 3000 while the limit is 2000. I do understand that, if we had forks, they them selves would have been able to open 2000 files of their own without triggering a problem. \r\n\r\nThis change is tries to test the hypothesis that if we increase the acceptable number then we won't see this issue again. If we do see the issue again and had 3000 open files, then we clearly know that the hypothesis is wrong and will revert the change."}}
{"actor":{"display_login":"suthagarht"}}