{"action":"created","issue":{"url":"https://api.github.com/repos/arachnys/cabot/issues/334","repository_url":"https://api.github.com/repos/arachnys/cabot","labels_url":"https://api.github.com/repos/arachnys/cabot/issues/334/labels{/name}","comments_url":"https://api.github.com/repos/arachnys/cabot/issues/334/comments","events_url":"https://api.github.com/repos/arachnys/cabot/issues/334/events","html_url":"https://github.com/arachnys/cabot/issues/334","id":146563895,"number":334,"title":"possible memory leak with celery processes","user":{"login":"marvinthepa","id":51218,"avatar_url":"https://avatars.githubusercontent.com/u/51218?v=3","gravatar_id":"","url":"https://api.github.com/users/marvinthepa","html_url":"https://github.com/marvinthepa","followers_url":"https://api.github.com/users/marvinthepa/followers","following_url":"https://api.github.com/users/marvinthepa/following{/other_user}","gists_url":"https://api.github.com/users/marvinthepa/gists{/gist_id}","starred_url":"https://api.github.com/users/marvinthepa/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/marvinthepa/subscriptions","organizations_url":"https://api.github.com/users/marvinthepa/orgs","repos_url":"https://api.github.com/users/marvinthepa/repos","events_url":"https://api.github.com/users/marvinthepa/events{/privacy}","received_events_url":"https://api.github.com/users/marvinthepa/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2016-04-07T09:42:56Z","updated_at":"2016-04-07T09:49:57Z","closed_at":null,"body":"Hey, we are currently running 176bac890 using our private fork of https://github.com/shoonoise/cabot-docker and we have an issue where cabot eats more and more memory after running for some time. The culprit seem to be the celery processes:\r\n\r\n```\r\n  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND\r\n22260 root      20   0 1431068 797888   3924 S   0.0  9.8  27:13.21 celery\r\n22256 root      20   0  630148 455200   4624 S   0.0  5.6   4:09.47 celery\r\n22245 root      20   0  632920 434316   4616 S   0.0  5.3   4:12.00 celery\r\n22254 root      20   0  634468 432172   4616 S   0.0  5.3   4:11.59 celery\r\n22246 root      20   0  629652 430004   4616 S   0.0  5.3   4:09.68 celery\r\n22249 root      20   0  630468 429840   4616 S   0.0  5.3   4:09.26 celery\r\n22255 root      20   0  628276 429236   4616 S   0.0  5.2   4:07.82 celery\r\n22251 root      20   0  628248 428748   4616 S   0.0  5.2   4:08.29 celery\r\n22259 root      20   0  630904 428668   4616 S   0.0  5.2   4:10.04 celery\r\n22247 root      20   0  630140 428564   4616 S   0.0  5.2   4:09.94 celery\r\n22252 root      20   0  630516 428000   4616 S   0.0  5.2   4:09.41 celery\r\n22248 root      20   0  629856 427748   4616 S   0.0  5.2   4:08.66 celery\r\n22258 root      20   0  629404 427008   4616 S   0.0  5.2   4:07.55 celery\r\n22253 root      20   0  628296 426928   4616 S   0.0  5.2   4:07.59 celery\r\n22250 root      20   0  625588 425296   4616 S   0.0  5.2   4:06.67 celery\r\n22257 root      20   0  624612 423320   4616 S   0.0  5.2   4:05.78 celery\r\n22244 root      20   0  626440 420056   4616 S   0.0  5.1   4:08.28 celery\r\n```\r\n\r\nAfter restarting the cabotapp container, it looks very different:\r\n```\r\n  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND\r\n 9646 999       20   0  227244 121036 120532 S   0.0  1.5   1:27.74 postgres\r\n26921 root      20   0  182180  57244   8000 S   0.0  0.7   0:02.85 gunicorn\r\n26927 root      20   0  180272  55260   8064 S   0.0  0.7   0:03.13 gunicorn\r\n26924 root      20   0  182072  55156   8092 S   0.0  0.7   0:03.05 gunicorn\r\n26937 root      20   0  176112  48244   8004 S   0.0  0.6   0:00.24 celery\r\n26912 root      20   0  173236  48104   8776 S   0.3  0.6   0:02.84 celery\r\n26945 root      20   0  176128  47348   7088 S   0.0  0.6   0:00.22 celery\r\n26941 root      20   0  176048  47308   7088 S   0.7  0.6   0:00.26 celery\r\n26942 root      20   0  176092  47252   7088 S   0.0  0.6   0:00.22 celery\r\n26935 root      20   0  176072  47236   7088 S   0.3  0.6   0:00.23 celery\r\n26943 root      20   0  176104  47232   7088 S   0.0  0.6   0:00.14 celery\r\n26936 root      20   0  175928  47112   7088 S   0.0  0.6   0:00.24 celery\r\n26934 root      20   0  175900  47096   7088 S   0.3  0.6   0:00.19 celery\r\n26938 root      20   0  175880  47072   7088 S   0.0  0.6   0:00.24 celery\r\n26932 root      20   0  175836  47068   7088 S   0.0  0.6   0:00.30 celery\r\n26944 root      20   0  175800  47032   7088 S   0.0  0.6   0:00.15 celery\r\n26939 root      20   0  175852  46988   7088 S   0.3  0.6   0:00.16 celery\r\n26931 root      20   0  175732  46984   7088 S   0.0  0.6   0:00.23 celery\r\n26930 root      20   0  175716  46956   7088 S   0.0  0.6   0:00.28 celery\r\n26933 root      20   0  175700  46852   7088 S   0.3  0.6   0:00.23 celery\r\n26940 root      20   0  173444  46644   7012 S   0.0  0.6   0:00.17 celery\r\n26946 root      20   0  172852  45832   6732 S   0.0  0.6   0:00.62 celery\r\n```\r\n\r\nI.e. 0.6% memory vs. 5% memory (the latter even lead to the system swapping).\r\n\r\nI found a few issues for \"celery memory leak\", but all the issues are very old and did not contain enough information for me to take any action.\r\n\r\nI don't want to set up a cron job to restart cabot regularly.\r\n\r\nPlease let me know if you have any suggestions or if I can provide any useful information."},"comment":{"url":"https://api.github.com/repos/arachnys/cabot/issues/comments/206790299","html_url":"https://github.com/arachnys/cabot/issues/334#issuecomment-206790299","issue_url":"https://api.github.com/repos/arachnys/cabot/issues/334","id":206790299,"user":{"login":"marvinthepa","id":51218,"avatar_url":"https://avatars.githubusercontent.com/u/51218?v=3","gravatar_id":"","url":"https://api.github.com/users/marvinthepa","html_url":"https://github.com/marvinthepa","followers_url":"https://api.github.com/users/marvinthepa/followers","following_url":"https://api.github.com/users/marvinthepa/following{/other_user}","gists_url":"https://api.github.com/users/marvinthepa/gists{/gist_id}","starred_url":"https://api.github.com/users/marvinthepa/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/marvinthepa/subscriptions","organizations_url":"https://api.github.com/users/marvinthepa/orgs","repos_url":"https://api.github.com/users/marvinthepa/repos","events_url":"https://api.github.com/users/marvinthepa/events{/privacy}","received_events_url":"https://api.github.com/users/marvinthepa/received_events","type":"User","site_admin":false},"created_at":"2016-04-07T09:49:57Z","updated_at":"2016-04-07T09:49:57Z","body":"After reading again, this might be related: https://github.com/celery/celery/issues/2927"}}