{"action":"created","issue":{"url":"https://api.github.com/repos/celery/celery/issues/3339","repository_url":"https://api.github.com/repos/celery/celery","labels_url":"https://api.github.com/repos/celery/celery/issues/3339/labels{/name}","comments_url":"https://api.github.com/repos/celery/celery/issues/3339/comments","events_url":"https://api.github.com/repos/celery/celery/issues/3339/events","html_url":"https://github.com/celery/celery/issues/3339","id":167437718,"number":3339,"title":"Memory leak on celery over Heroku (actually I don't think it's a celery issue), just can't figure out what's happening","user":{"login":"filwaitman","id":2631143,"avatar_url":"https://avatars.githubusercontent.com/u/2631143?v=3","gravatar_id":"","url":"https://api.github.com/users/filwaitman","html_url":"https://github.com/filwaitman","followers_url":"https://api.github.com/users/filwaitman/followers","following_url":"https://api.github.com/users/filwaitman/following{/other_user}","gists_url":"https://api.github.com/users/filwaitman/gists{/gist_id}","starred_url":"https://api.github.com/users/filwaitman/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/filwaitman/subscriptions","organizations_url":"https://api.github.com/users/filwaitman/orgs","repos_url":"https://api.github.com/users/filwaitman/repos","events_url":"https://api.github.com/users/filwaitman/events{/privacy}","received_events_url":"https://api.github.com/users/filwaitman/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2016-07-25T18:48:22Z","updated_at":"2016-08-04T00:39:54Z","closed_at":null,"body":"Hey @ask! How are you doing?\r\n\r\nI'm getting a weird behavior on celery working over Heroku.\r\n\r\nSounds like I'm having a memory leak. Actually it behaves like a memory leak, but I don't think this is the case - just don't know what's this about, though.\r\n\r\nFor some reason seems like my scheduled tasks are not releasing memory after they're finished.\r\nIt doesn't seem to be related to scheduled tasks code, since I removed all code running on it and this \"leak\" is still present (see details below).\r\n\r\nI know this \"memory releasing\" part is Python's responsibility, and I don't expect the memory to be released right after task is executed. But I'm getting my celery machines on a memory usage rate of 170% (by using swap and getting a bunch of [R14](https://devcenter.heroku.com/articles/error-codes#r14-memory-quota-exceeded) errors). Check it out:\r\n\r\n![screenshot from 2016-07-25 15-33-28](https://cloud.githubusercontent.com/assets/2631143/17112845/2ba96030-527e-11e6-8f12-4ae921702f55.png)\r\n\r\n(I have restarted celery at 14pm UTC. That's why memory was released.)\r\n\r\nMy pip requirements:\r\n```\r\ncelery==3.1.20\r\ndjango-celery==3.1.17\r\n```\r\n\r\nHow I'm using celery on this machine (Procfile):\r\n```\r\npython manage.py celery worker -E -B -l INFO  # -E because I'm using celerycam\r\n```\r\n\r\nMy celery configs:\r\n```\r\nBROKER_URL = env('BROKER_URL', default='redis://127.0.0.1:6379')  # Being defined on heroku settings var\r\nCELERY_ACCEPT_CONTENT = ['json']\r\nCELERY_TASK_SERIALIZER = 'json'\r\nCELERY_RESULT_SERIALIZER = 'json'\r\nCELERY_TIMEZONE = TIME_ZONE\r\nCELERY_RESULT_BACKEND = 'djcelery.backends.database:DatabaseBackend'\r\nCELERYBEAT_SCHEDULER = 'djcelery.schedulers.DatabaseScheduler'\r\n\r\nCELERYBEAT_SCHEDULE = {}\r\n```\r\n\r\n(let me know if you need any additional info)\r\n\r\nI used to have a bunch of things on my celery scheduler. For debugging purposes I erased them (there's no scheduled tasks running). Without a single scheduled task running the \"leak\" has vanished.\r\nAfter this I created a (stupid?) scheduled task just for testing purposes:\r\n```\r\n# settings.py\r\nCELERYBEAT_SCHEDULE = {\r\n    'Nothing at all (test for memory leak)': {\r\n        'task': 'main.tasks.nothing_at_all',\r\n        'schedule': timedelta(minutes=10)\r\n    },\r\n}\r\n\r\n# tasks.py\r\n@shared_task\r\ndef nothing_at_all():\r\n    a = range(1000000)\r\n    print a[0]\r\n```\r\n\r\nRandom notes:\r\n* I don't really think it's a celery issue since I used it many times before and it worked flawlessly.\r\n* I can't reproduce it locally. At least I tried and it worked as it should.\r\n* On a desperate measure I tried setting `CELERYD_MAX_TASKS_PER_CHILD` on heroku machines. Not even this stopped the \"leak\".\r\n* I tried disabling the celerycam (just in case). Didn't solve the issue.\r\n* I'm able to flush memory by restarting the celery service (this may be obvious, just wanted to mention that)\r\n\r\nAs I mentioned before I'm pretty convinced this is not an issue on your side. But maybe you know what's happening here. I confess I'm a bit lost now.  :laughing: \r\n\r\nLet me know if you have any clue what's going on.\r\n\r\nThanks!"},"comment":{"url":"https://api.github.com/repos/celery/celery/issues/comments/237417762","html_url":"https://github.com/celery/celery/issues/3339#issuecomment-237417762","issue_url":"https://api.github.com/repos/celery/celery/issues/3339","id":237417762,"user":{"login":"ask","id":1558,"avatar_url":"https://avatars.githubusercontent.com/u/1558?v=3","gravatar_id":"","url":"https://api.github.com/users/ask","html_url":"https://github.com/ask","followers_url":"https://api.github.com/users/ask/followers","following_url":"https://api.github.com/users/ask/following{/other_user}","gists_url":"https://api.github.com/users/ask/gists{/gist_id}","starred_url":"https://api.github.com/users/ask/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ask/subscriptions","organizations_url":"https://api.github.com/users/ask/orgs","repos_url":"https://api.github.com/users/ask/repos","events_url":"https://api.github.com/users/ask/events{/privacy}","received_events_url":"https://api.github.com/users/ask/received_events","type":"User","site_admin":false},"created_at":"2016-08-04T00:39:54Z","updated_at":"2016-08-04T00:39:54Z","body":"Btw, maxtasksperchild should be releasing the memory since that will kill the child process.  Are you sure it's the child process here that is consuming the memory?"}}
{"actor":{"display_login":"ask"}}