{"action":"created","issue":{"url":"https://api.github.com/repos/celery/celery/issues/2218","repository_url":"https://api.github.com/repos/celery/celery","labels_url":"https://api.github.com/repos/celery/celery/issues/2218/labels{/name}","comments_url":"https://api.github.com/repos/celery/celery/issues/2218/comments","events_url":"https://api.github.com/repos/celery/celery/issues/2218/events","html_url":"https://github.com/celery/celery/issues/2218","id":41368136,"number":2218,"title":"ETA Memory Usage is too high leading to out of memory exceptions.","user":{"login":"nicolasbaer","id":2022400,"avatar_url":"https://avatars.githubusercontent.com/u/2022400?v=3","gravatar_id":"","url":"https://api.github.com/users/nicolasbaer","html_url":"https://github.com/nicolasbaer","followers_url":"https://api.github.com/users/nicolasbaer/followers","following_url":"https://api.github.com/users/nicolasbaer/following{/other_user}","gists_url":"https://api.github.com/users/nicolasbaer/gists{/gist_id}","starred_url":"https://api.github.com/users/nicolasbaer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nicolasbaer/subscriptions","organizations_url":"https://api.github.com/users/nicolasbaer/orgs","repos_url":"https://api.github.com/users/nicolasbaer/repos","events_url":"https://api.github.com/users/nicolasbaer/events{/privacy}","received_events_url":"https://api.github.com/users/nicolasbaer/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/celery/celery/labels/Issue%20Type:%20Feature%20Request","name":"Issue Type: Feature Request","color":"0d0000"}],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":6,"created_at":"2014-08-28T08:54:16Z","updated_at":"2016-06-30T00:26:45Z","closed_at":"2016-06-23T23:50:28Z","body":"Is there any way to reduce the memory consumption of ETA tasks? In case a queue holds thousands of ETA tasks, the workers will try to load all tasks into memory possibly leading to OOM situations. I looked into the `prefetch` option, but apparently this is not applicable for ETA tasks."},"comment":{"url":"https://api.github.com/repos/celery/celery/issues/comments/229528070","html_url":"https://github.com/celery/celery/issues/2218#issuecomment-229528070","issue_url":"https://api.github.com/repos/celery/celery/issues/2218","id":229528070,"user":{"login":"bartdag","id":464322,"avatar_url":"https://avatars.githubusercontent.com/u/464322?v=3","gravatar_id":"","url":"https://api.github.com/users/bartdag","html_url":"https://github.com/bartdag","followers_url":"https://api.github.com/users/bartdag/followers","following_url":"https://api.github.com/users/bartdag/following{/other_user}","gists_url":"https://api.github.com/users/bartdag/gists{/gist_id}","starred_url":"https://api.github.com/users/bartdag/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bartdag/subscriptions","organizations_url":"https://api.github.com/users/bartdag/orgs","repos_url":"https://api.github.com/users/bartdag/repos","events_url":"https://api.github.com/users/bartdag/events{/privacy}","received_events_url":"https://api.github.com/users/bartdag/received_events","type":"User","site_admin":false},"created_at":"2016-06-30T00:26:45Z","updated_at":"2016-06-30T00:26:45Z","body":"We are working on a different workaround. Basically we:\r\n\r\n1. Store task information with eta time in a db.\r\n2. Have a celery periodic task that checks every minute if there are tasks due.\r\n3. If such due tasks exist, launch the tasks. The task will use the appropriate routing and will be added to the same queue as if it had been initially launched with a standard eta.\r\n\r\nTasks thus no longer linger in the workers, do not consume memory (only disk/db space), we can schedule tasks a long time in advance, and we can easily manage them (e.g., cancel tasks).\r\n\r\nIt's not a general solution because:\r\n\r\n1. It uses our extension of django db result (https://github.com/resulto-admin/django-celery-fulldbresult) to store all the parameters necessary to launch the task later.\r\n2. We relax the guarantees usually provided by rabbitmq/celery: we can make sure that a task is never launched twice, but it may never launch if a crash happens at a very bad time. It's possible though to identify those tasks that were never processed.\r\n3. Depending on the kind of ETA and task volume, it would be wise to adopt a less naive approach than our single periodic task running every minute.\r\n\r\nWe'll publish this workaround in django-celery-fulldbresult once we are satisfied with the implementation."}}
{"actor":{"display_login":"bartdag"}}