{"action":"closed","issue":{"url":"https://api.github.com/repos/coleifer/huey/issues/24","labels_url":"https://api.github.com/repos/coleifer/huey/issues/24/labels{/name}","comments_url":"https://api.github.com/repos/coleifer/huey/issues/24/comments","events_url":"https://api.github.com/repos/coleifer/huey/issues/24/events","html_url":"https://github.com/coleifer/huey/issues/24","id":14319574,"number":24,"title":"Add multiprocessing support (redux)","user":{"login":"apendleton","id":78930,"avatar_url":"https://avatars.githubusercontent.com/u/78930?v=3","gravatar_id":"","url":"https://api.github.com/users/apendleton","html_url":"https://github.com/apendleton","followers_url":"https://api.github.com/users/apendleton/followers","following_url":"https://api.github.com/users/apendleton/following{/other_user}","gists_url":"https://api.github.com/users/apendleton/gists{/gist_id}","starred_url":"https://api.github.com/users/apendleton/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apendleton/subscriptions","organizations_url":"https://api.github.com/users/apendleton/orgs","repos_url":"https://api.github.com/users/apendleton/repos","events_url":"https://api.github.com/users/apendleton/events{/privacy}","received_events_url":"https://api.github.com/users/apendleton/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":7,"created_at":"2013-05-14T16:49:56Z","updated_at":"2016-01-02T13:35:44Z","closed_at":"2016-01-02T13:35:44Z","body":"Per your comment, here's an issue to discuss adding multiprocessing support.\r\n\r\nThe main limitations that became obvious when trying to do this last time were around the fact that pretty much all inter-worker communication using the Python multiprocessing stack takes place via Pickle (e.g., when you push something onto a shared queue, it gets pickled, and reconstructed on the way out of the queue when a child process retrieves it).  This had/has a couple of concrete ramifications:\r\n* Everything that gets shared has to be picklable.\r\n* Care has to be taken to keep object graphs from getting out of control, since you can end up inadvertently pickling a *lot* of stuff if your objects have attributes that have attributes that have attributes, ad infinitum, that all have to be pickled -- and the first point makes it worse, since all of that has to be picklable.  I was able to work around some of these issues by hiding some stuff from pickle and papering over it via property decorators, but it ended up feeling pretty hacky.  Particular example: one of the objects that needed to be shared in the implementation held a reference to the command queue, which in turn, when using the redis backend, held a connection to the redis connection, which wasn't picklable.\r\n* You can't rely on the master and the workers operating on the same copy of a data structure, since it gets copied during pickling/unpickling.  As an example, this caused weirdness within the consumer code with tracking retries of commands, because retries were stored as a property on the command object, so child workers' decrements of this value weren't seen by the parent process."}}