{"action":"created","issue":{"url":"https://api.github.com/repos/pydata/pandas/issues/5751","repository_url":"https://api.github.com/repos/pydata/pandas","labels_url":"https://api.github.com/repos/pydata/pandas/issues/5751/labels{/name}","comments_url":"https://api.github.com/repos/pydata/pandas/issues/5751/comments","events_url":"https://api.github.com/repos/pydata/pandas/issues/5751/events","html_url":"https://github.com/pydata/pandas/issues/5751","id":24587639,"number":5751,"title":"Parallelization for embarrassingly parallel tasks","user":{"login":"michaelaye","id":69774,"avatar_url":"https://avatars.githubusercontent.com/u/69774?v=3","gravatar_id":"","url":"https://api.github.com/users/michaelaye","html_url":"https://github.com/michaelaye","followers_url":"https://api.github.com/users/michaelaye/followers","following_url":"https://api.github.com/users/michaelaye/following{/other_user}","gists_url":"https://api.github.com/users/michaelaye/gists{/gist_id}","starred_url":"https://api.github.com/users/michaelaye/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/michaelaye/subscriptions","organizations_url":"https://api.github.com/users/michaelaye/orgs","repos_url":"https://api.github.com/users/michaelaye/repos","events_url":"https://api.github.com/users/michaelaye/events{/privacy}","received_events_url":"https://api.github.com/users/michaelaye/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/pydata/pandas/labels/Difficulty%20Novice","name":"Difficulty Novice","color":"fbca04"},{"url":"https://api.github.com/repos/pydata/pandas/labels/Docs","name":"Docs","color":"3465A4"},{"url":"https://api.github.com/repos/pydata/pandas/labels/Effort%20Low","name":"Effort Low","color":"006b75"},{"url":"https://api.github.com/repos/pydata/pandas/labels/Groupby","name":"Groupby","color":"729FCF"},{"url":"https://api.github.com/repos/pydata/pandas/labels/IO%20HDF5","name":"IO HDF5","color":"5319e7"},{"url":"https://api.github.com/repos/pydata/pandas/labels/Parallel","name":"Parallel","color":"e11d21"},{"url":"https://api.github.com/repos/pydata/pandas/labels/Performance","name":"Performance","color":"a10c02"},{"url":"https://api.github.com/repos/pydata/pandas/labels/Prio-medium","name":"Prio-medium","color":"FF8C00"}],"state":"open","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/pydata/pandas/milestones/38","html_url":"https://github.com/pydata/pandas/milestones/0.18.1","labels_url":"https://api.github.com/repos/pydata/pandas/milestones/38/labels","id":1570594,"number":38,"title":"0.18.1","description":"after 0.18.0 of course!","creator":{"login":"jreback","id":953992,"avatar_url":"https://avatars.githubusercontent.com/u/953992?v=3","gravatar_id":"","url":"https://api.github.com/users/jreback","html_url":"https://github.com/jreback","followers_url":"https://api.github.com/users/jreback/followers","following_url":"https://api.github.com/users/jreback/following{/other_user}","gists_url":"https://api.github.com/users/jreback/gists{/gist_id}","starred_url":"https://api.github.com/users/jreback/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jreback/subscriptions","organizations_url":"https://api.github.com/users/jreback/orgs","repos_url":"https://api.github.com/users/jreback/repos","events_url":"https://api.github.com/users/jreback/events{/privacy}","received_events_url":"https://api.github.com/users/jreback/received_events","type":"User","site_admin":false},"open_issues":57,"closed_issues":2,"state":"open","created_at":"2016-02-08T15:29:59Z","updated_at":"2016-02-23T18:47:40Z","due_on":"2016-03-31T04:00:00Z","closed_at":null},"comments":52,"created_at":"2013-12-19T21:03:19Z","updated_at":"2016-02-24T01:13:31Z","closed_at":null,"body":"I would like to promote the idea of applying multiprocessing.Pool() execution of embarrassingly parallel tasks, e.g. the application of a function to a large number of columns.\r\nObviously, there's some overhead to set it up, so there will be a lower limit of number of columns from which on this only can be faster than the already fast cython approach. I will be adding my performance tests later to this issue.\r\nI was emailing @jreback about this and he added the following remarks/complications:\r\n\r\n* transferring numpy data is not that efficient - but still prob good enough\r\n* you can transfer (pickle) lambda type functions so maybe need to look at using a library called dill ( which solves this problem) - possibly could slightly modify msgpack to do this though (and is already pretty efficient at transferring other types of objects)\r\n* could also investigate joblib - I think statsmodels uses it [ed: That seems to be correct, I read in their group about joblib]\r\n* I would create a new top level dir core/parallel for this type of stuff\r\n* the strategy in this link could be a good way to follow: http://stackoverflow.com/questions/17785275/share-large-read-only-numpy-array-between-multiprocessing-processes \r\n\r\nlinks:\r\nhttp://stackoverflow.com/questions/13065172/multiprocess-python-numpy-code-for-processing-data-faster\r\nhttp://docs.cython.org/src/userguide/parallelism.html\r\nhttp://distarray.readthedocs.org/en/v0.5.0/"},"comment":{"url":"https://api.github.com/repos/pydata/pandas/issues/comments/187997387","html_url":"https://github.com/pydata/pandas/issues/5751#issuecomment-187997387","issue_url":"https://api.github.com/repos/pydata/pandas/issues/5751","id":187997387,"user":{"login":"jreback","id":953992,"avatar_url":"https://avatars.githubusercontent.com/u/953992?v=3","gravatar_id":"","url":"https://api.github.com/users/jreback","html_url":"https://github.com/jreback","followers_url":"https://api.github.com/users/jreback/followers","following_url":"https://api.github.com/users/jreback/following{/other_user}","gists_url":"https://api.github.com/users/jreback/gists{/gist_id}","starred_url":"https://api.github.com/users/jreback/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jreback/subscriptions","organizations_url":"https://api.github.com/users/jreback/orgs","repos_url":"https://api.github.com/users/jreback/repos","events_url":"https://api.github.com/users/jreback/events{/privacy}","received_events_url":"https://api.github.com/users/jreback/received_events","type":"User","site_admin":false},"created_at":"2016-02-24T01:13:31Z","updated_at":"2016-02-24T01:13:31Z","body":"so if you do arithmetic ops inside ``sphericity`` (e.g. ``df + df``) or like. These defer to ``numexpr``, which utilizes multi-cores by default. Can you provide a sketch of this function?\r\n\r\nThese are performance things that pandas does w/o the user being specifically aware."}}