{"action":"opened","issue":{"url":"https://api.github.com/repos/dask/distributed/issues/465","repository_url":"https://api.github.com/repos/dask/distributed","labels_url":"https://api.github.com/repos/dask/distributed/issues/465/labels{/name}","comments_url":"https://api.github.com/repos/dask/distributed/issues/465/comments","events_url":"https://api.github.com/repos/dask/distributed/issues/465/events","html_url":"https://github.com/dask/distributed/issues/465","id":173620182,"number":465,"title":"Error if joblib used within Executor.map (PicklingError)","user":{"login":"stsievert","id":1320475,"avatar_url":"https://avatars.githubusercontent.com/u/1320475?v=3","gravatar_id":"","url":"https://api.github.com/users/stsievert","html_url":"https://github.com/stsievert","followers_url":"https://api.github.com/users/stsievert/followers","following_url":"https://api.github.com/users/stsievert/following{/other_user}","gists_url":"https://api.github.com/users/stsievert/gists{/gist_id}","starred_url":"https://api.github.com/users/stsievert/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/stsievert/subscriptions","organizations_url":"https://api.github.com/users/stsievert/orgs","repos_url":"https://api.github.com/users/stsievert/repos","events_url":"https://api.github.com/users/stsievert/events{/privacy}","received_events_url":"https://api.github.com/users/stsievert/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2016-08-27T23:15:24Z","updated_at":"2016-08-27T23:15:24Z","closed_at":null,"body":"I have a function I would like to run over many different inputs. I'm using `Executor.map` which provides the natural interface for this.\r\n\r\nIn the function I call with many different inputs, I call another function many times. In my case, this serves the purpose of obtaining some average performance. The natural interface to use is a joblib for-loop. See the code below for an example of what I'm describing.\r\n\r\nMy first thought to get around this is to use an example in the docs to have double mapped functions. However, I'd rather not further complicate my function definitions and have to pass `Executor.map` into `repeated_tests` below.\r\n\r\nI have verified that the example in the docs of using distributed.joblib for the joblib interface worked.\r\n\r\n``` python\r\ndef model_performance(x):\r\n    \"\"\" returns 0 or 1 depending if model succeed or failed \"\"\"\r\n    time.sleep(2)\r\n    return np.random.randint(2)\r\n\r\ndef repeated_tests(x, n=10):\r\n    \"\"\" calls model_performance n times to get average performance \"\"\"\r\n    y = Parallel(n_jobs=-1)(delayed(model_performance)(x)\r\n                            for _ in range(n))\r\n    return sum(y) / n\r\n    \r\nwith joblib.parallel_backend('distributed', scheduler_host=host):\r\n    model_params = [1, 2, 3, 4]\r\n    \r\n    # normal\r\n    a = map(repeated_tests, model_params)\r\n    print(list(a))\r\n    \r\n    # dask.distributed\r\n    b = e.map(repeated_tests, model_params)\r\n    b = e.gather(b)\r\n    print(b)\r\n```\r\n\r\nThe error message I get is below, but the full stack trace can be found in [a gist]\r\n\r\n``` python\r\nPicklingError: Can't pickle <function model_performance at 0x7f0fac1c8950>: attribute lookup model_performance on __main__ failed\r\n```\r\n\r\n[a gist]:https://gist.github.com/stsievert/43391861410203ebbec6511ef3d0e4e4"}}
{"actor":{"display_login":"stsievert"}}