{"action":"created","issue":{"url":"https://api.github.com/repos/desihub/desitarget/issues/108","repository_url":"https://api.github.com/repos/desihub/desitarget","labels_url":"https://api.github.com/repos/desihub/desitarget/issues/108/labels{/name}","comments_url":"https://api.github.com/repos/desihub/desitarget/issues/108/comments","events_url":"https://api.github.com/repos/desihub/desitarget/issues/108/events","html_url":"https://github.com/desihub/desitarget/issues/108","id":186676849,"number":108,"title":"reading MWS mocks v.0.02. (with multiprocessing) crashes","user":{"login":"forero","id":1848618,"avatar_url":"https://avatars.githubusercontent.com/u/1848618?v=3","gravatar_id":"","url":"https://api.github.com/users/forero","html_url":"https://github.com/forero","followers_url":"https://api.github.com/users/forero/followers","following_url":"https://api.github.com/users/forero/following{/other_user}","gists_url":"https://api.github.com/users/forero/gists{/gist_id}","starred_url":"https://api.github.com/users/forero/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/forero/subscriptions","organizations_url":"https://api.github.com/users/forero/orgs","repos_url":"https://api.github.com/users/forero/repos","events_url":"https://api.github.com/users/forero/events{/privacy}","received_events_url":"https://api.github.com/users/forero/received_events","type":"User","site_admin":false},"labels":[{"id":181685972,"url":"https://api.github.com/repos/desihub/desitarget/labels/bug","name":"bug","color":"fc2929","default":true}],"state":"open","locked":false,"assignee":{"login":"sbailey","id":218471,"avatar_url":"https://avatars.githubusercontent.com/u/218471?v=3","gravatar_id":"","url":"https://api.github.com/users/sbailey","html_url":"https://github.com/sbailey","followers_url":"https://api.github.com/users/sbailey/followers","following_url":"https://api.github.com/users/sbailey/following{/other_user}","gists_url":"https://api.github.com/users/sbailey/gists{/gist_id}","starred_url":"https://api.github.com/users/sbailey/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sbailey/subscriptions","organizations_url":"https://api.github.com/users/sbailey/orgs","repos_url":"https://api.github.com/users/sbailey/repos","events_url":"https://api.github.com/users/sbailey/events{/privacy}","received_events_url":"https://api.github.com/users/sbailey/received_events","type":"User","site_admin":false},"assignees":[{"login":"sbailey","id":218471,"avatar_url":"https://avatars.githubusercontent.com/u/218471?v=3","gravatar_id":"","url":"https://api.github.com/users/sbailey","html_url":"https://github.com/sbailey","followers_url":"https://api.github.com/users/sbailey/followers","following_url":"https://api.github.com/users/sbailey/following{/other_user}","gists_url":"https://api.github.com/users/sbailey/gists{/gist_id}","starred_url":"https://api.github.com/users/sbailey/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sbailey/subscriptions","organizations_url":"https://api.github.com/users/sbailey/orgs","repos_url":"https://api.github.com/users/sbailey/repos","events_url":"https://api.github.com/users/sbailey/events{/privacy}","received_events_url":"https://api.github.com/users/sbailey/received_events","type":"User","site_admin":false}],"milestone":null,"comments":1,"created_at":"2016-11-01T23:36:43Z","updated_at":"2016-11-02T00:23:53Z","closed_at":null,"body":"The following code\r\n```python\r\nimport desitarget.io\r\nimport desitarget.mock.io\r\nmock_dir='/project/projectdirs/desi/mocks/mws/galaxia/alpha/0.0.2/bricks'\r\niter_mock_files = desitarget.io.iter_files(mock_dir, '', ext=\"fits\")\r\nimport multiprocessing\r\nfile_list = list(iter_mock_files)\r\nnfiles = len(file_list)\r\nncpu = max(1, multiprocessing.cpu_count() // 2)\r\nprint('using {} parallel readers'.format(ncpu))\r\np = multiprocessing.Pool(ncpu)\r\ntarget_list = p.map(desitarget.mock.io._load_mock_mws_file, file_list)\r\n```\r\ngives a long error message starting with\r\n```\r\nTraceback (most recent call last):\r\n  File \"multiproc_bug.py\", line 13, in <module>\r\n    target_list = p.map(desitarget.mock.io._load_mock_mws_file, file_list)        \r\n  File \"/global/common/edison/contrib/desi/conda/conda_3.5-20160829/lib/python3.5/multiprocessing/pool.py\", line 260, in map\r\n    return self._map_async(func, iterable, mapstar, chunksize).get()\r\n  File \"/global/common/edison/contrib/desi/conda/conda_3.5-20160829/lib/python3.5/multiprocessing/pool.py\", line 608, in get\r\n    raise self._value\r\nmultiprocessing.pool.MaybeEncodingError: Error sending result: \r\n````\r\nending with\r\n```\r\n Reason: 'error(\"'i' format requires -2147483648 <= number <= 2147483647\",)'\r\n```\r\n\r\nThis seems to be caused by too large arrays (present in v.0.0.2 of the mocks, not in version v.0.0.1).\r\n\r\nCan you reproduce the error?\r\nAny idea for a workaround?"},"comment":{"url":"https://api.github.com/repos/desihub/desitarget/issues/comments/257738727","html_url":"https://github.com/desihub/desitarget/issues/108#issuecomment-257738727","issue_url":"https://api.github.com/repos/desihub/desitarget/issues/108","id":257738727,"user":{"login":"sbailey","id":218471,"avatar_url":"https://avatars.githubusercontent.com/u/218471?v=3","gravatar_id":"","url":"https://api.github.com/users/sbailey","html_url":"https://github.com/sbailey","followers_url":"https://api.github.com/users/sbailey/followers","following_url":"https://api.github.com/users/sbailey/following{/other_user}","gists_url":"https://api.github.com/users/sbailey/gists{/gist_id}","starred_url":"https://api.github.com/users/sbailey/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sbailey/subscriptions","organizations_url":"https://api.github.com/users/sbailey/orgs","repos_url":"https://api.github.com/users/sbailey/repos","events_url":"https://api.github.com/users/sbailey/events{/privacy}","received_events_url":"https://api.github.com/users/sbailey/received_events","type":"User","site_admin":false},"created_at":"2016-11-02T00:23:53Z","updated_at":"2016-11-02T00:23:53Z","body":"Yes, I can reproduce it, and it appears to be [python issue #17560](https://bugs.python.org/issue17560): `multiprocessing.Pool.map` can't return objects with more than 2**32-1 bytes since it uses a signed 32-bit int for encoding how many bytes it is returning (!).\r\n\r\nUgh.\r\n\r\nThe short term workaround is in PR #110, which in addition to reading Lyman-alpha mocks also returns to using serial I/O for the galaxia mocks.  I'll think about whether there is another easy way to do the parallel I/O ."}}
{"actor":{"display_login":"sbailey"}}