{"action":"created","issue":{"url":"https://api.github.com/repos/mne-tools/mne-python/issues/3190","repository_url":"https://api.github.com/repos/mne-tools/mne-python","labels_url":"https://api.github.com/repos/mne-tools/mne-python/issues/3190/labels{/name}","comments_url":"https://api.github.com/repos/mne-tools/mne-python/issues/3190/comments","events_url":"https://api.github.com/repos/mne-tools/mne-python/issues/3190/events","html_url":"https://github.com/mne-tools/mne-python/issues/3190","id":151743903,"number":3190,"title":"BUG: mne.parallel crashes when too much data","user":{"login":"kingjr","id":4881164,"avatar_url":"https://avatars.githubusercontent.com/u/4881164?v=3","gravatar_id":"","url":"https://api.github.com/users/kingjr","html_url":"https://github.com/kingjr","followers_url":"https://api.github.com/users/kingjr/followers","following_url":"https://api.github.com/users/kingjr/following{/other_user}","gists_url":"https://api.github.com/users/kingjr/gists{/gist_id}","starred_url":"https://api.github.com/users/kingjr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kingjr/subscriptions","organizations_url":"https://api.github.com/users/kingjr/orgs","repos_url":"https://api.github.com/users/kingjr/repos","events_url":"https://api.github.com/users/kingjr/events{/privacy}","received_events_url":"https://api.github.com/users/kingjr/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":27,"created_at":"2016-04-28T21:09:02Z","updated_at":"2016-07-12T21:26:11Z","closed_at":null,"body":"I'm encountering some weird error when I try to decod too much data e.g.\r\n\r\n```\r\nimport numpy as np\r\nfrom mne import create_info, EpochsArray\r\nfrom mne.decoding import TimeDecoding\r\n\r\n# simulate data\r\nn_trial, n_chan, n_time = 800, 300, 1445\r\ninfo = create_info(n_chan, 128., 'mag')\r\nepochs = EpochsArray(np.random.rand(n_trial, n_chan, n_time),\r\n                     info=info,\r\n                     events=np.zeros((n_trial, 3), int))\r\ny = np.random.randint(0, 2, n_trial)\r\nepochs._data[y==1, :, :] += 1\r\n\r\n# fit\r\ntd = TimeDecoding(n_jobs=-1, predict_mode = 'mean-prediction')\r\ntd.fit(epochs, y=y)\r\n```\r\n\r\nworks fine but\r\n\r\n```\r\ntd.predict(epochs)\r\n```\r\n\r\nreturns\r\n\r\n```\r\n---------------------------------------------------------------------------\r\nSystemError                               Traceback (most recent call last)\r\n<ipython-input-22-274c297025ed> in <module>()\r\n     13 td = TimeDecoding(n_jobs=-1)\r\n     14 td.fit(epochs, y=y)\r\n---> 15 y_pred = td.predict(epochs)\r\n\r\n/home/ubuntu/mne-python/mne/decoding/time_gen.py in predict(self, epochs)\r\n   1339         \"\"\"  # noqa\r\n   1340         self._prep_times()\r\n-> 1341         super(TimeDecoding, self).predict(epochs)\r\n   1342         self._clean_times()\r\n   1343         return self.y_pred_\r\n\r\n/home/ubuntu/mne-python/mne/decoding/time_gen.py in predict(self, epochs)\r\n    291             n_orig_epochs=n_orig_epochs, test_epochs=test_epochs,\r\n    292             **dict(zip(['X', 'train_times'], _chunk_data(X, chunk))))\r\n--> 293             for chunk in chunks)\r\n    294 \r\n    295         # Concatenate chunks across test time dimension.\r\n\r\n/home/ubuntu/anaconda/lib/python2.7/site-packages/joblib/parallel.pyc in __call__(self, iterable)\r\n    808                 # consumption.\r\n    809                 self._iterating = False\r\n--> 810             self.retrieve()\r\n    811             # Make sure that we get a last message telling us we are done\r\n    812             elapsed_time = time.time() - self._start_time\r\n\r\n/home/ubuntu/anaconda/lib/python2.7/site-packages/joblib/parallel.pyc in retrieve(self)\r\n    725                 job = self._jobs.pop(0)\r\n    726             try:\r\n--> 727                 self._output.extend(job.get())\r\n    728             except tuple(self.exceptions) as exception:\r\n    729                 # Stop dispatching any new job in the async callback thread\r\n\r\n/home/ubuntu/anaconda/lib/python2.7/multiprocessing/pool.pyc in get(self, timeout)\r\n    565             return self._value\r\n    566         else:\r\n--> 567             raise self._value\r\n    568 \r\n    569     def _set(self, i, obj):\r\n\r\nSystemError: error return without exception set\r\n```\r\n\r\nor\r\n\r\n```\r\nIOError: bad message length\r\n```\r\n\r\nThere is no error if I set `n_jobs=1`, if I reduce the dimensionality of the data (either `n_chan`, `n_time` or `n_trial`), and if I predict a subset: e.g.\r\n\r\n```\r\ntd.predict(epochs[:596]) \r\n```\r\n\r\nworks fine, but from 597, it crashes\r\n\r\n```\r\ntd.predict(epochs[:597]) \r\n```\r\n\r\nI am running on an aws m4.4xlarge running with ubuntu anaconda and mne dev, joblib is 0.9.4.\r\n\r\nI have no idea what to do..."},"comment":{"url":"https://api.github.com/repos/mne-tools/mne-python/issues/comments/232186639","html_url":"https://github.com/mne-tools/mne-python/issues/3190#issuecomment-232186639","issue_url":"https://api.github.com/repos/mne-tools/mne-python/issues/3190","id":232186639,"user":{"login":"agramfort","id":161052,"avatar_url":"https://avatars.githubusercontent.com/u/161052?v=3","gravatar_id":"","url":"https://api.github.com/users/agramfort","html_url":"https://github.com/agramfort","followers_url":"https://api.github.com/users/agramfort/followers","following_url":"https://api.github.com/users/agramfort/following{/other_user}","gists_url":"https://api.github.com/users/agramfort/gists{/gist_id}","starred_url":"https://api.github.com/users/agramfort/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/agramfort/subscriptions","organizations_url":"https://api.github.com/users/agramfort/orgs","repos_url":"https://api.github.com/users/agramfort/repos","events_url":"https://api.github.com/users/agramfort/events{/privacy}","received_events_url":"https://api.github.com/users/agramfort/received_events","type":"User","site_admin":false},"created_at":"2016-07-12T21:26:11Z","updated_at":"2016-07-12T21:26:11Z","body":"whatever works but please bench if you change the default behavior. I trust\nthe joblib folks about good defaults though\n"}}
{"actor":{"display_login":"agramfort"}}