{"action":"created","issue":{"url":"https://api.github.com/repos/dask/distributed/issues/2080","repository_url":"https://api.github.com/repos/dask/distributed","labels_url":"https://api.github.com/repos/dask/distributed/issues/2080/labels{/name}","comments_url":"https://api.github.com/repos/dask/distributed/issues/2080/comments","events_url":"https://api.github.com/repos/dask/distributed/issues/2080/events","html_url":"https://github.com/dask/distributed/issues/2080","id":336675517,"node_id":"MDU6SXNzdWUzMzY2NzU1MTc=","number":2080,"title":"bad marshal data (invalid reference)","user":{"login":"jacobtomlinson","id":1610850,"node_id":"MDQ6VXNlcjE2MTA4NTA=","avatar_url":"https://avatars3.githubusercontent.com/u/1610850?v=4","gravatar_id":"","url":"https://api.github.com/users/jacobtomlinson","html_url":"https://github.com/jacobtomlinson","followers_url":"https://api.github.com/users/jacobtomlinson/followers","following_url":"https://api.github.com/users/jacobtomlinson/following{/other_user}","gists_url":"https://api.github.com/users/jacobtomlinson/gists{/gist_id}","starred_url":"https://api.github.com/users/jacobtomlinson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jacobtomlinson/subscriptions","organizations_url":"https://api.github.com/users/jacobtomlinson/orgs","repos_url":"https://api.github.com/users/jacobtomlinson/repos","events_url":"https://api.github.com/users/jacobtomlinson/events{/privacy}","received_events_url":"https://api.github.com/users/jacobtomlinson/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-06-28T15:40:09Z","updated_at":"2018-06-28T15:48:23Z","closed_at":null,"author_association":"NONE","body":"I am intermittently getting the following error when using dask-kubernetes on an AWS based Pangeo deployment.\r\n\r\nThe cluster has 50 workers and the error seems to happen randomly.\r\n\r\nI've had a google but have not found useful references for that error. Could someone please point me in the right direction?\r\n\r\n```\r\ndistributed.scheduler - ERROR - error from worker tcp://100.96.102.208:45331: bad marshal data (invalid reference)\r\ndistributed.utils - ERROR - bad marshal data (invalid reference)\r\nTraceback (most recent call last):\r\n  File \"/opt/conda/lib/python3.6/site-packages/distributed/utils.py\", line 238, in f\r\n    result[0] = yield make_coro()\r\n  File \"/opt/conda/lib/python3.6/site-packages/tornado/gen.py\", line 1055, in run\r\n    value = future.result()\r\n  File \"/opt/conda/lib/python3.6/site-packages/tornado/concurrent.py\", line 238, in result\r\n    raise_exc_info(self._exc_info)\r\n  File \"<string>\", line 4, in raise_exc_info\r\n  File \"/opt/conda/lib/python3.6/site-packages/tornado/gen.py\", line 1063, in run\r\n    yielded = self.gen.throw(*exc_info)\r\n  File \"/opt/conda/lib/python3.6/site-packages/distributed/client.py\", line 1364, in _gather\r\n    traceback)\r\n  File \"/opt/conda/lib/python3.6/site-packages/six.py\", line 692, in reraise\r\n    raise value.with_traceback(tb)\r\n  File \"/opt/conda/lib/python3.6/site-packages/distributed/protocol/pickle.py\", line 59, in loads\r\n    return pickle.loads(x)\r\n  File \"<frozen importlib._bootstrap>\", line 971, in _find_and_load\r\n  File \"<frozen importlib._bootstrap>\", line 955, in _find_and_load_unlocked\r\n  File \"<frozen importlib._bootstrap>\", line 665, in _load_unlocked\r\n  File \"<frozen importlib._bootstrap_external>\", line 674, in exec_module\r\n  File \"<frozen importlib._bootstrap_external>\", line 779, in get_code\r\n  File \"<frozen importlib._bootstrap_external>\", line 487, in _compile_bytecode\r\nValueError: bad marshal data (invalid reference)\r\ndistributed.scheduler - ERROR - error from worker tcp://100.96.102.210:34063: bad marshal data (invalid reference)\r\ndistributed.scheduler - ERROR - error from worker tcp://100.96.106.207:45867: bad marshal data (unknown type code)\r\n\r\n---------------------------------------------------------------------------\r\nValueError                                Traceback (most recent call last)\r\n<timed exec> in <module>()\r\n\r\n~/eupheme/distributed_version/region_attn.py in load_process_data(startyear, endyear, eventyear, startmonth, endmonth, region, obs, modelroot, modelname, varn, varname_o, freq, lc)\r\n    620 \r\n    621 def load_process_data(startyear,endyear,eventyear,startmonth,endmonth,region,obs,modelroot,modelname,varn,varname_o,freq,lc):\r\n--> 622     all_anom,nat_anom=loadandmask_cmip5da(startyear,endyear,region,obs,modelroot,modelname,varn,varname_o,freq)#GENERIC ATTRIBUTION load routine\r\n    623 \r\n    624     #Look at a single season. Event attribution example. Make sure this is within the model run period!\r\n\r\n~/eupheme/distributed_version/region_attn.py in loadandmask_cmip5da(startyear, endyear, region, obs, modelroot, modelname, varn, varname_o, freq)\r\n    523 def loadandmask_cmip5da(startyear,endyear,region,obs,modelroot,modelname,varn,varname_o,freq):\r\n    524     all_ens=load_region_allforcings_ensemble(region,startyear,endyear,obs[0],modelroot,modelname,varn)#All forcings\r\n--> 525     nat_ens=load_region_ensemble('historicalNat',region,startyear,endyear,obs[0],modelroot,modelname,varn,freq)#Natural forcings\r\n    526 \r\n    527     notlandmask=fl.get_landsea_mask(modelroot+modelname+'/sftlf_fx_'+modelname+'_historical_r0i0p0.nc',obs)\r\n\r\n~/eupheme/distributed_version/region_attn.py in load_region_ensemble(forcinglabel, region, startyear, endyear, obs, modelroot, modelname, diagstr, freq, save)\r\n    175 \r\n    176     list_of_cubes = [regridit(i, i_memb) for i,i_memb in enumerate(uniquemembs)]\r\n--> 177     regionens=dask.delayed(iris.cube.CubeList)(list_of_cubes).compute()\r\n    178 \r\n    179 \r\n\r\n/opt/conda/lib/python3.6/site-packages/dask/base.py in compute(self, **kwargs)\r\n    141         dask.base.compute\r\n    142         \"\"\"\r\n--> 143         (result,) = compute(self, traverse=False, **kwargs)\r\n    144         return result\r\n    145 \r\n\r\n/opt/conda/lib/python3.6/site-packages/dask/base.py in compute(*args, **kwargs)\r\n    390     postcomputes = [a.__dask_postcompute__() if is_dask_collection(a)\r\n    391                     else (None, a) for a in args]\r\n--> 392     results = get(dsk, keys, **kwargs)\r\n    393     results_iter = iter(results)\r\n    394     return tuple(a if f is None else f(next(results_iter), *a)\r\n\r\n/opt/conda/lib/python3.6/site-packages/distributed/client.py in get(self, dsk, keys, restrictions, loose_restrictions, resources, sync, asynchronous, **kwargs)\r\n   2051                     should_rejoin = False\r\n   2052             try:\r\n-> 2053                 results = self.gather(packed, asynchronous=asynchronous)\r\n   2054             finally:\r\n   2055                 for f in futures.values():\r\n\r\n/opt/conda/lib/python3.6/site-packages/distributed/client.py in gather(self, futures, errors, maxsize, direct, asynchronous)\r\n   1484             return self.sync(self._gather, futures, errors=errors,\r\n   1485                              direct=direct, local_worker=local_worker,\r\n-> 1486                              asynchronous=asynchronous)\r\n   1487 \r\n   1488     @gen.coroutine\r\n\r\n/opt/conda/lib/python3.6/site-packages/distributed/client.py in sync(self, func, *args, **kwargs)\r\n    606             return future\r\n    607         else:\r\n--> 608             return sync(self.loop, func, *args, **kwargs)\r\n    609 \r\n    610     def __repr__(self):\r\n\r\n/opt/conda/lib/python3.6/site-packages/distributed/utils.py in sync(loop, func, *args, **kwargs)\r\n    252             e.wait(10)\r\n    253     if error[0]:\r\n--> 254         six.reraise(*error[0])\r\n    255     else:\r\n    256         return result[0]\r\n\r\n/opt/conda/lib/python3.6/site-packages/six.py in reraise(tp, value, tb)\r\n    691             if value.__traceback__ is not tb:\r\n    692                 raise value.with_traceback(tb)\r\n--> 693             raise value\r\n    694         finally:\r\n    695             value = None\r\n\r\n/opt/conda/lib/python3.6/site-packages/distributed/utils.py in f()\r\n    236             yield gen.moment\r\n    237             thread_state.asynchronous = True\r\n--> 238             result[0] = yield make_coro()\r\n    239         except Exception as exc:\r\n    240             logger.exception(exc)\r\n\r\n/opt/conda/lib/python3.6/site-packages/tornado/gen.py in run(self)\r\n   1053 \r\n   1054                     try:\r\n-> 1055                         value = future.result()\r\n   1056                     except Exception:\r\n   1057                         self.had_exception = True\r\n\r\n/opt/conda/lib/python3.6/site-packages/tornado/concurrent.py in result(self, timeout)\r\n    236         if self._exc_info is not None:\r\n    237             try:\r\n--> 238                 raise_exc_info(self._exc_info)\r\n    239             finally:\r\n    240                 self = None\r\n\r\n/opt/conda/lib/python3.6/site-packages/tornado/util.py in raise_exc_info(exc_info)\r\n\r\n/opt/conda/lib/python3.6/site-packages/tornado/gen.py in run(self)\r\n   1061                     if exc_info is not None:\r\n   1062                         try:\r\n-> 1063                             yielded = self.gen.throw(*exc_info)\r\n   1064                         finally:\r\n   1065                             # Break up a reference to itself\r\n\r\n/opt/conda/lib/python3.6/site-packages/distributed/client.py in _gather(self, futures, errors, direct, local_worker)\r\n   1362                             six.reraise(type(exception),\r\n   1363                                         exception,\r\n-> 1364                                         traceback)\r\n   1365                     if errors == 'skip':\r\n   1366                         bad_keys.add(key)\r\n\r\n/opt/conda/lib/python3.6/site-packages/six.py in reraise(tp, value, tb)\r\n    690                 value = tp()\r\n    691             if value.__traceback__ is not tb:\r\n--> 692                 raise value.with_traceback(tb)\r\n    693             raise value\r\n    694         finally:\r\n\r\n/opt/conda/lib/python3.6/site-packages/distributed/protocol/pickle.py in loads()\r\n     57 def loads(x):\r\n     58     try:\r\n---> 59         return pickle.loads(x)\r\n     60     except Exception:\r\n     61         logger.info(\"Failed to deserialize %s\", x[:10000], exc_info=True)\r\n\r\n/opt/conda/lib/python3.6/importlib/_bootstrap.py in _find_and_load()\r\n\r\n/opt/conda/lib/python3.6/importlib/_bootstrap.py in _find_and_load_unlocked()\r\n\r\n/opt/conda/lib/python3.6/importlib/_bootstrap.py in _load_unlocked()\r\n\r\n/opt/conda/lib/python3.6/importlib/_bootstrap_external.py in exec_module()\r\n\r\n/opt/conda/lib/python3.6/importlib/_bootstrap_external.py in get_code()\r\n\r\n/opt/conda/lib/python3.6/importlib/_bootstrap_external.py in _compile_bytecode()\r\n\r\nValueError: bad marshal data (invalid reference)\r\n```"},"comment":{"url":"https://api.github.com/repos/dask/distributed/issues/comments/401081667","html_url":"https://github.com/dask/distributed/issues/2080#issuecomment-401081667","issue_url":"https://api.github.com/repos/dask/distributed/issues/2080","id":401081667,"node_id":"MDEyOklzc3VlQ29tbWVudDQwMTA4MTY2Nw==","user":{"login":"mrocklin","id":306380,"node_id":"MDQ6VXNlcjMwNjM4MA==","avatar_url":"https://avatars3.githubusercontent.com/u/306380?v=4","gravatar_id":"","url":"https://api.github.com/users/mrocklin","html_url":"https://github.com/mrocklin","followers_url":"https://api.github.com/users/mrocklin/followers","following_url":"https://api.github.com/users/mrocklin/following{/other_user}","gists_url":"https://api.github.com/users/mrocklin/gists{/gist_id}","starred_url":"https://api.github.com/users/mrocklin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mrocklin/subscriptions","organizations_url":"https://api.github.com/users/mrocklin/orgs","repos_url":"https://api.github.com/users/mrocklin/repos","events_url":"https://api.github.com/users/mrocklin/events{/privacy}","received_events_url":"https://api.github.com/users/mrocklin/received_events","type":"User","site_admin":false},"created_at":"2018-06-28T15:48:22Z","updated_at":"2018-06-28T15:48:22Z","author_association":"MEMBER","body":"At first glance I don't know what's going on.  It looks like something\nisn't deserializing properly.  It looks like we might dump a bit of the\nserialized data to the worker's log.  Maybe that gives a clue?\n\nOn Thu, Jun 28, 2018 at 11:40 AM, Jacob Tomlinson <notifications@github.com>\nwrote:\n\n> I am intermittently getting the following error when using dask-kubernetes\n> on an AWS based Pangeo deployment.\n>\n> The cluster has 50 workers and the error seems to happen randomly.\n>\n> I've had a google but have not found useful references for that error.\n> Could someone please point me in the right direction?\n>\n> distributed.scheduler - ERROR - error from worker tcp://100.96.102.208:45331: bad marshal data (invalid reference)\n> distributed.utils - ERROR - bad marshal data (invalid reference)\n> Traceback (most recent call last):\n>   File \"/opt/conda/lib/python3.6/site-packages/distributed/utils.py\", line 238, in f\n>     result[0] = yield make_coro()\n>   File \"/opt/conda/lib/python3.6/site-packages/tornado/gen.py\", line 1055, in run\n>     value = future.result()\n>   File \"/opt/conda/lib/python3.6/site-packages/tornado/concurrent.py\", line 238, in result\n>     raise_exc_info(self._exc_info)\n>   File \"<string>\", line 4, in raise_exc_info\n>   File \"/opt/conda/lib/python3.6/site-packages/tornado/gen.py\", line 1063, in run\n>     yielded = self.gen.throw(*exc_info)\n>   File \"/opt/conda/lib/python3.6/site-packages/distributed/client.py\", line 1364, in _gather\n>     traceback)\n>   File \"/opt/conda/lib/python3.6/site-packages/six.py\", line 692, in reraise\n>     raise value.with_traceback(tb)\n>   File \"/opt/conda/lib/python3.6/site-packages/distributed/protocol/pickle.py\", line 59, in loads\n>     return pickle.loads(x)\n>   File \"<frozen importlib._bootstrap>\", line 971, in _find_and_load\n>   File \"<frozen importlib._bootstrap>\", line 955, in _find_and_load_unlocked\n>   File \"<frozen importlib._bootstrap>\", line 665, in _load_unlocked\n>   File \"<frozen importlib._bootstrap_external>\", line 674, in exec_module\n>   File \"<frozen importlib._bootstrap_external>\", line 779, in get_code\n>   File \"<frozen importlib._bootstrap_external>\", line 487, in _compile_bytecode\n> ValueError: bad marshal data (invalid reference)\n> distributed.scheduler - ERROR - error from worker tcp://100.96.102.210:34063: bad marshal data (invalid reference)\n> distributed.scheduler - ERROR - error from worker tcp://100.96.106.207:45867: bad marshal data (unknown type code)\n>\n> ---------------------------------------------------------------------------\n> ValueError                                Traceback (most recent call last)\n> <timed exec> in <module>()\n>\n> ~/eupheme/distributed_version/region_attn.py in load_process_data(startyear, endyear, eventyear, startmonth, endmonth, region, obs, modelroot, modelname, varn, varname_o, freq, lc)\n>     620\n>     621 def load_process_data(startyear,endyear,eventyear,startmonth,endmonth,region,obs,modelroot,modelname,varn,varname_o,freq,lc):\n> --> 622     all_anom,nat_anom=loadandmask_cmip5da(startyear,endyear,region,obs,modelroot,modelname,varn,varname_o,freq)#GENERIC ATTRIBUTION load routine\n>     623\n>     624     #Look at a single season. Event attribution example. Make sure this is within the model run period!\n>\n> ~/eupheme/distributed_version/region_attn.py in loadandmask_cmip5da(startyear, endyear, region, obs, modelroot, modelname, varn, varname_o, freq)\n>     523 def loadandmask_cmip5da(startyear,endyear,region,obs,modelroot,modelname,varn,varname_o,freq):\n>     524     all_ens=load_region_allforcings_ensemble(region,startyear,endyear,obs[0],modelroot,modelname,varn)#All forcings\n> --> 525     nat_ens=load_region_ensemble('historicalNat',region,startyear,endyear,obs[0],modelroot,modelname,varn,freq)#Natural forcings\n>     526\n>     527     notlandmask=fl.get_landsea_mask(modelroot+modelname+'/sftlf_fx_'+modelname+'_historical_r0i0p0.nc',obs)\n>\n> ~/eupheme/distributed_version/region_attn.py in load_region_ensemble(forcinglabel, region, startyear, endyear, obs, modelroot, modelname, diagstr, freq, save)\n>     175\n>     176     list_of_cubes = [regridit(i, i_memb) for i,i_memb in enumerate(uniquemembs)]\n> --> 177     regionens=dask.delayed(iris.cube.CubeList)(list_of_cubes).compute()\n>     178\n>     179\n>\n> /opt/conda/lib/python3.6/site-packages/dask/base.py in compute(self, **kwargs)\n>     141         dask.base.compute\n>     142         \"\"\"\n> --> 143         (result,) = compute(self, traverse=False, **kwargs)\n>     144         return result\n>     145\n>\n> /opt/conda/lib/python3.6/site-packages/dask/base.py in compute(*args, **kwargs)\n>     390     postcomputes = [a.__dask_postcompute__() if is_dask_collection(a)\n>     391                     else (None, a) for a in args]\n> --> 392     results = get(dsk, keys, **kwargs)\n>     393     results_iter = iter(results)\n>     394     return tuple(a if f is None else f(next(results_iter), *a)\n>\n> /opt/conda/lib/python3.6/site-packages/distributed/client.py in get(self, dsk, keys, restrictions, loose_restrictions, resources, sync, asynchronous, **kwargs)\n>    2051                     should_rejoin = False\n>    2052             try:\n> -> 2053                 results = self.gather(packed, asynchronous=asynchronous)\n>    2054             finally:\n>    2055                 for f in futures.values():\n>\n> /opt/conda/lib/python3.6/site-packages/distributed/client.py in gather(self, futures, errors, maxsize, direct, asynchronous)\n>    1484             return self.sync(self._gather, futures, errors=errors,\n>    1485                              direct=direct, local_worker=local_worker,\n> -> 1486                              asynchronous=asynchronous)\n>    1487\n>    1488     @gen.coroutine\n>\n> /opt/conda/lib/python3.6/site-packages/distributed/client.py in sync(self, func, *args, **kwargs)\n>     606             return future\n>     607         else:\n> --> 608             return sync(self.loop, func, *args, **kwargs)\n>     609\n>     610     def __repr__(self):\n>\n> /opt/conda/lib/python3.6/site-packages/distributed/utils.py in sync(loop, func, *args, **kwargs)\n>     252             e.wait(10)\n>     253     if error[0]:\n> --> 254         six.reraise(*error[0])\n>     255     else:\n>     256         return result[0]\n>\n> /opt/conda/lib/python3.6/site-packages/six.py in reraise(tp, value, tb)\n>     691             if value.__traceback__ is not tb:\n>     692                 raise value.with_traceback(tb)\n> --> 693             raise value\n>     694         finally:\n>     695             value = None\n>\n> /opt/conda/lib/python3.6/site-packages/distributed/utils.py in f()\n>     236             yield gen.moment\n>     237             thread_state.asynchronous = True\n> --> 238             result[0] = yield make_coro()\n>     239         except Exception as exc:\n>     240             logger.exception(exc)\n>\n> /opt/conda/lib/python3.6/site-packages/tornado/gen.py in run(self)\n>    1053\n>    1054                     try:\n> -> 1055                         value = future.result()\n>    1056                     except Exception:\n>    1057                         self.had_exception = True\n>\n> /opt/conda/lib/python3.6/site-packages/tornado/concurrent.py in result(self, timeout)\n>     236         if self._exc_info is not None:\n>     237             try:\n> --> 238                 raise_exc_info(self._exc_info)\n>     239             finally:\n>     240                 self = None\n>\n> /opt/conda/lib/python3.6/site-packages/tornado/util.py in raise_exc_info(exc_info)\n>\n> /opt/conda/lib/python3.6/site-packages/tornado/gen.py in run(self)\n>    1061                     if exc_info is not None:\n>    1062                         try:\n> -> 1063                             yielded = self.gen.throw(*exc_info)\n>    1064                         finally:\n>    1065                             # Break up a reference to itself\n>\n> /opt/conda/lib/python3.6/site-packages/distributed/client.py in _gather(self, futures, errors, direct, local_worker)\n>    1362                             six.reraise(type(exception),\n>    1363                                         exception,\n> -> 1364                                         traceback)\n>    1365                     if errors == 'skip':\n>    1366                         bad_keys.add(key)\n>\n> /opt/conda/lib/python3.6/site-packages/six.py in reraise(tp, value, tb)\n>     690                 value = tp()\n>     691             if value.__traceback__ is not tb:\n> --> 692                 raise value.with_traceback(tb)\n>     693             raise value\n>     694         finally:\n>\n> /opt/conda/lib/python3.6/site-packages/distributed/protocol/pickle.py in loads()\n>      57 def loads(x):\n>      58     try:\n> ---> 59         return pickle.loads(x)\n>      60     except Exception:\n>      61         logger.info(\"Failed to deserialize %s\", x[:10000], exc_info=True)\n>\n> /opt/conda/lib/python3.6/importlib/_bootstrap.py in _find_and_load()\n>\n> /opt/conda/lib/python3.6/importlib/_bootstrap.py in _find_and_load_unlocked()\n>\n> /opt/conda/lib/python3.6/importlib/_bootstrap.py in _load_unlocked()\n>\n> /opt/conda/lib/python3.6/importlib/_bootstrap_external.py in exec_module()\n>\n> /opt/conda/lib/python3.6/importlib/_bootstrap_external.py in get_code()\n>\n> /opt/conda/lib/python3.6/importlib/_bootstrap_external.py in _compile_bytecode()\n>\n> ValueError: bad marshal data (invalid reference)\n>\n> —\n> You are receiving this because you are subscribed to this thread.\n> Reply to this email directly, view it on GitHub\n> <https://github.com/dask/distributed/issues/2080>, or mute the thread\n> <https://github.com/notifications/unsubscribe-auth/AASszKKviUbbw6lgwcmQpg_lGkNBuCLCks5uBPjegaJpZM4U7n-W>\n> .\n>\n"}}
{"actor":{"display_login":"mrocklin"}}