{"action":"created","issue":{"url":"https://api.github.com/repos/XENON1T/pax/issues/463","repository_url":"https://api.github.com/repos/XENON1T/pax","labels_url":"https://api.github.com/repos/XENON1T/pax/issues/463/labels{/name}","comments_url":"https://api.github.com/repos/XENON1T/pax/issues/463/comments","events_url":"https://api.github.com/repos/XENON1T/pax/issues/463/events","html_url":"https://github.com/XENON1T/pax/issues/463","id":191574437,"number":463,"title":"Pax multiprocessing crash ProcessBatchQueue","user":{"login":"pdeperio","id":8642192,"avatar_url":"https://avatars.githubusercontent.com/u/8642192?v=3","gravatar_id":"","url":"https://api.github.com/users/pdeperio","html_url":"https://github.com/pdeperio","followers_url":"https://api.github.com/users/pdeperio/followers","following_url":"https://api.github.com/users/pdeperio/following{/other_user}","gists_url":"https://api.github.com/users/pdeperio/gists{/gist_id}","starred_url":"https://api.github.com/users/pdeperio/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pdeperio/subscriptions","organizations_url":"https://api.github.com/users/pdeperio/orgs","repos_url":"https://api.github.com/users/pdeperio/repos","events_url":"https://api.github.com/users/pdeperio/events{/privacy}","received_events_url":"https://api.github.com/users/pdeperio/received_events","type":"User","site_admin":false},"labels":[{"id":112979072,"url":"https://api.github.com/repos/XENON1T/pax/labels/bug","name":"bug","color":"fc2929","default":true},{"id":112979076,"url":"https://api.github.com/repos/XENON1T/pax/labels/invalid","name":"invalid","color":"e6e6e6","default":true},{"id":215488778,"url":"https://api.github.com/repos/XENON1T/pax/labels/processed%20data%20io","name":"processed data io","color":"fad8c7","default":false}],"state":"open","locked":false,"assignee":{"login":"Ludwig1987","id":15890679,"avatar_url":"https://avatars.githubusercontent.com/u/15890679?v=3","gravatar_id":"","url":"https://api.github.com/users/Ludwig1987","html_url":"https://github.com/Ludwig1987","followers_url":"https://api.github.com/users/Ludwig1987/followers","following_url":"https://api.github.com/users/Ludwig1987/following{/other_user}","gists_url":"https://api.github.com/users/Ludwig1987/gists{/gist_id}","starred_url":"https://api.github.com/users/Ludwig1987/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Ludwig1987/subscriptions","organizations_url":"https://api.github.com/users/Ludwig1987/orgs","repos_url":"https://api.github.com/users/Ludwig1987/repos","events_url":"https://api.github.com/users/Ludwig1987/events{/privacy}","received_events_url":"https://api.github.com/users/Ludwig1987/received_events","type":"User","site_admin":false},"assignees":[{"login":"Ludwig1987","id":15890679,"avatar_url":"https://avatars.githubusercontent.com/u/15890679?v=3","gravatar_id":"","url":"https://api.github.com/users/Ludwig1987","html_url":"https://github.com/Ludwig1987","followers_url":"https://api.github.com/users/Ludwig1987/followers","following_url":"https://api.github.com/users/Ludwig1987/following{/other_user}","gists_url":"https://api.github.com/users/Ludwig1987/gists{/gist_id}","starred_url":"https://api.github.com/users/Ludwig1987/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Ludwig1987/subscriptions","organizations_url":"https://api.github.com/users/Ludwig1987/orgs","repos_url":"https://api.github.com/users/Ludwig1987/repos","events_url":"https://api.github.com/users/Ludwig1987/events{/privacy}","received_events_url":"https://api.github.com/users/Ludwig1987/received_events","type":"User","site_admin":false}],"milestone":null,"comments":21,"created_at":"2016-11-24T18:11:58Z","updated_at":"2016-12-05T15:37:21Z","closed_at":null,"body":"_From @lucrlom on November 24, 2016 17:5_\r\n\r\nI had this problem at least in two runs 4696 and 4700 processing with `massive-cax --once --run 4xxx` with pax_v6.1.0 environment:\r\n```\r\n`cax_v4.10.1 - 2016-11-24 04:15:23,038 [CRITICAL] Exception caught from task ProcessBatchQueue\r\nTraceback (most recent call last):\r\n  File \"/project/lgrandi/anaconda3/envs/pax_v6.1.0/lib/python3.4/site-packages/cax-4.10.1-py3.4.egg/cax/main.py\", line 120, in main\r\n    task.go(args.run)\r\n  File \"/project/lgrandi/anaconda3/envs/pax_v6.1.0/lib/python3.4/site-packages/cax-4.10.1-py3.4.egg/cax/task.py\", line 65, in go\r\n    self.each_run()\r\n  File \"/project/lgrandi/anaconda3/envs/pax_v6.1.0/lib/python3.4/site-packages/cax-4.10.1-py3.4.egg/cax/tasks/process.py\", line 211, in each_run\r\n    ncpus)\r\n  File \"/project/lgrandi/anaconda3/envs/pax_v6.1.0/lib/python3.4/site-packages/cax-4.10.1-py3.4.egg/cax/tasks/process.py\", line 98, in _process\r\n    parallel.multiprocess_locally(n_cpus=ncpus, **pax_kwargs)\r\n  File \"/project/lgrandi/anaconda3/envs/pax_v6.1.0/lib/python3.4/site-packages/pax-6.1.0-py3.4.egg/pax/parallel.py\", line 205, in multiprocess_locally\r\n    traceback)\r\npax.exceptions.EventBlockHeapSizeExceededException: Pax multiprocessing crashed due to exception in one of the workers. Dumping traceback:\r\nTraceback (most recent call last):\r\n  File \"/project/lgrandi/anaconda3/envs/pax_v6.1.0/lib/python3.4/site-packages/pax-6.1.0-py3.4.egg/pax/parallel.py\", line 356, in safe_processor\r\n    Processor(**kwargs)\r\n  File \"/project/lgrandi/anaconda3/envs/pax_v6.1.0/lib/python3.4/site-packages/pax-6.1.0-py3.4.egg/pax/core.py\", line 186, in __init__\r\n    self.run()\r\n  File \"/project/lgrandi/anaconda3/envs/pax_v6.1.0/lib/python3.4/site-packages/pax-6.1.0-py3.4.egg/pax/core.py\", line 310, in run\r\n    total=self.number_of_events)):\r\n  File \"/project/lgrandi/anaconda3/envs/pax_v6.1.0/lib/python3.4/site-packages/pax-6.1.0-py3.4.egg/pax/plugins/io/Queues.py\", line 97, in get_events\r\n    self.max_blocks_on_heap, block_id + 1))\r\npax.exceptions.EventBlockHeapSizeExceededException: We have received over 250 blocks without receiving the next block id (1739) in order. Likely one of the block producers has died without telling anyone.\r\n\r\ncax_v4.10.1 - 2016-11-24 04:15:23,149 [ERROR] Pax multiprocessing crashed due to exception in one of the workers. Dumping traceback:\r\nTraceback (most recent call last):\r\n  File \"/project/lgrandi/anaconda3/envs/pax_v6.1.0/lib/python3.4/site-packages/pax-6.1.0-py3.4.egg/pax/parallel.py\", line 356, in safe_processor\r\n    Processor(**kwargs)\r\n  File \"/project/lgrandi/anaconda3/envs/pax_v6.1.0/lib/python3.4/site-packages/pax-6.1.0-py3.4.egg/pax/core.py\", line 186, in __init__\r\n    self.run()\r\n  File \"/project/lgrandi/anaconda3/envs/pax_v6.1.0/lib/python3.4/site-packages/pax-6.1.0-py3.4.egg/pax/core.py\", line 310, in run\r\n    total=self.number_of_events)):\r\n  File \"/project/lgrandi/anaconda3/envs/pax_v6.1.0/lib/python3.4/site-packages/pax-6.1.0-py3.4.egg/pax/plugins/io/Queues.py\", line 97, in get_events\r\n    self.max_blocks_on_heap, block_id + 1))\r\npax.exceptions.EventBlockHeapSizeExceededException: We have received over 250 blocks without receiving the next block id (1739) in order. Likely one of the block producers has died without telling anyone`\r\n```\r\n\r\n_Copied from original issue: XENON1T/cax#60_"},"comment":{"url":"https://api.github.com/repos/XENON1T/pax/issues/comments/264886161","html_url":"https://github.com/XENON1T/pax/issues/463#issuecomment-264886161","issue_url":"https://api.github.com/repos/XENON1T/pax/issues/463","id":264886161,"user":{"login":"pdeperio","id":8642192,"avatar_url":"https://avatars.githubusercontent.com/u/8642192?v=3","gravatar_id":"","url":"https://api.github.com/users/pdeperio","html_url":"https://github.com/pdeperio","followers_url":"https://api.github.com/users/pdeperio/followers","following_url":"https://api.github.com/users/pdeperio/following{/other_user}","gists_url":"https://api.github.com/users/pdeperio/gists{/gist_id}","starred_url":"https://api.github.com/users/pdeperio/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pdeperio/subscriptions","organizations_url":"https://api.github.com/users/pdeperio/orgs","repos_url":"https://api.github.com/users/pdeperio/repos","events_url":"https://api.github.com/users/pdeperio/events{/privacy}","received_events_url":"https://api.github.com/users/pdeperio/received_events","type":"User","site_admin":false},"created_at":"2016-12-05T15:37:21Z","updated_at":"2016-12-05T15:37:21Z","body":"Just died again now and disk seems fine, e.g. ```/project/lgrandi/xenon1t/cax/5025_v6.1.0/5025_v6.1.0_22207657.log```"}}
{"actor":{"display_login":"pdeperio"}}