{"action":"closed","issue":{"url":"https://api.github.com/repos/celery/celery/issues/3128","repository_url":"https://api.github.com/repos/celery/celery","labels_url":"https://api.github.com/repos/celery/celery/issues/3128/labels{/name}","comments_url":"https://api.github.com/repos/celery/celery/issues/3128/comments","events_url":"https://api.github.com/repos/celery/celery/issues/3128/events","html_url":"https://github.com/celery/celery/issues/3128","id":142561275,"number":3128,"title":"WorkerLostError on numpy.dot operation","user":{"login":"allwinpb","id":4397532,"avatar_url":"https://avatars.githubusercontent.com/u/4397532?v=3","gravatar_id":"","url":"https://api.github.com/users/allwinpb","html_url":"https://github.com/allwinpb","followers_url":"https://api.github.com/users/allwinpb/followers","following_url":"https://api.github.com/users/allwinpb/following{/other_user}","gists_url":"https://api.github.com/users/allwinpb/gists{/gist_id}","starred_url":"https://api.github.com/users/allwinpb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/allwinpb/subscriptions","organizations_url":"https://api.github.com/users/allwinpb/orgs","repos_url":"https://api.github.com/users/allwinpb/repos","events_url":"https://api.github.com/users/allwinpb/events{/privacy}","received_events_url":"https://api.github.com/users/allwinpb/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2016-03-22T06:38:24Z","updated_at":"2016-06-24T01:35:53Z","closed_at":"2016-06-24T01:35:53Z","body":"As above. A numpy dot operation on a medium sized numpy 2d matrix seems to segfault when run on a worker but works correctly when run on the regular python interpreter. Very trivial cases execute without issues. For example,\r\n```\r\nimport numpy as np\r\nsample_a = [[0,0],[0,1]]\r\nsample_b = [[1,1],[0,1]]\r\nprint np.dot(sample_a, sample_b)\r\n```\r\nwill execute and print properly in both regular python and inside a Celery worker.\r\n\r\nHowever, larger ndarrays will Segfault and throw the above mentioned WorkerLostError when run inside a worker. Works perfectly fine when run on the interpreter.\r\n```\r\nsample_a = np.load('sample_a.npy')\r\nsample_b = np.load('sample_b.npy')\r\nsample_c = np.dot(sample_a, sample_b)\r\n```\r\nThe above mentioned `sample_a.npy` and `sample_b.npy` have shapes `(633,113)` and `(113, 633)` respectively. They can be downloaded here: https://www.dropbox.com/sh/caie9f07b34hzs3/AAC7g3Ns_ttNOUeLYYovDqMla?dl=0. I can't seem to attach it with this Issue because of Github restrictions. In any case the error should be reproducible with any numpy array of similar shape.\r\n\r\nI'm using the Jobtastic wrapper on top of Celery for the extra features but I'm fairly certain the problem is not there. Please correct if this is a bad assumption.\r\n\r\nTraceback with `faulthandler`:\r\n```\r\nFatal Python error: Segmentation fault\r\n\r\nCurrent thread 0x00007fff723f2000 (most recent call first):\r\n  File \"/Users/allwin/Projects/contrast/tweetsearch/analyzers.py\", line 133 in calculate_result\r\n  File \"/Users/allwin/Envs/fyp/lib/python2.7/site-packages/jobtastic/task.py\", line 362 in run\r\n  File \"/Users/allwin/Envs/fyp/lib/python2.7/site-packages/celery/app/trace.py\", line 438 in __protected_call__\r\n  File \"/Users/allwin/Envs/fyp/lib/python2.7/site-packages/celery/app/trace.py\", line 240 in trace_task\r\n  File \"/Users/allwin/Envs/fyp/lib/python2.7/site-packages/celery/app/trace.py\", line 349 in _fast_trace_task\r\n  File \"/Users/allwin/Envs/fyp/lib/python2.7/site-packages/billiard/pool.py\", line 367 in workloop\r\n  File \"/Users/allwin/Envs/fyp/lib/python2.7/site-packages/billiard/pool.py\", line 295 in run\r\n  File \"/Users/allwin/Envs/fyp/lib/python2.7/site-packages/billiard/process.py\", line 292 in _bootstrap\r\n  File \"/Users/allwin/Envs/fyp/lib/python2.7/site-packages/billiard/forking.py\", line 105 in __init__\r\n  File \"/Users/allwin/Envs/fyp/lib/python2.7/site-packages/billiard/process.py\", line 137 in start\r\n  File \"/Users/allwin/Envs/fyp/lib/python2.7/site-packages/billiard/pool.py\", line 1068 in _create_worker_process\r\n  File \"/Users/allwin/Envs/fyp/lib/python2.7/site-packages/celery/concurrency/asynpool.py\", line 415 in _create_worker_process\r\n  File \"/Users/allwin/Envs/fyp/lib/python2.7/site-packages/billiard/pool.py\", line 972 in __init__\r\n  File \"/Users/allwin/Envs/fyp/lib/python2.7/site-packages/celery/concurrency/asynpool.py\", line 401 in __init__\r\n  File \"/Users/allwin/Envs/fyp/lib/python2.7/site-packages/celery/concurrency/prefork.py\", line 119 in on_start\r\n  File \"/Users/allwin/Envs/fyp/lib/python2.7/site-packages/celery/concurrency/base.py\", line 131 in start\r\n  File \"/Users/allwin/Envs/fyp/lib/python2.7/site-packages/celery/bootsteps.py\", line 374 in start\r\n  File \"/Users/allwin/Envs/fyp/lib/python2.7/site-packages/celery/bootsteps.py\", line 123 in start\r\n  File \"/Users/allwin/Envs/fyp/lib/python2.7/site-packages/celery/worker/__init__.py\", line 206 in start\r\n  File \"/Users/allwin/Envs/fyp/lib/python2.7/site-packages/celery/bin/worker.py\", line 212 in run\r\n  File \"/Users/allwin/Envs/fyp/lib/python2.7/site-packages/celery/bin/base.py\", line 274 in __call__\r\n  File \"/Users/allwin/Envs/fyp/lib/python2.7/site-packages/celery/bin/worker.py\", line 179 in run_from_argv\r\n  File \"/Users/allwin/Envs/fyp/lib/python2.7/site-packages/celery/bin/celery.py\", line 717 in execute\r\n  File \"/Users/allwin/Envs/fyp/lib/python2.7/site-packages/celery/bin/celery.py\", line 785 in handle_argv\r\n  File \"/Users/allwin/Envs/fyp/lib/python2.7/site-packages/celery/bin/base.py\", line 311 in execute_from_commandline\r\n  File \"/Users/allwin/Envs/fyp/lib/python2.7/site-packages/celery/bin/celery.py\", line 793 in execute_from_commandline\r\n  File \"/Users/allwin/Envs/fyp/lib/python2.7/site-packages/celery/bin/celery.py\", line 81 in main\r\n  File \"/Users/allwin/Envs/fyp/lib/python2.7/site-packages/celery/__main__.py\", line 30 in main\r\n  File \"/Users/allwin/Envs/fyp/bin/celery\", line 11 in <module>\r\n[2016-03-22 06:16:53,326: ERROR/MainProcess] Process 'Worker-2' pid:32104 exited with 'signal 11 (SIGSEGV)'\r\n[2016-03-22 06:16:53,350: ERROR/MainProcess] Task tweetsearch.analyzers.TweetDataAnalyzer[cdd5320d-f9eb-4392-af22-0c76037cad11] raised unexpected: WorkerLostError('Worker exited prematurely: signal 11 (SIGSEGV).',)\r\nTraceback (most recent call last):\r\n  File \"/Users/allwin/Envs/fyp/lib/python2.7/site-packages/billiard/pool.py\", line 1175, in mark_as_worker_lost\r\n    human_status(exitcode)),\r\nWorkerLostError: Worker exited prematurely: signal 11 (SIGSEGV).\r\n```\r\n\r\nOutput of `celery -A tweetsearch.analyzers report`\r\n\r\n```\r\nsoftware -> celery:3.1.23 (Cipater) kombu:3.0.34 py:2.7.8\r\n            billiard:3.3.0.23 redis:2.10.3\r\nplatform -> system:Darwin arch:64bit imp:CPython\r\nloader   -> celery.loaders.app.AppLoader\r\nsettings -> transport:redis results:redis://localhost/\r\n```\r\nOutput of `pip freeze` (using virtualenv)\r\n\r\n```\r\nDjango==1.8.4\r\namqp==1.4.9\r\nanyjson==0.3.3\r\nbilliard==3.3.0.23\r\ncelery==3.1.23\r\ncycler==0.9.0\r\ndjango-websocket-redis==0.4.4\r\nfaulthandler==2.4\r\ngevent==1.0.2\r\ngreenlet==0.4.9\r\njobtastic==0.3.1\r\nkombu==3.0.34\r\nmatplotlib==1.5.0\r\nmongoengine==0.10.0\r\nmysqlclient==1.3.7\r\nnltk==3.1\r\nnumpy==1.10.4\r\noauthlib==1.0.3\r\npandas==0.17.0\r\npsutil==3.2.1\r\npymongo==3.0.3\r\npyparsing==2.0.6\r\npython-dateutil==2.4.2\r\npytz==2016.1\r\nredis==2.10.3\r\nrequests==2.8.0\r\nrequests-oauthlib==0.6.0\r\nscikit-learn==0.17.1\r\nscipy==0.16.0\r\nsix==1.10.0\r\ntwython==3.3.0\r\nwsgiref==0.1.2\r\n```\r\nThank you all!\r\n"}}
{"actor":{"display_login":"ask"}}