{"action":"created","issue":{"url":"https://api.github.com/repos/scrapinghub/crawlera-tools/issues/3","repository_url":"https://api.github.com/repos/scrapinghub/crawlera-tools","labels_url":"https://api.github.com/repos/scrapinghub/crawlera-tools/issues/3/labels{/name}","comments_url":"https://api.github.com/repos/scrapinghub/crawlera-tools/issues/3/comments","events_url":"https://api.github.com/repos/scrapinghub/crawlera-tools/issues/3/events","html_url":"https://github.com/scrapinghub/crawlera-tools/issues/3","id":41034085,"number":3,"title":"Possibly unclean threads termination","user":{"login":"qrilka","id":70110,"avatar_url":"https://avatars.githubusercontent.com/u/70110?v=3","gravatar_id":"","url":"https://api.github.com/users/qrilka","html_url":"https://github.com/qrilka","followers_url":"https://api.github.com/users/qrilka/followers","following_url":"https://api.github.com/users/qrilka/following{/other_user}","gists_url":"https://api.github.com/users/qrilka/gists{/gist_id}","starred_url":"https://api.github.com/users/qrilka/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/qrilka/subscriptions","organizations_url":"https://api.github.com/users/qrilka/orgs","repos_url":"https://api.github.com/users/qrilka/repos","events_url":"https://api.github.com/users/qrilka/events{/privacy}","received_events_url":"https://api.github.com/users/qrilka/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2014-08-25T05:48:05Z","updated_at":"2016-02-09T14:28:21Z","closed_at":null,"body":"Alain Quenneville seen the following exception (running python 2.7.3 on Linux Ubuntu 12.04.3 LTS):\r\n```\r\nException in thread Thread-1 (most likely raised during interpreter shutdown):\r\nTraceback (most recent call last):\r\n  File \"/usr/lib/python2.7/threading.py\", line 551, in __bootstrap_inner\r\n  File \"/usr/lib/python2.7/threading.py\", line 504, in run\r\n  File \"./crawlera-bench\", line 30, in worker_request\r\n  File \"/usr/lib/python2.7/Queue.py\", line 168, in get\r\n  File \"/usr/lib/python2.7/threading.py\", line 236, in wait\r\n<type 'exceptions.TypeError'>: 'NoneType' object is not callable\r\n```\r\nprobably it comes from the fact that worker threads are demonized"},"comment":{"url":"https://api.github.com/repos/scrapinghub/crawlera-tools/issues/comments/181887543","html_url":"https://github.com/scrapinghub/crawlera-tools/issues/3#issuecomment-181887543","issue_url":"https://api.github.com/repos/scrapinghub/crawlera-tools/issues/3","id":181887543,"user":{"login":"pawelmhm","id":2700942,"avatar_url":"https://avatars.githubusercontent.com/u/2700942?v=3","gravatar_id":"","url":"https://api.github.com/users/pawelmhm","html_url":"https://github.com/pawelmhm","followers_url":"https://api.github.com/users/pawelmhm/followers","following_url":"https://api.github.com/users/pawelmhm/following{/other_user}","gists_url":"https://api.github.com/users/pawelmhm/gists{/gist_id}","starred_url":"https://api.github.com/users/pawelmhm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pawelmhm/subscriptions","organizations_url":"https://api.github.com/users/pawelmhm/orgs","repos_url":"https://api.github.com/users/pawelmhm/repos","events_url":"https://api.github.com/users/pawelmhm/events{/privacy}","received_events_url":"https://api.github.com/users/pawelmhm/received_events","type":"User","site_admin":false},"created_at":"2016-02-09T14:28:21Z","updated_at":"2016-02-09T14:28:21Z","body":"one thing I noticed is that we exit using [os._exit(1)](https://github.com/scrapinghub/crawlera-tools/blob/master/crawlera-bench#L71)\r\n\r\n> os._exit(n)\r\nExit the process with status n, without calling cleanup handlers, flushing stdio buffers, etc.\r\n\r\n[docs](https://docs.python.org/2/library/os.html)\r\n\r\nI stumbled on some issues with bench when running crawlera benchmarks that take bit longer, seems like fab task using crawlera bench gets stuck. I think there are two problems here first of all exiting with 1 means that we're exiting with error on unix, this means that fab or any other script that relies on exit codes is getting confused. Secondly we're not doing any kind of cleanup of all those threads that are running, we just interrupt process in the middle. \r\n\r\nAs the code is currently written it seems difficult to refactor all this into better interruptions. Perhaps we could just migrate script to async library? e.g. gevents? It would give us same performance but without problems related to threads."}}