{"action":"created","issue":{"url":"https://api.github.com/repos/Yelp/mrjob/issues/1396","repository_url":"https://api.github.com/repos/Yelp/mrjob","labels_url":"https://api.github.com/repos/Yelp/mrjob/issues/1396/labels{/name}","comments_url":"https://api.github.com/repos/Yelp/mrjob/issues/1396/comments","events_url":"https://api.github.com/repos/Yelp/mrjob/issues/1396/events","html_url":"https://github.com/Yelp/mrjob/pull/1396","id":170435150,"number":1396,"title":"Fix job crash during cleanup on local hadoop cluster","user":{"login":"etiennebatise","id":5644956,"avatar_url":"https://avatars.githubusercontent.com/u/5644956?v=3","gravatar_id":"","url":"https://api.github.com/users/etiennebatise","html_url":"https://github.com/etiennebatise","followers_url":"https://api.github.com/users/etiennebatise/followers","following_url":"https://api.github.com/users/etiennebatise/following{/other_user}","gists_url":"https://api.github.com/users/etiennebatise/gists{/gist_id}","starred_url":"https://api.github.com/users/etiennebatise/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/etiennebatise/subscriptions","organizations_url":"https://api.github.com/users/etiennebatise/orgs","repos_url":"https://api.github.com/users/etiennebatise/repos","events_url":"https://api.github.com/users/etiennebatise/events{/privacy}","received_events_url":"https://api.github.com/users/etiennebatise/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2016-08-10T14:44:31Z","updated_at":"2016-08-14T21:41:13Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/Yelp/mrjob/pulls/1396","html_url":"https://github.com/Yelp/mrjob/pull/1396","diff_url":"https://github.com/Yelp/mrjob/pull/1396.diff","patch_url":"https://github.com/Yelp/mrjob/pull/1396.patch"},"body":"When running a word count on local Hadoop cluster, program crashed during cleanup step causing this stack trace:\r\n```\r\nTraceback (most recent call last):\r\n  File \"word_count.py\", line 23, in <module>\r\n    MRWordFreqCount.run()\r\n  File \"/Users/etiennebatise/.virtualenvs/ENV-MRJ/lib/python3.5/site-packages/mrjob/job.py\", line 430, in run\r\n    mr_job.execute()\r\n  File \"/Users/etiennebatise/.virtualenvs/ENV-MRJ/lib/python3.5/site-packages/mrjob/job.py\", line 448, in execute\r\n    super(MRJob, self).execute()\r\n  File \"/Users/etiennebatise/.virtualenvs/ENV-MRJ/lib/python3.5/site-packages/mrjob/launch.py\", line 160, in execute\r\n    self.run_job()\r\n  File \"/Users/etiennebatise/.virtualenvs/ENV-MRJ/lib/python3.5/site-packages/mrjob/launch.py\", line 238, in run_job\r\n    for line in runner.stream_output():\r\n  File \"/Users/etiennebatise/.virtualenvs/ENV-MRJ/lib/python3.5/site-packages/mrjob/runner.py\", line 507, in stream_output\r\n    for line in self.fs._cat_file(filename):\r\n  File \"/Users/etiennebatise/.virtualenvs/ENV-MRJ/lib/python3.5/site-packages/mrjob/fs/composite.py\", line 72, in _cat_file\r\n    for line in self._do_action('_cat_file', path):\r\n  File \"/Users/etiennebatise/.virtualenvs/ENV-MRJ/lib/python3.5/site-packages/mrjob/util.py\", line 416, in read_file\r\n    cleanup()\r\n  File \"/Users/etiennebatise/.virtualenvs/ENV-MRJ/lib/python3.5/site-packages/mrjob/fs/hadoop.py\", line 270, in cleanup\r\n    log.error('STDERR: ' + line)\r\nTypeError: Can't convert 'bytes' object to str implicitly\r\n```\r\n\r\nLooking rapidly into the test case for this function, I realized this statement was never called, and coverage confirmed it: \r\n<img width=\"383\" alt=\"screen shot 2016-08-10 at 16 41 23\" src=\"https://cloud.githubusercontent.com/assets/5644956/17557979/769689f4-5f19-11e6-9a57-53d9fc793b34.png\">\r\n\r\nI haven't figured out what causes the error in my case.\r\n\r\nOSX El Capitan, Python3.5, Hadoop 2.7.2"},"comment":{"url":"https://api.github.com/repos/Yelp/mrjob/issues/comments/239699527","html_url":"https://github.com/Yelp/mrjob/pull/1396#issuecomment-239699527","issue_url":"https://api.github.com/repos/Yelp/mrjob/issues/1396","id":239699527,"user":{"login":"davidmarin","id":50076,"avatar_url":"https://avatars.githubusercontent.com/u/50076?v=3","gravatar_id":"","url":"https://api.github.com/users/davidmarin","html_url":"https://github.com/davidmarin","followers_url":"https://api.github.com/users/davidmarin/followers","following_url":"https://api.github.com/users/davidmarin/following{/other_user}","gists_url":"https://api.github.com/users/davidmarin/gists{/gist_id}","starred_url":"https://api.github.com/users/davidmarin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/davidmarin/subscriptions","organizations_url":"https://api.github.com/users/davidmarin/orgs","repos_url":"https://api.github.com/users/davidmarin/repos","events_url":"https://api.github.com/users/davidmarin/events{/privacy}","received_events_url":"https://api.github.com/users/davidmarin/received_events","type":"User","site_admin":false},"created_at":"2016-08-14T21:41:13Z","updated_at":"2016-08-14T21:41:13Z","body":"Huh, I wonder how I missed that one! We should actually be using `mrjob.py2.to_string()`, in case the line isn't UTF-8-decodable, will fix that when I merge.\r\n\r\nThanks!"}}
{"actor":{"display_login":"davidmarin"}}