{"action":"created","issue":{"url":"https://api.github.com/repos/Yelp/mrjob/issues/1324","repository_url":"https://api.github.com/repos/Yelp/mrjob","labels_url":"https://api.github.com/repos/Yelp/mrjob/issues/1324/labels{/name}","comments_url":"https://api.github.com/repos/Yelp/mrjob/issues/1324/comments","events_url":"https://api.github.com/repos/Yelp/mrjob/issues/1324/events","html_url":"https://github.com/Yelp/mrjob/issues/1324","id":158636172,"number":1324,"title":"JSONDecodeError while running job on EMR","user":{"login":"mpenkov","id":1101396,"avatar_url":"https://avatars.githubusercontent.com/u/1101396?v=3","gravatar_id":"","url":"https://api.github.com/users/mpenkov","html_url":"https://github.com/mpenkov","followers_url":"https://api.github.com/users/mpenkov/followers","following_url":"https://api.github.com/users/mpenkov/following{/other_user}","gists_url":"https://api.github.com/users/mpenkov/gists{/gist_id}","starred_url":"https://api.github.com/users/mpenkov/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mpenkov/subscriptions","organizations_url":"https://api.github.com/users/mpenkov/orgs","repos_url":"https://api.github.com/users/mpenkov/repos","events_url":"https://api.github.com/users/mpenkov/events{/privacy}","received_events_url":"https://api.github.com/users/mpenkov/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/Yelp/mrjob/labels/Bug","name":"Bug","color":"d95757"}],"state":"open","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/Yelp/mrjob/milestones/26","html_url":"https://github.com/Yelp/mrjob/milestones/v0.5.3","labels_url":"https://api.github.com/repos/Yelp/mrjob/milestones/26/labels","id":1779871,"number":26,"title":"v0.5.3","description":"","creator":{"login":"davidmarin","id":50076,"avatar_url":"https://avatars.githubusercontent.com/u/50076?v=3","gravatar_id":"","url":"https://api.github.com/users/davidmarin","html_url":"https://github.com/davidmarin","followers_url":"https://api.github.com/users/davidmarin/followers","following_url":"https://api.github.com/users/davidmarin/following{/other_user}","gists_url":"https://api.github.com/users/davidmarin/gists{/gist_id}","starred_url":"https://api.github.com/users/davidmarin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/davidmarin/subscriptions","organizations_url":"https://api.github.com/users/davidmarin/orgs","repos_url":"https://api.github.com/users/davidmarin/repos","events_url":"https://api.github.com/users/davidmarin/events{/privacy}","received_events_url":"https://api.github.com/users/davidmarin/received_events","type":"User","site_admin":false},"open_issues":14,"closed_issues":2,"state":"open","created_at":"2016-05-20T23:25:03Z","updated_at":"2016-06-07T21:29:17Z","due_on":null,"closed_at":null},"comments":5,"created_at":"2016-06-06T09:30:21Z","updated_at":"2016-06-09T16:34:07Z","closed_at":null,"body":"I'm attempting to run a job on EMR. I'm seeing my job fail due to raised JSONDecodeError in underlying code.\r\n\r\nThere are several places where this exception is raised. For example, it happens while the combiner/reducer are reading their input:\r\n\r\n```\r\nERROR:root:Expecting value: line 1 column 1 (char 0)\r\nTraceback (most recent call last):\r\n  File \"mr_stage_a.py\", line 177, in combiner\r\n    for (score, output_dict) in sorted(tuples):\r\n  File \"/usr/lib/python2.7/site-packages/mrjob/job.py\", line 593, in <genexpr>\r\n    values = (v for k, v in kv_pairs)\r\n  File \"/usr/lib/python2.7/site-packages/mrjob/job.py\", line 672, in read_lines\r\n    key, value = read(line.rstrip(b'\\r\\n'))\r\n  File \"/usr/lib/python2.7/site-packages/mrjob/protocol.py\", line 88, in read\r\n    self._last_key_decoded = self._loads(raw_key)\r\n  File \"/usr/lib/python2.7/site-packages/mrjob/protocol.py\", line 158, in _loads\r\n    return simplejson.loads(value)\r\n  File \"/usr/lib64/python2.7/site-packages/simplejson/__init__.py\", line 516, in loads\r\n    return _default_decoder.decode(s)\r\n  File \"/usr/lib64/python2.7/site-packages/simplejson/decoder.py\", line 370, in decode\r\n    obj, end = self.raw_decode(s)\r\n  File \"/usr/lib64/python2.7/site-packages/simplejson/decoder.py\", line 400, in raw_decode\r\n    return self.scan_once(s, idx=_w(s, idx).end())\r\nJSONDecodeError: Expecting value: line 1 column 1 (char 0)\r\n```\r\n\r\nIn this case, mr_stage_a.py is my code, so I can catch the exception with something like:\r\n\r\n```\r\n    def combiner(self, domain, tuples):\r\n        try:\r\n            for (score, output_dict) in sorted(tuples):\r\n                # do stuff here\r\n        except ValueError as e:\r\n            #\r\n            # https://github.com/Yelp/mrjob/issues/544\r\n            #\r\n            if e.message.startswith(\"Expecting value: \"):\r\n                logging.error(\"combiner: unable to decode JSON\")\r\n                logging.exception(e)\r\n            else:\r\n                raise\r\n```\r\n\r\nHowever, the same exception is also raised by internal mrjob code:\r\n\r\n```\r\nJSONDecodeError: Expecting value: line 1 column 1 (char 0)\r\nTraceback (most recent call last):\r\n  File \"mr_stage_a.py\", line 295, in <module>\r\n    MrStageA.run()\r\n  File \"/usr/lib/python2.7/site-packages/mrjob/job.py\", line 430, in run\r\n    mr_job.execute()\r\n  File \"/usr/lib/python2.7/site-packages/mrjob/job.py\", line 442, in execute\r\n    self.run_combiner(self.options.step_num)\r\n  File \"/usr/lib/python2.7/site-packages/mrjob/job.py\", line 592, in run_combiner\r\n    key=lambda k_v1: k_v1[0]):\r\n  File \"/usr/lib/python2.7/site-packages/mrjob/job.py\", line 672, in read_lines\r\n    key, value = read(line.rstrip(b'\\r\\n'))\r\n  File \"/usr/lib/python2.7/site-packages/mrjob/protocol.py\", line 88, in read\r\n    self._last_key_decoded = self._loads(raw_key)\r\n  File \"/usr/lib/python2.7/site-packages/mrjob/protocol.py\", line 158, in _loads\r\n    return simplejson.loads(value)\r\n  File \"/usr/lib64/python2.7/site-packages/simplejson/__init__.py\", line 516, in loads\r\n    return _default_decoder.decode(s)\r\n  File \"/usr/lib64/python2.7/site-packages/simplejson/decoder.py\", line 370, in decode\r\n    obj, end = self.raw_decode(s)\r\n  File \"/usr/lib64/python2.7/site-packages/simplejson/decoder.py\", line 400, in raw_decode\r\n    return self.scan_once(s, idx=_w(s, idx).end())\r\nsimplejson.scanner.JSONDecodeError: Expecting value: line 1 column 5 (char 4)\r\n```\r\n\r\nIn this case, I can't do anything to catch the error, so it crashes the entire job.\r\n\r\nWhat's going on and how can I avoid these errors?"},"comment":{"url":"https://api.github.com/repos/Yelp/mrjob/issues/comments/224951588","html_url":"https://github.com/Yelp/mrjob/issues/1324#issuecomment-224951588","issue_url":"https://api.github.com/repos/Yelp/mrjob/issues/1324","id":224951588,"user":{"login":"mpenkov","id":1101396,"avatar_url":"https://avatars.githubusercontent.com/u/1101396?v=3","gravatar_id":"","url":"https://api.github.com/users/mpenkov","html_url":"https://github.com/mpenkov","followers_url":"https://api.github.com/users/mpenkov/followers","following_url":"https://api.github.com/users/mpenkov/following{/other_user}","gists_url":"https://api.github.com/users/mpenkov/gists{/gist_id}","starred_url":"https://api.github.com/users/mpenkov/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mpenkov/subscriptions","organizations_url":"https://api.github.com/users/mpenkov/orgs","repos_url":"https://api.github.com/users/mpenkov/repos","events_url":"https://api.github.com/users/mpenkov/events{/privacy}","received_events_url":"https://api.github.com/users/mpenkov/received_events","type":"User","site_admin":false},"created_at":"2016-06-09T16:34:07Z","updated_at":"2016-06-09T16:34:07Z","body":"OK, I've packaged together a job and shared it as a [gist](https://gist.github.com/mpenkov/401f62731709cd05ca3de860d8ee7d17).\r\n\r\nTo run:\r\n\r\n    python mr_stage_a.py test-1.warc -r emr --conf-path mrjob.conf --output-dir s3://your-bucket/test-1.out --no-output\r\n\r\nGrab the results:\r\n\r\n    aws s3 sync s3://your-bucket/test-1.out test-1.out\r\n\r\nCheck:\r\n\r\n    python check.py < test-1.out/part-00000\r\n\r\nYou should get several badnesses reported, on examination, it seems that they're all log statements (from Python's builtin logging module) that have *somehow* made their way into the output. \r\n\r\nDoes that seem right to you?\r\n\r\nI think the problem is this line in my script:\r\n\r\n    logger.addHandler(logging.StreamHandler(sys.stdout))\r\n\r\nI've changed it to:\r\n\r\n    logger.addHandler(logging.StreamHandler(sys.stderr))\r\n\r\nand retried. The badnesses from the output are gone.\r\n\r\nShould streaming logging to stdout cause this kind of problem?"}}
{"actor":{"display_login":"mpenkov"}}