{"action":"created","issue":{"url":"https://api.github.com/repos/Yelp/mrjob/issues/1736","repository_url":"https://api.github.com/repos/Yelp/mrjob","labels_url":"https://api.github.com/repos/Yelp/mrjob/issues/1736/labels{/name}","comments_url":"https://api.github.com/repos/Yelp/mrjob/issues/1736/comments","events_url":"https://api.github.com/repos/Yelp/mrjob/issues/1736/events","html_url":"https://github.com/Yelp/mrjob/issues/1736","id":300779912,"number":1736,"title":"`sys.stdin` , MRJob and Spark","user":{"login":"dfdeshom","id":1263,"avatar_url":"https://avatars3.githubusercontent.com/u/1263?v=4","gravatar_id":"","url":"https://api.github.com/users/dfdeshom","html_url":"https://github.com/dfdeshom","followers_url":"https://api.github.com/users/dfdeshom/followers","following_url":"https://api.github.com/users/dfdeshom/following{/other_user}","gists_url":"https://api.github.com/users/dfdeshom/gists{/gist_id}","starred_url":"https://api.github.com/users/dfdeshom/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dfdeshom/subscriptions","organizations_url":"https://api.github.com/users/dfdeshom/orgs","repos_url":"https://api.github.com/users/dfdeshom/repos","events_url":"https://api.github.com/users/dfdeshom/events{/privacy}","received_events_url":"https://api.github.com/users/dfdeshom/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-02-27T20:24:35Z","updated_at":"2018-02-27T21:21:39Z","closed_at":"2018-02-27T21:21:39Z","author_association":"NONE","body":"Some MRJob Spark jobs trigger `sys.stdin` to be pickled (`MRJob` holds a reference to it via `self.stdin`). This in turn triggers a failure (since file handles are not serializable)  which leads to a long stack trace and the message `pickle.PicklingError: Cannot pickle standard input` message from Spark's `cloudpickle` package. \r\n\r\nI don't really know why this issue is appears only on certain kinds of jobs and not others. On simple jobs, things run fine. But for more complicated jobs, I keep running into this error. Here is an example of a job that is consistently failing for me: https://gist.github.com/dfdeshom/3865731e3fe36b79c7a1f516ddef9cc1 . \r\n\r\nNote that the job runs fine if I set `self.stdin` to `None` before doing anything. "},"comment":{"url":"https://api.github.com/repos/Yelp/mrjob/issues/comments/369030366","html_url":"https://github.com/Yelp/mrjob/issues/1736#issuecomment-369030366","issue_url":"https://api.github.com/repos/Yelp/mrjob/issues/1736","id":369030366,"user":{"login":"davidmarin","id":50076,"avatar_url":"https://avatars1.githubusercontent.com/u/50076?v=4","gravatar_id":"","url":"https://api.github.com/users/davidmarin","html_url":"https://github.com/davidmarin","followers_url":"https://api.github.com/users/davidmarin/followers","following_url":"https://api.github.com/users/davidmarin/following{/other_user}","gists_url":"https://api.github.com/users/davidmarin/gists{/gist_id}","starred_url":"https://api.github.com/users/davidmarin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/davidmarin/subscriptions","organizations_url":"https://api.github.com/users/davidmarin/orgs","repos_url":"https://api.github.com/users/davidmarin/repos","events_url":"https://api.github.com/users/davidmarin/events{/privacy}","received_events_url":"https://api.github.com/users/davidmarin/received_events","type":"User","site_admin":false},"created_at":"2018-02-27T21:21:39Z","updated_at":"2018-02-27T21:21:39Z","author_association":"COLLABORATOR","body":"I'm guessing the problem is that you're making Spark pickle instance methods, which reference the job instance itself, which includes sys.stdin.\r\n\r\nIn your example, I think you'd be okay if you made `merge_candidate_resume()` a staticmethod too.\r\n"}}
{"actor":{"display_login":"davidmarin"}}