{"action":"created","issue":{"url":"https://api.github.com/repos/Yelp/mrjob/issues/236","repository_url":"https://api.github.com/repos/Yelp/mrjob","labels_url":"https://api.github.com/repos/Yelp/mrjob/issues/236/labels{/name}","comments_url":"https://api.github.com/repos/Yelp/mrjob/issues/236/comments","events_url":"https://api.github.com/repos/Yelp/mrjob/issues/236/events","html_url":"https://github.com/Yelp/mrjob/issues/236","id":2037630,"number":236,"title":"actually stream data from stdin","user":{"login":"davidmarin","id":50076,"avatar_url":"https://avatars.githubusercontent.com/u/50076?v=3","gravatar_id":"","url":"https://api.github.com/users/davidmarin","html_url":"https://github.com/davidmarin","followers_url":"https://api.github.com/users/davidmarin/followers","following_url":"https://api.github.com/users/davidmarin/following{/other_user}","gists_url":"https://api.github.com/users/davidmarin/gists{/gist_id}","starred_url":"https://api.github.com/users/davidmarin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/davidmarin/subscriptions","organizations_url":"https://api.github.com/users/davidmarin/orgs","repos_url":"https://api.github.com/users/davidmarin/repos","events_url":"https://api.github.com/users/davidmarin/events{/privacy}","received_events_url":"https://api.github.com/users/davidmarin/received_events","type":"User","site_admin":false},"labels":[{"id":52189,"url":"https://api.github.com/repos/Yelp/mrjob/labels/Feature","name":"Feature","color":"008500","default":false}],"state":"closed","locked":false,"assignee":{"login":"ssweens","id":1149151,"avatar_url":"https://avatars.githubusercontent.com/u/1149151?v=3","gravatar_id":"","url":"https://api.github.com/users/ssweens","html_url":"https://github.com/ssweens","followers_url":"https://api.github.com/users/ssweens/followers","following_url":"https://api.github.com/users/ssweens/following{/other_user}","gists_url":"https://api.github.com/users/ssweens/gists{/gist_id}","starred_url":"https://api.github.com/users/ssweens/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ssweens/subscriptions","organizations_url":"https://api.github.com/users/ssweens/orgs","repos_url":"https://api.github.com/users/ssweens/repos","events_url":"https://api.github.com/users/ssweens/events{/privacy}","received_events_url":"https://api.github.com/users/ssweens/received_events","type":"User","site_admin":false},"assignees":[{"login":"ssweens","id":1149151,"avatar_url":"https://avatars.githubusercontent.com/u/1149151?v=3","gravatar_id":"","url":"https://api.github.com/users/ssweens","html_url":"https://github.com/ssweens","followers_url":"https://api.github.com/users/ssweens/followers","following_url":"https://api.github.com/users/ssweens/following{/other_user}","gists_url":"https://api.github.com/users/ssweens/gists{/gist_id}","starred_url":"https://api.github.com/users/ssweens/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ssweens/subscriptions","organizations_url":"https://api.github.com/users/ssweens/orgs","repos_url":"https://api.github.com/users/ssweens/repos","events_url":"https://api.github.com/users/ssweens/events{/privacy}","received_events_url":"https://api.github.com/users/ssweens/received_events","type":"User","site_admin":false}],"milestone":null,"comments":2,"created_at":"2011-10-24T19:44:11Z","updated_at":"2016-12-28T01:20:23Z","closed_at":"2016-12-28T01:20:23Z","body":"`MRJob` current accepts data from stdin, but all the runners actually dump stdin to a file because it makes the code simpler.\n\nAt least one user (Scott Sweeney) wants to pipe a lot of data into `MRJob` (in `local` mode), so we should re-think this.\n\nWhat we can do:\n- **inline** mode can start streaming immediately\n- **local** mode can start streaming immediately (even if there are multiple mappers, we can round-robin)\n- **hadoop** mode can use `hadoop fs -put - ...` to stream your data into HDFS\n- **emr** mode can, in theory, stream data to S3\n\nInline mode and hadoop mode should be pretty straightforward. Local mode might be trickier but is do-able.\n\nStreaming to S3 is not well-supported by `boto` for files larger than about 5GB; you have to split them into pieces, and boto doesn't support streaming multipart uploads (it has no problem doing multipart uploads from files). The logical thing would be to add that feature to boto first, and then have mrjob use it. Also, if your md5 checksum fails, you're kind of out of luck because you don't have a copy of the data you just streamed.\n\nOne caveat is that it's valid to pass stdin several times to the same job (e.g. `mr_your_job.py - - -`), in which case we still need to dump it to a file. Kind of a pointless thing to support, so we could drop that in the next major release (v0.4).\n\nWe might also want to think about whether it's useful to support streams in a more general way (e.g. from unix sockets), and treat files as a \"special\" (though overwhelmingly common) case. We can use `os.path.isfile()` to check if something is actually a file.\n"},"comment":{"url":"https://api.github.com/repos/Yelp/mrjob/issues/comments/269408648","html_url":"https://github.com/Yelp/mrjob/issues/236#issuecomment-269408648","issue_url":"https://api.github.com/repos/Yelp/mrjob/issues/236","id":269408648,"user":{"login":"davidmarin","id":50076,"avatar_url":"https://avatars.githubusercontent.com/u/50076?v=3","gravatar_id":"","url":"https://api.github.com/users/davidmarin","html_url":"https://github.com/davidmarin","followers_url":"https://api.github.com/users/davidmarin/followers","following_url":"https://api.github.com/users/davidmarin/following{/other_user}","gists_url":"https://api.github.com/users/davidmarin/gists{/gist_id}","starred_url":"https://api.github.com/users/davidmarin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/davidmarin/subscriptions","organizations_url":"https://api.github.com/users/davidmarin/orgs","repos_url":"https://api.github.com/users/davidmarin/repos","events_url":"https://api.github.com/users/davidmarin/events{/privacy}","received_events_url":"https://api.github.com/users/davidmarin/received_events","type":"User","site_admin":false},"created_at":"2016-12-28T01:20:23Z","updated_at":"2016-12-28T01:20:23Z","body":"This seems not worth the trouble. Local mode is just for testing, and if your input is too big to want to dump to a file, you shouldn't be piping it."}}
{"actor":{"display_login":"davidmarin"}}