{"action":"created","issue":{"url":"https://api.github.com/repos/dmwm/WMArchive/issues/58","labels_url":"https://api.github.com/repos/dmwm/WMArchive/issues/58/labels{/name}","comments_url":"https://api.github.com/repos/dmwm/WMArchive/issues/58/comments","events_url":"https://api.github.com/repos/dmwm/WMArchive/issues/58/events","html_url":"https://github.com/dmwm/WMArchive/issues/58","id":126222255,"number":58,"title":"Write MR implementation for common data-ops use cases","user":{"login":"vkuznet","id":125138,"avatar_url":"https://avatars.githubusercontent.com/u/125138?v=3","gravatar_id":"","url":"https://api.github.com/users/vkuznet","html_url":"https://github.com/vkuznet","followers_url":"https://api.github.com/users/vkuznet/followers","following_url":"https://api.github.com/users/vkuznet/following{/other_user}","gists_url":"https://api.github.com/users/vkuznet/gists{/gist_id}","starred_url":"https://api.github.com/users/vkuznet/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vkuznet/subscriptions","organizations_url":"https://api.github.com/users/vkuznet/orgs","repos_url":"https://api.github.com/users/vkuznet/repos","events_url":"https://api.github.com/users/vkuznet/events{/privacy}","received_events_url":"https://api.github.com/users/vkuznet/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"stiegerb","id":4990806,"avatar_url":"https://avatars.githubusercontent.com/u/4990806?v=3","gravatar_id":"","url":"https://api.github.com/users/stiegerb","html_url":"https://github.com/stiegerb","followers_url":"https://api.github.com/users/stiegerb/followers","following_url":"https://api.github.com/users/stiegerb/following{/other_user}","gists_url":"https://api.github.com/users/stiegerb/gists{/gist_id}","starred_url":"https://api.github.com/users/stiegerb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/stiegerb/subscriptions","organizations_url":"https://api.github.com/users/stiegerb/orgs","repos_url":"https://api.github.com/users/stiegerb/repos","events_url":"https://api.github.com/users/stiegerb/events{/privacy}","received_events_url":"https://api.github.com/users/stiegerb/received_events","type":"User","site_admin":false},"milestone":null,"comments":3,"created_at":"2016-01-12T16:50:41Z","updated_at":"2016-01-21T13:54:44Z","closed_at":null,"body":"Based on #7 ticket now we can write MR mapper/reducer implementation for MR jobs. Assign this to Benjamin."},"comment":{"url":"https://api.github.com/repos/dmwm/WMArchive/issues/comments/173575818","html_url":"https://github.com/dmwm/WMArchive/issues/58#issuecomment-173575818","issue_url":"https://api.github.com/repos/dmwm/WMArchive/issues/58","id":173575818,"user":{"login":"stiegerb","id":4990806,"avatar_url":"https://avatars.githubusercontent.com/u/4990806?v=3","gravatar_id":"","url":"https://api.github.com/users/stiegerb","html_url":"https://github.com/stiegerb","followers_url":"https://api.github.com/users/stiegerb/followers","following_url":"https://api.github.com/users/stiegerb/following{/other_user}","gists_url":"https://api.github.com/users/stiegerb/gists{/gist_id}","starred_url":"https://api.github.com/users/stiegerb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/stiegerb/subscriptions","organizations_url":"https://api.github.com/users/stiegerb/orgs","repos_url":"https://api.github.com/users/stiegerb/repos","events_url":"https://api.github.com/users/stiegerb/events{/privacy}","received_events_url":"https://api.github.com/users/stiegerb/received_events","type":"User","site_admin":false},"created_at":"2016-01-21T13:54:44Z","updated_at":"2016-01-21T13:54:44Z","body":"Here's the help from the mrjob.py script which is up to date:\r\n\r\n```\r\n[vocms013][14:52:13] ~/work/wma > mrjob -h\r\nusage: mrjob [-h] [--hdir HDIR] [--idir IDIR] [--odir ODIR] [--schema SCHEMA]\r\n             [--mrpy MRPY] [--pydoop PYDOOP] [--avro AVRO] [--execute]\r\n             [--verbose] [--loglevel LOGLEVEL] [--hdfs_prefix HDFS_PREFIX]\r\n\r\nTool to generate MapReduce (MR) scripts and launch jobs. The code is generated\r\nfrom a skeleton provided by WMArchive and a user based MR file. The latter\r\nmust contain definitions for two functions: mapper(ctx) and reducer(ctx) for\r\ngiven context. A trivial example can be found in: /afs/cern.ch/work/s/stiegerb\r\n/wma/WMArchive/src/python/WMArchive/MapReduce/mruser.py Based on this code\r\nplease create your own mapper/reducer functions and use this tool to generate\r\nthe final MR script.\r\n\r\noptional arguments:\r\n  -h, --help            show this help message and exit\r\n  --hdir HDIR           hdfs input data directory,\r\n                        default=/user/stiegerb/test/\r\n  --idir IDIR           hdfs input directory for MR jobs, inside hdir,\r\n                        default=data/\r\n  --odir ODIR           hdfs output directory for MR jobs, inside hdir,\r\n                        default=mrout/\r\n  --schema SCHEMA       Name of data schema file on hdfs, inside hdir,\r\n                        default=schema.avsc\r\n  --mrpy MRPY           MapReduce python script\r\n  --pydoop PYDOOP       pydoop archive file, e.g. /path/pydoop.tgz\r\n  --avro AVRO           avro archive file, e.g. /path/avro.tgz\r\n  --execute             Execute the generated mr job script (i.e. launch the\r\n                        job)\r\n  --verbose             Verbose output\r\n  --loglevel LOGLEVEL   pydoop log level, default=INFO\r\n  --hdfs_prefix HDFS_PREFIX\r\n                        hdfs prefix,\r\n                        default=hdfs://p01001532965510.cern.ch:9000\r\n[vocms013][14:52:16] ~/work/wma > \r\n```\r\n\r\nNote that by default the script picks up your username and runs everything from the `/user/username/test/` directory.\r\n\r\nAn example for running a job would be:\r\n```\r\nmrjob --odir output --idir data3 --mrpy WMArchive/src/python/WMArchive/MapReduce/mr_avgeventcpu_by_task.py --pydoop pydoop.tgz --avro avro.tgz --verbose --execute --schema fwjr.avsc\r\n```\r\n\r\nNote that we didn't provide the pydoop.tgz and avro.tgz archives, as far as I know. We should find out whether there's a possibility to make it work without them.\r\n\r\nI will add some example logs later."}}