{"action":"created","comment":{"url":"https://api.github.com/repos/allenai/amti/pulls/comments/232795076","pull_request_review_id":174072498,"id":232795076,"node_id":"MDI0OlB1bGxSZXF1ZXN0UmV2aWV3Q29tbWVudDIzMjc5NTA3Ng==","diff_hunk":"@@ -4,29 +4,35 @@\n \n import click\n \n-from amti import actions\n+from amti import clis\n \n \n logger = logging.getLogger(__name__)\n \n \n-@click.command(\n+@click.group(\n     context_settings={\n         'help_option_names': ['--help', '-h']\n     })\n-@click.argument(\n-    'batch_dir',\n-    type=click.Path(exists=True, file_okay=False, dir_okay=True))\n-@click.argument(\n-    'output_dir',\n-    type=click.Path(exists=True, file_okay=False, dir_okay=True))\n-def extract_xml(batch_dir, output_dir):\n-    \"\"\"Extract XML data from assignments in BATCH_DIR to OUTPUT_DIR.\n-\n-    Given a directory (BATCH_DIR) that represents a batch of HITs that\n-    have been reviewed and saved, extract the XML data from the\n-    assignments to OUTPUT_DIR.\n+def extract():\n+    \"\"\"Extract data from a batch to various formats.\n+\n+    See the subcommands for extracting batch data into a specific\n+    format.\n     \"\"\"\n-    actions.extract_xml(batch_dir=batch_dir, output_dir=output_dir)\n+    pass\n+\n+\n+subcommands = [","path":"amti/clis/extract.py","position":38,"original_position":38,"commit_id":"e04235b5b1a8589abc18f78cd9ba3053ecbb332d","original_commit_id":"e04235b5b1a8589abc18f78cd9ba3053ecbb332d","user":{"login":"nalourie-ai2","id":23320271,"node_id":"MDQ6VXNlcjIzMzIwMjcx","avatar_url":"https://avatars2.githubusercontent.com/u/23320271?v=4","gravatar_id":"","url":"https://api.github.com/users/nalourie-ai2","html_url":"https://github.com/nalourie-ai2","followers_url":"https://api.github.com/users/nalourie-ai2/followers","following_url":"https://api.github.com/users/nalourie-ai2/following{/other_user}","gists_url":"https://api.github.com/users/nalourie-ai2/gists{/gist_id}","starred_url":"https://api.github.com/users/nalourie-ai2/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nalourie-ai2/subscriptions","organizations_url":"https://api.github.com/users/nalourie-ai2/orgs","repos_url":"https://api.github.com/users/nalourie-ai2/repos","events_url":"https://api.github.com/users/nalourie-ai2/events{/privacy}","received_events_url":"https://api.github.com/users/nalourie-ai2/received_events","type":"User","site_admin":false},"body":"My thinking was commands should map onto the structure of the extracted data, while a flag can set the serialization format.","created_at":"2018-11-12T20:06:10Z","updated_at":"2018-11-12T20:06:10Z","html_url":"https://github.com/allenai/amti/pull/5#discussion_r232795076","pull_request_url":"https://api.github.com/repos/allenai/amti/pulls/5","author_association":"COLLABORATOR","_links":{"self":{"href":"https://api.github.com/repos/allenai/amti/pulls/comments/232795076"},"html":{"href":"https://github.com/allenai/amti/pull/5#discussion_r232795076"},"pull_request":{"href":"https://api.github.com/repos/allenai/amti/pulls/5"}},"in_reply_to_id":232542049},"pull_request":{"url":"https://api.github.com/repos/allenai/amti/pulls/5","id":229292692,"node_id":"MDExOlB1bGxSZXF1ZXN0MjI5MjkyNjky","html_url":"https://github.com/allenai/amti/pull/5","diff_url":"https://github.com/allenai/amti/pull/5.diff","patch_url":"https://github.com/allenai/amti/pull/5.patch","issue_url":"https://api.github.com/repos/allenai/amti/issues/5","number":5,"state":"open","locked":false,"title":"Add extract tabular","user":{"login":"nalourie-ai2","id":23320271,"node_id":"MDQ6VXNlcjIzMzIwMjcx","avatar_url":"https://avatars2.githubusercontent.com/u/23320271?v=4","gravatar_id":"","url":"https://api.github.com/users/nalourie-ai2","html_url":"https://github.com/nalourie-ai2","followers_url":"https://api.github.com/users/nalourie-ai2/followers","following_url":"https://api.github.com/users/nalourie-ai2/following{/other_user}","gists_url":"https://api.github.com/users/nalourie-ai2/gists{/gist_id}","starred_url":"https://api.github.com/users/nalourie-ai2/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nalourie-ai2/subscriptions","organizations_url":"https://api.github.com/users/nalourie-ai2/orgs","repos_url":"https://api.github.com/users/nalourie-ai2/repos","events_url":"https://api.github.com/users/nalourie-ai2/events{/privacy}","received_events_url":"https://api.github.com/users/nalourie-ai2/received_events","type":"User","site_admin":false},"body":"This pull request adds a command to extract all the batch data in a tabular format.\r\n\r\nThe new command is:\r\n\r\n```\r\n$ amti extract tabular --help\r\nUsage: amti extract tabular [OPTIONS] BATCH_DIR OUTPUT_PATH\r\n\r\n  Extract data from BATCH_DIR to OUTPUT_PATH in a tabular format.\r\n\r\n  Given a directory (BATCH_DIR) that represents a batch of HITs that have\r\n  been reviewed and saved, extract the data to OUTPUT_PATH in a tabular\r\n  format. Every row of the table is an assignment, where each form field has\r\n  a column and also there are additional columns for assignment metadata. By\r\n  default, the table will be saved as JSON Lines, but other formats may be\r\n  specified with the --format option.\r\n\r\nOptions:\r\n  -f, --format [csv|json|jsonl]  The desired output file format.\r\n  -h, --help                     Show this message and exit.\r\n```\r\n\r\nThe command should be helpful in pulling out all the batch data into a form that's easier to work with than the directory tree of XML files we used previously.\r\n\r\nAdditionally, this pull request changes the command line interface by taking the old `extract_xml` command and grouping it with the new `tabular` command as `xml` under the `extract` command group.","created_at":"2018-11-08T09:04:43Z","updated_at":"2018-11-12T20:06:10Z","closed_at":null,"merged_at":null,"merge_commit_sha":"42d4929fede7a0a57a3a7e1507898b384000ecec","assignee":null,"assignees":[],"requested_reviewers":[{"login":"keisks","id":694378,"node_id":"MDQ6VXNlcjY5NDM3OA==","avatar_url":"https://avatars1.githubusercontent.com/u/694378?v=4","gravatar_id":"","url":"https://api.github.com/users/keisks","html_url":"https://github.com/keisks","followers_url":"https://api.github.com/users/keisks/followers","following_url":"https://api.github.com/users/keisks/following{/other_user}","gists_url":"https://api.github.com/users/keisks/gists{/gist_id}","starred_url":"https://api.github.com/users/keisks/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/keisks/subscriptions","organizations_url":"https://api.github.com/users/keisks/orgs","repos_url":"https://api.github.com/users/keisks/repos","events_url":"https://api.github.com/users/keisks/events{/privacy}","received_events_url":"https://api.github.com/users/keisks/received_events","type":"User","site_admin":false}],"requested_teams":[],"labels":[],"milestone":null,"commits_url":"https://api.github.com/repos/allenai/amti/pulls/5/commits","review_comments_url":"https://api.github.com/repos/allenai/amti/pulls/5/comments","review_comment_url":"https://api.github.com/repos/allenai/amti/pulls/comments{/number}","comments_url":"https://api.github.com/repos/allenai/amti/issues/5/comments","statuses_url":"https://api.github.com/repos/allenai/amti/statuses/e04235b5b1a8589abc18f78cd9ba3053ecbb332d","head":{"label":"nalourie-ai2:add-extract-json","ref":"add-extract-json","sha":"e04235b5b1a8589abc18f78cd9ba3053ecbb332d","user":{"login":"nalourie-ai2","id":23320271,"node_id":"MDQ6VXNlcjIzMzIwMjcx","avatar_url":"https://avatars2.githubusercontent.com/u/23320271?v=4","gravatar_id":"","url":"https://api.github.com/users/nalourie-ai2","html_url":"https://github.com/nalourie-ai2","followers_url":"https://api.github.com/users/nalourie-ai2/followers","following_url":"https://api.github.com/users/nalourie-ai2/following{/other_user}","gists_url":"https://api.github.com/users/nalourie-ai2/gists{/gist_id}","starred_url":"https://api.github.com/users/nalourie-ai2/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nalourie-ai2/subscriptions","organizations_url":"https://api.github.com/users/nalourie-ai2/orgs","repos_url":"https://api.github.com/users/nalourie-ai2/repos","events_url":"https://api.github.com/users/nalourie-ai2/events{/privacy}","received_events_url":"https://api.github.com/users/nalourie-ai2/received_events","type":"User","site_admin":false},"repo":{"id":156651361,"node_id":"MDEwOlJlcG9zaXRvcnkxNTY2NTEzNjE=","name":"amti","full_name":"nalourie-ai2/amti","private":false,"owner":{"login":"nalourie-ai2","id":23320271,"node_id":"MDQ6VXNlcjIzMzIwMjcx","avatar_url":"https://avatars2.githubusercontent.com/u/23320271?v=4","gravatar_id":"","url":"https://api.github.com/users/nalourie-ai2","html_url":"https://github.com/nalourie-ai2","followers_url":"https://api.github.com/users/nalourie-ai2/followers","following_url":"https://api.github.com/users/nalourie-ai2/following{/other_user}","gists_url":"https://api.github.com/users/nalourie-ai2/gists{/gist_id}","starred_url":"https://api.github.com/users/nalourie-ai2/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nalourie-ai2/subscriptions","organizations_url":"https://api.github.com/users/nalourie-ai2/orgs","repos_url":"https://api.github.com/users/nalourie-ai2/repos","events_url":"https://api.github.com/users/nalourie-ai2/events{/privacy}","received_events_url":"https://api.github.com/users/nalourie-ai2/received_events","type":"User","site_admin":false},"html_url":"https://github.com/nalourie-ai2/amti","description":"Alexandria's Mechanical Turk Interface (amti) 🤖","fork":true,"url":"https://api.github.com/repos/nalourie-ai2/amti","forks_url":"https://api.github.com/repos/nalourie-ai2/amti/forks","keys_url":"https://api.github.com/repos/nalourie-ai2/amti/keys{/key_id}","collaborators_url":"https://api.github.com/repos/nalourie-ai2/amti/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/nalourie-ai2/amti/teams","hooks_url":"https://api.github.com/repos/nalourie-ai2/amti/hooks","issue_events_url":"https://api.github.com/repos/nalourie-ai2/amti/issues/events{/number}","events_url":"https://api.github.com/repos/nalourie-ai2/amti/events","assignees_url":"https://api.github.com/repos/nalourie-ai2/amti/assignees{/user}","branches_url":"https://api.github.com/repos/nalourie-ai2/amti/branches{/branch}","tags_url":"https://api.github.com/repos/nalourie-ai2/amti/tags","blobs_url":"https://api.github.com/repos/nalourie-ai2/amti/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/nalourie-ai2/amti/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/nalourie-ai2/amti/git/refs{/sha}","trees_url":"https://api.github.com/repos/nalourie-ai2/amti/git/trees{/sha}","statuses_url":"https://api.github.com/repos/nalourie-ai2/amti/statuses/{sha}","languages_url":"https://api.github.com/repos/nalourie-ai2/amti/languages","stargazers_url":"https://api.github.com/repos/nalourie-ai2/amti/stargazers","contributors_url":"https://api.github.com/repos/nalourie-ai2/amti/contributors","subscribers_url":"https://api.github.com/repos/nalourie-ai2/amti/subscribers","subscription_url":"https://api.github.com/repos/nalourie-ai2/amti/subscription","commits_url":"https://api.github.com/repos/nalourie-ai2/amti/commits{/sha}","git_commits_url":"https://api.github.com/repos/nalourie-ai2/amti/git/commits{/sha}","comments_url":"https://api.github.com/repos/nalourie-ai2/amti/comments{/number}","issue_comment_url":"https://api.github.com/repos/nalourie-ai2/amti/issues/comments{/number}","contents_url":"https://api.github.com/repos/nalourie-ai2/amti/contents/{+path}","compare_url":"https://api.github.com/repos/nalourie-ai2/amti/compare/{base}...{head}","merges_url":"https://api.github.com/repos/nalourie-ai2/amti/merges","archive_url":"https://api.github.com/repos/nalourie-ai2/amti/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/nalourie-ai2/amti/downloads","issues_url":"https://api.github.com/repos/nalourie-ai2/amti/issues{/number}","pulls_url":"https://api.github.com/repos/nalourie-ai2/amti/pulls{/number}","milestones_url":"https://api.github.com/repos/nalourie-ai2/amti/milestones{/number}","notifications_url":"https://api.github.com/repos/nalourie-ai2/amti/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/nalourie-ai2/amti/labels{/name}","releases_url":"https://api.github.com/repos/nalourie-ai2/amti/releases{/id}","deployments_url":"https://api.github.com/repos/nalourie-ai2/amti/deployments","created_at":"2018-11-08T04:45:00Z","updated_at":"2018-11-08T04:45:02Z","pushed_at":"2018-11-09T08:47:17Z","git_url":"git://github.com/nalourie-ai2/amti.git","ssh_url":"git@github.com:nalourie-ai2/amti.git","clone_url":"https://github.com/nalourie-ai2/amti.git","svn_url":"https://github.com/nalourie-ai2/amti","homepage":null,"size":58,"stargazers_count":0,"watchers_count":0,"language":"Python","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"archived":false,"open_issues_count":0,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0","node_id":"MDc6TGljZW5zZTI="},"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"allenai:master","ref":"master","sha":"fa137e9ac8890dc08bdd759925bfb8a4e2d507d6","user":{"login":"allenai","id":5667695,"node_id":"MDEyOk9yZ2FuaXphdGlvbjU2Njc2OTU=","avatar_url":"https://avatars3.githubusercontent.com/u/5667695?v=4","gravatar_id":"","url":"https://api.github.com/users/allenai","html_url":"https://github.com/allenai","followers_url":"https://api.github.com/users/allenai/followers","following_url":"https://api.github.com/users/allenai/following{/other_user}","gists_url":"https://api.github.com/users/allenai/gists{/gist_id}","starred_url":"https://api.github.com/users/allenai/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/allenai/subscriptions","organizations_url":"https://api.github.com/users/allenai/orgs","repos_url":"https://api.github.com/users/allenai/repos","events_url":"https://api.github.com/users/allenai/events{/privacy}","received_events_url":"https://api.github.com/users/allenai/received_events","type":"Organization","site_admin":false},"repo":{"id":123380305,"node_id":"MDEwOlJlcG9zaXRvcnkxMjMzODAzMDU=","name":"amti","full_name":"allenai/amti","private":false,"owner":{"login":"allenai","id":5667695,"node_id":"MDEyOk9yZ2FuaXphdGlvbjU2Njc2OTU=","avatar_url":"https://avatars3.githubusercontent.com/u/5667695?v=4","gravatar_id":"","url":"https://api.github.com/users/allenai","html_url":"https://github.com/allenai","followers_url":"https://api.github.com/users/allenai/followers","following_url":"https://api.github.com/users/allenai/following{/other_user}","gists_url":"https://api.github.com/users/allenai/gists{/gist_id}","starred_url":"https://api.github.com/users/allenai/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/allenai/subscriptions","organizations_url":"https://api.github.com/users/allenai/orgs","repos_url":"https://api.github.com/users/allenai/repos","events_url":"https://api.github.com/users/allenai/events{/privacy}","received_events_url":"https://api.github.com/users/allenai/received_events","type":"Organization","site_admin":false},"html_url":"https://github.com/allenai/amti","description":"A Mechanical Turk Interface (amti) 🤖","fork":false,"url":"https://api.github.com/repos/allenai/amti","forks_url":"https://api.github.com/repos/allenai/amti/forks","keys_url":"https://api.github.com/repos/allenai/amti/keys{/key_id}","collaborators_url":"https://api.github.com/repos/allenai/amti/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/allenai/amti/teams","hooks_url":"https://api.github.com/repos/allenai/amti/hooks","issue_events_url":"https://api.github.com/repos/allenai/amti/issues/events{/number}","events_url":"https://api.github.com/repos/allenai/amti/events","assignees_url":"https://api.github.com/repos/allenai/amti/assignees{/user}","branches_url":"https://api.github.com/repos/allenai/amti/branches{/branch}","tags_url":"https://api.github.com/repos/allenai/amti/tags","blobs_url":"https://api.github.com/repos/allenai/amti/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/allenai/amti/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/allenai/amti/git/refs{/sha}","trees_url":"https://api.github.com/repos/allenai/amti/git/trees{/sha}","statuses_url":"https://api.github.com/repos/allenai/amti/statuses/{sha}","languages_url":"https://api.github.com/repos/allenai/amti/languages","stargazers_url":"https://api.github.com/repos/allenai/amti/stargazers","contributors_url":"https://api.github.com/repos/allenai/amti/contributors","subscribers_url":"https://api.github.com/repos/allenai/amti/subscribers","subscription_url":"https://api.github.com/repos/allenai/amti/subscription","commits_url":"https://api.github.com/repos/allenai/amti/commits{/sha}","git_commits_url":"https://api.github.com/repos/allenai/amti/git/commits{/sha}","comments_url":"https://api.github.com/repos/allenai/amti/comments{/number}","issue_comment_url":"https://api.github.com/repos/allenai/amti/issues/comments{/number}","contents_url":"https://api.github.com/repos/allenai/amti/contents/{+path}","compare_url":"https://api.github.com/repos/allenai/amti/compare/{base}...{head}","merges_url":"https://api.github.com/repos/allenai/amti/merges","archive_url":"https://api.github.com/repos/allenai/amti/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/allenai/amti/downloads","issues_url":"https://api.github.com/repos/allenai/amti/issues{/number}","pulls_url":"https://api.github.com/repos/allenai/amti/pulls{/number}","milestones_url":"https://api.github.com/repos/allenai/amti/milestones{/number}","notifications_url":"https://api.github.com/repos/allenai/amti/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/allenai/amti/labels{/name}","releases_url":"https://api.github.com/repos/allenai/amti/releases{/id}","deployments_url":"https://api.github.com/repos/allenai/amti/deployments","created_at":"2018-03-01T04:05:02Z","updated_at":"2018-11-09T20:53:58Z","pushed_at":"2018-11-09T08:47:18Z","git_url":"git://github.com/allenai/amti.git","ssh_url":"git@github.com:allenai/amti.git","clone_url":"https://github.com/allenai/amti.git","svn_url":"https://github.com/allenai/amti","homepage":"","size":47,"stargazers_count":10,"watchers_count":10,"language":"Python","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":2,"mirror_url":null,"archived":false,"open_issues_count":3,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0","node_id":"MDc6TGljZW5zZTI="},"forks":2,"open_issues":3,"watchers":10,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/allenai/amti/pulls/5"},"html":{"href":"https://github.com/allenai/amti/pull/5"},"issue":{"href":"https://api.github.com/repos/allenai/amti/issues/5"},"comments":{"href":"https://api.github.com/repos/allenai/amti/issues/5/comments"},"review_comments":{"href":"https://api.github.com/repos/allenai/amti/pulls/5/comments"},"review_comment":{"href":"https://api.github.com/repos/allenai/amti/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/allenai/amti/pulls/5/commits"},"statuses":{"href":"https://api.github.com/repos/allenai/amti/statuses/e04235b5b1a8589abc18f78cd9ba3053ecbb332d"}},"author_association":"COLLABORATOR"}}
{"actor":{"display_login":"nalourie-ai2"}}