{"action":"created","comment":{"url":"https://api.github.com/repos/datalad/datalad/pulls/comments/76295992","id":76295992,"diff_hunk":"@@ -116,6 +116,14 @@ class Create(Interface):\n             of datasets across platforms (especially those with limited\n             path lengths)\"\"\",\n             nargs=1),\n+        native_metadata_type=Parameter(\n+            args=('--native-metadata-type',),\n+            metavar='LABEL',\n+            action='append',\n+            constraints=EnsureStr() | EnsureNone(),\n+            doc=\"\"\"Metadata type label. Match match the name of the respective\n+            parser implementation in Datalad (i.e. \"bids\").[CMD:  This option","path":"datalad/distribution/create.py","position":null,"original_position":10,"commit_id":"8df39cb1fbd44233e5a0d53eda6e2360fe6addfb","original_commit_id":"f7adc3bee6ab0ea6518a5466c431500d5dac39e4","user":{"login":"hanke","id":136479,"avatar_url":"https://avatars.githubusercontent.com/u/136479?v=3","gravatar_id":"","url":"https://api.github.com/users/hanke","html_url":"https://github.com/hanke","followers_url":"https://api.github.com/users/hanke/followers","following_url":"https://api.github.com/users/hanke/following{/other_user}","gists_url":"https://api.github.com/users/hanke/gists{/gist_id}","starred_url":"https://api.github.com/users/hanke/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hanke/subscriptions","organizations_url":"https://api.github.com/users/hanke/orgs","repos_url":"https://api.github.com/users/hanke/repos","events_url":"https://api.github.com/users/hanke/events{/privacy}","received_events_url":"https://api.github.com/users/hanke/received_events","type":"User","site_admin":false},"body":"Done.","created_at":"2016-08-25T18:11:35Z","updated_at":"2016-08-25T18:11:35Z","html_url":"https://github.com/datalad/datalad/pull/682#discussion_r76295992","pull_request_url":"https://api.github.com/repos/datalad/datalad/pulls/682","_links":{"self":{"href":"https://api.github.com/repos/datalad/datalad/pulls/comments/76295992"},"html":{"href":"https://github.com/datalad/datalad/pull/682#discussion_r76295992"},"pull_request":{"href":"https://api.github.com/repos/datalad/datalad/pulls/682"}}},"pull_request":{"url":"https://api.github.com/repos/datalad/datalad/pulls/682","id":79729587,"html_url":"https://github.com/datalad/datalad/pull/682","diff_url":"https://github.com/datalad/datalad/pull/682.diff","patch_url":"https://github.com/datalad/datalad/pull/682.patch","issue_url":"https://api.github.com/repos/datalad/datalad/issues/682","number":682,"state":"open","locked":false,"title":"WiP: JSON-LD based metadata","user":{"login":"hanke","id":136479,"avatar_url":"https://avatars.githubusercontent.com/u/136479?v=3","gravatar_id":"","url":"https://api.github.com/users/hanke","html_url":"https://github.com/hanke","followers_url":"https://api.github.com/users/hanke/followers","following_url":"https://api.github.com/users/hanke/following{/other_user}","gists_url":"https://api.github.com/users/hanke/gists{/gist_id}","starred_url":"https://api.github.com/users/hanke/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hanke/subscriptions","organizations_url":"https://api.github.com/users/hanke/orgs","repos_url":"https://api.github.com/users/hanke/repos","events_url":"https://api.github.com/users/hanke/events{/privacy}","received_events_url":"https://api.github.com/users/hanke/received_events","type":"User","site_admin":false},"body":"FIrst the picture of the current state:\r\n\r\n````\r\n{\r\n  \"@context\": \"http://schema.org/\",\r\n  \"@graph\": [\r\n    {\r\n      \"id\": \"2146817a-d7ab-46e5-9133-4a725306dbc5\",\r\n      \"type\": \"Dataset\",\r\n      \"dc:hasPart\": {\r\n        \"id\": \"5a798601-8d66-42a9-ada9-4dceab591cbb\"\r\n      },\r\n      \"dc:hasVersion\": {\r\n        \"id\": \"974d1090-32e4-4d0c-86aa-b345c5c2ba41\"\r\n      },\r\n      \"name\": \"the mother\"\r\n    },\r\n    {\r\n      \"id\": \"5a798601-8d66-42a9-ada9-4dceab591cbb\",\r\n      \"type\": \"Dataset\",\r\n      \"location\": \"sub\",\r\n      \"name\": \"child\"\r\n    }\r\n  ]\r\n}\r\n````\r\n\r\nThis is the metadata of a dataset with one subdataset, that also has a clone somewhere else (see datalad/metadata/tests/test_base.py:test_basic_metadata). The `name` properties are coming from a BIDS structure (presence of BIDS metadata was auto-detected), the rest comes from Git/Git-annex itself.\r\n\r\nThe goal here is to get a list (one item per dataset) of dicts with a more-or-less plain key-value mapping (no nested structures) -- `@id` being the only and necessary exception to get valid JSON-LD. Such metadata structure should be easily convertible into a queryable form (SQL, or NO-SQL, etc).\r\n\r\nConfession:\r\n\r\nThe above is not quiet true, as this is a compacted and flattened graph, for which a call to pyld's `jsonld.flatten()` is not yet pushed. This is a rather expensive call (needs network (or local cache) for term resolution), but it allows us to condense pretty much arbitrary metadata sets into a minimal form. It only needs to be done when metadata is aggregated and cached, not for query. pyld is in Debian and available through pip.\r\n\r\nIn contrast to what I said before, I now think we should store metadata as JSON-LD and not as triples. JSON-LD can get us triples if we need them. But JSON-LD is plain JSON, hence we can limit metadata read requirements to just `json`, and do not need to worry about JSON-LD.\r\n\r\nIf you want to play with the formats look here: http://tinyurl.com/jb62uhu\r\nThe top show the actual metadata structure generated by the current code. Various other flavors are available at the bottom. For expansion to triples/quads the UUIDs would get prefixed with something like `http://db.datalad.org/ds/`. Where we could provide information on the datasets that we know. The fact that the UUID are clone-specific should not hurt. The current PR already stores the origin UUID for newly created datasets, and the graph info would allow us to trace clones.\r\n\r\nMore info:  http://json-ld.org\r\n\r\nTODO:\r\n\r\n- cache metadata\r\n- retrieve cached metadata for query, instead of reparse\r\n","created_at":"2016-08-02T12:52:37Z","updated_at":"2016-08-25T18:11:35Z","closed_at":null,"merged_at":null,"merge_commit_sha":"b32aa86f636cde6ee8b5f56fa80d038c3b06e530","assignee":null,"assignees":[],"milestone":null,"commits_url":"https://api.github.com/repos/datalad/datalad/pulls/682/commits","review_comments_url":"https://api.github.com/repos/datalad/datalad/pulls/682/comments","review_comment_url":"https://api.github.com/repos/datalad/datalad/pulls/comments{/number}","comments_url":"https://api.github.com/repos/datalad/datalad/issues/682/comments","statuses_url":"https://api.github.com/repos/datalad/datalad/statuses/8df39cb1fbd44233e5a0d53eda6e2360fe6addfb","head":{"label":"hanke:nf-metaparser","ref":"nf-metaparser","sha":"8df39cb1fbd44233e5a0d53eda6e2360fe6addfb","user":{"login":"hanke","id":136479,"avatar_url":"https://avatars.githubusercontent.com/u/136479?v=3","gravatar_id":"","url":"https://api.github.com/users/hanke","html_url":"https://github.com/hanke","followers_url":"https://api.github.com/users/hanke/followers","following_url":"https://api.github.com/users/hanke/following{/other_user}","gists_url":"https://api.github.com/users/hanke/gists{/gist_id}","starred_url":"https://api.github.com/users/hanke/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hanke/subscriptions","organizations_url":"https://api.github.com/users/hanke/orgs","repos_url":"https://api.github.com/users/hanke/repos","events_url":"https://api.github.com/users/hanke/events{/privacy}","received_events_url":"https://api.github.com/users/hanke/received_events","type":"User","site_admin":false},"repo":{"id":34503646,"name":"datalad","full_name":"hanke/datalad","owner":{"login":"hanke","id":136479,"avatar_url":"https://avatars.githubusercontent.com/u/136479?v=3","gravatar_id":"","url":"https://api.github.com/users/hanke","html_url":"https://github.com/hanke","followers_url":"https://api.github.com/users/hanke/followers","following_url":"https://api.github.com/users/hanke/following{/other_user}","gists_url":"https://api.github.com/users/hanke/gists{/gist_id}","starred_url":"https://api.github.com/users/hanke/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hanke/subscriptions","organizations_url":"https://api.github.com/users/hanke/orgs","repos_url":"https://api.github.com/users/hanke/repos","events_url":"https://api.github.com/users/hanke/events{/privacy}","received_events_url":"https://api.github.com/users/hanke/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/hanke/datalad","description":"Keep scientific data under control by git  and git-annex","fork":true,"url":"https://api.github.com/repos/hanke/datalad","forks_url":"https://api.github.com/repos/hanke/datalad/forks","keys_url":"https://api.github.com/repos/hanke/datalad/keys{/key_id}","collaborators_url":"https://api.github.com/repos/hanke/datalad/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/hanke/datalad/teams","hooks_url":"https://api.github.com/repos/hanke/datalad/hooks","issue_events_url":"https://api.github.com/repos/hanke/datalad/issues/events{/number}","events_url":"https://api.github.com/repos/hanke/datalad/events","assignees_url":"https://api.github.com/repos/hanke/datalad/assignees{/user}","branches_url":"https://api.github.com/repos/hanke/datalad/branches{/branch}","tags_url":"https://api.github.com/repos/hanke/datalad/tags","blobs_url":"https://api.github.com/repos/hanke/datalad/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/hanke/datalad/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/hanke/datalad/git/refs{/sha}","trees_url":"https://api.github.com/repos/hanke/datalad/git/trees{/sha}","statuses_url":"https://api.github.com/repos/hanke/datalad/statuses/{sha}","languages_url":"https://api.github.com/repos/hanke/datalad/languages","stargazers_url":"https://api.github.com/repos/hanke/datalad/stargazers","contributors_url":"https://api.github.com/repos/hanke/datalad/contributors","subscribers_url":"https://api.github.com/repos/hanke/datalad/subscribers","subscription_url":"https://api.github.com/repos/hanke/datalad/subscription","commits_url":"https://api.github.com/repos/hanke/datalad/commits{/sha}","git_commits_url":"https://api.github.com/repos/hanke/datalad/git/commits{/sha}","comments_url":"https://api.github.com/repos/hanke/datalad/comments{/number}","issue_comment_url":"https://api.github.com/repos/hanke/datalad/issues/comments{/number}","contents_url":"https://api.github.com/repos/hanke/datalad/contents/{+path}","compare_url":"https://api.github.com/repos/hanke/datalad/compare/{base}...{head}","merges_url":"https://api.github.com/repos/hanke/datalad/merges","archive_url":"https://api.github.com/repos/hanke/datalad/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/hanke/datalad/downloads","issues_url":"https://api.github.com/repos/hanke/datalad/issues{/number}","pulls_url":"https://api.github.com/repos/hanke/datalad/pulls{/number}","milestones_url":"https://api.github.com/repos/hanke/datalad/milestones{/number}","notifications_url":"https://api.github.com/repos/hanke/datalad/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/hanke/datalad/labels{/name}","releases_url":"https://api.github.com/repos/hanke/datalad/releases{/id}","deployments_url":"https://api.github.com/repos/hanke/datalad/deployments","created_at":"2015-04-24T07:11:39Z","updated_at":"2016-02-23T15:27:28Z","pushed_at":"2016-08-25T18:11:21Z","git_url":"git://github.com/hanke/datalad.git","ssh_url":"git@github.com:hanke/datalad.git","clone_url":"https://github.com/hanke/datalad.git","svn_url":"https://github.com/hanke/datalad","homepage":null,"size":4645,"stargazers_count":0,"watchers_count":0,"language":"Python","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"datalad:master","ref":"master","sha":"10e9df160f8d0db57b3976ca61d60aceb09f1d77","user":{"login":"datalad","id":8927200,"avatar_url":"https://avatars.githubusercontent.com/u/8927200?v=3","gravatar_id":"","url":"https://api.github.com/users/datalad","html_url":"https://github.com/datalad","followers_url":"https://api.github.com/users/datalad/followers","following_url":"https://api.github.com/users/datalad/following{/other_user}","gists_url":"https://api.github.com/users/datalad/gists{/gist_id}","starred_url":"https://api.github.com/users/datalad/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/datalad/subscriptions","organizations_url":"https://api.github.com/users/datalad/orgs","repos_url":"https://api.github.com/users/datalad/repos","events_url":"https://api.github.com/users/datalad/events{/privacy}","received_events_url":"https://api.github.com/users/datalad/received_events","type":"Organization","site_admin":false},"repo":{"id":14052034,"name":"datalad","full_name":"datalad/datalad","owner":{"login":"datalad","id":8927200,"avatar_url":"https://avatars.githubusercontent.com/u/8927200?v=3","gravatar_id":"","url":"https://api.github.com/users/datalad","html_url":"https://github.com/datalad","followers_url":"https://api.github.com/users/datalad/followers","following_url":"https://api.github.com/users/datalad/following{/other_user}","gists_url":"https://api.github.com/users/datalad/gists{/gist_id}","starred_url":"https://api.github.com/users/datalad/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/datalad/subscriptions","organizations_url":"https://api.github.com/users/datalad/orgs","repos_url":"https://api.github.com/users/datalad/repos","events_url":"https://api.github.com/users/datalad/events{/privacy}","received_events_url":"https://api.github.com/users/datalad/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/datalad/datalad","description":"Keep scientific data under control with git  and git-annex","fork":false,"url":"https://api.github.com/repos/datalad/datalad","forks_url":"https://api.github.com/repos/datalad/datalad/forks","keys_url":"https://api.github.com/repos/datalad/datalad/keys{/key_id}","collaborators_url":"https://api.github.com/repos/datalad/datalad/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/datalad/datalad/teams","hooks_url":"https://api.github.com/repos/datalad/datalad/hooks","issue_events_url":"https://api.github.com/repos/datalad/datalad/issues/events{/number}","events_url":"https://api.github.com/repos/datalad/datalad/events","assignees_url":"https://api.github.com/repos/datalad/datalad/assignees{/user}","branches_url":"https://api.github.com/repos/datalad/datalad/branches{/branch}","tags_url":"https://api.github.com/repos/datalad/datalad/tags","blobs_url":"https://api.github.com/repos/datalad/datalad/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/datalad/datalad/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/datalad/datalad/git/refs{/sha}","trees_url":"https://api.github.com/repos/datalad/datalad/git/trees{/sha}","statuses_url":"https://api.github.com/repos/datalad/datalad/statuses/{sha}","languages_url":"https://api.github.com/repos/datalad/datalad/languages","stargazers_url":"https://api.github.com/repos/datalad/datalad/stargazers","contributors_url":"https://api.github.com/repos/datalad/datalad/contributors","subscribers_url":"https://api.github.com/repos/datalad/datalad/subscribers","subscription_url":"https://api.github.com/repos/datalad/datalad/subscription","commits_url":"https://api.github.com/repos/datalad/datalad/commits{/sha}","git_commits_url":"https://api.github.com/repos/datalad/datalad/git/commits{/sha}","comments_url":"https://api.github.com/repos/datalad/datalad/comments{/number}","issue_comment_url":"https://api.github.com/repos/datalad/datalad/issues/comments{/number}","contents_url":"https://api.github.com/repos/datalad/datalad/contents/{+path}","compare_url":"https://api.github.com/repos/datalad/datalad/compare/{base}...{head}","merges_url":"https://api.github.com/repos/datalad/datalad/merges","archive_url":"https://api.github.com/repos/datalad/datalad/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/datalad/datalad/downloads","issues_url":"https://api.github.com/repos/datalad/datalad/issues{/number}","pulls_url":"https://api.github.com/repos/datalad/datalad/pulls{/number}","milestones_url":"https://api.github.com/repos/datalad/datalad/milestones{/number}","notifications_url":"https://api.github.com/repos/datalad/datalad/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/datalad/datalad/labels{/name}","releases_url":"https://api.github.com/repos/datalad/datalad/releases{/id}","deployments_url":"https://api.github.com/repos/datalad/datalad/deployments","created_at":"2013-11-01T19:40:08Z","updated_at":"2016-08-15T01:13:11Z","pushed_at":"2016-08-25T18:11:22Z","git_url":"git://github.com/datalad/datalad.git","ssh_url":"git@github.com:datalad/datalad.git","clone_url":"https://github.com/datalad/datalad.git","svn_url":"https://github.com/datalad/datalad","homepage":"http://datalad.org","size":8792,"stargazers_count":37,"watchers_count":37,"language":"Python","has_issues":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":9,"mirror_url":null,"open_issues_count":163,"forks":9,"open_issues":163,"watchers":37,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/datalad/datalad/pulls/682"},"html":{"href":"https://github.com/datalad/datalad/pull/682"},"issue":{"href":"https://api.github.com/repos/datalad/datalad/issues/682"},"comments":{"href":"https://api.github.com/repos/datalad/datalad/issues/682/comments"},"review_comments":{"href":"https://api.github.com/repos/datalad/datalad/pulls/682/comments"},"review_comment":{"href":"https://api.github.com/repos/datalad/datalad/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/datalad/datalad/pulls/682/commits"},"statuses":{"href":"https://api.github.com/repos/datalad/datalad/statuses/8df39cb1fbd44233e5a0d53eda6e2360fe6addfb"}}}}
{"actor":{"display_login":"hanke"}}