{"action":"created","issue":{"url":"https://api.github.com/repos/datalad/datalad/issues/682","repository_url":"https://api.github.com/repos/datalad/datalad","labels_url":"https://api.github.com/repos/datalad/datalad/issues/682/labels{/name}","comments_url":"https://api.github.com/repos/datalad/datalad/issues/682/comments","events_url":"https://api.github.com/repos/datalad/datalad/issues/682/events","html_url":"https://github.com/datalad/datalad/pull/682","id":168877520,"number":682,"title":"NF: JSON-LD based metadata","user":{"login":"mih","id":136479,"avatar_url":"https://avatars.githubusercontent.com/u/136479?v=3","gravatar_id":"","url":"https://api.github.com/users/mih","html_url":"https://github.com/mih","followers_url":"https://api.github.com/users/mih/followers","following_url":"https://api.github.com/users/mih/following{/other_user}","gists_url":"https://api.github.com/users/mih/gists{/gist_id}","starred_url":"https://api.github.com/users/mih/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mih/subscriptions","organizations_url":"https://api.github.com/users/mih/orgs","repos_url":"https://api.github.com/users/mih/repos","events_url":"https://api.github.com/users/mih/events{/privacy}","received_events_url":"https://api.github.com/users/mih/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":68,"created_at":"2016-08-02T12:52:37Z","updated_at":"2016-09-05T13:29:24Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/datalad/datalad/pulls/682","html_url":"https://github.com/datalad/datalad/pull/682","diff_url":"https://github.com/datalad/datalad/pull/682.diff","patch_url":"https://github.com/datalad/datalad/pull/682.patch"},"body":"FIrst the picture of the current state:\r\n\r\n````\r\n{\r\n  \"@context\": \"http://schema.org/\",\r\n  \"@graph\": [\r\n    {\r\n      \"id\": \"2146817a-d7ab-46e5-9133-4a725306dbc5\",\r\n      \"type\": \"Dataset\",\r\n      \"dc:hasPart\": {\r\n        \"id\": \"5a798601-8d66-42a9-ada9-4dceab591cbb\"\r\n      },\r\n      \"dc:hasVersion\": {\r\n        \"id\": \"974d1090-32e4-4d0c-86aa-b345c5c2ba41\"\r\n      },\r\n      \"name\": \"the mother\"\r\n    },\r\n    {\r\n      \"id\": \"5a798601-8d66-42a9-ada9-4dceab591cbb\",\r\n      \"type\": \"Dataset\",\r\n      \"location\": \"sub\",\r\n      \"name\": \"child\"\r\n    }\r\n  ]\r\n}\r\n````\r\n\r\nThis is the metadata of a dataset with one subdataset, that also has a clone somewhere else (see datalad/metadata/tests/test_base.py:test_basic_metadata). The `name` properties are coming from a BIDS structure (presence of BIDS metadata was auto-detected), the rest comes from Git/Git-annex itself.\r\n\r\nThe goal here is to get a list (one item per dataset) of dicts with a more-or-less plain key-value mapping (no nested structures) -- `@id` being the only and necessary exception to get valid JSON-LD. Such metadata structure should be easily convertible into a queryable form (SQL, or NO-SQL, etc).\r\n\r\nConfession:\r\n\r\nThe above is not quiet true, as this is a compacted and flattened graph, for which a call to pyld's `jsonld.flatten()` is not yet pushed. This is a rather expensive call (needs network (or local cache) for term resolution), but it allows us to condense pretty much arbitrary metadata sets into a minimal form. It only needs to be done when metadata is aggregated and cached, not for query. pyld is in Debian and available through pip.\r\n\r\nIn contrast to what I said before, I now think we should store metadata as JSON-LD and not as triples. JSON-LD can get us triples if we need them. But JSON-LD is plain JSON, hence we can limit metadata read requirements to just `json`, and do not need to worry about JSON-LD.\r\n\r\nIf you want to play with the formats look here: http://tinyurl.com/jb62uhu\r\nThe top show the actual metadata structure generated by the current code. Various other flavors are available at the bottom. For expansion to triples/quads the UUIDs would get prefixed with something like `http://db.datalad.org/ds/`. Where we could provide information on the datasets that we know. The fact that the UUID are clone-specific should not hurt. The current PR already stores the origin UUID for newly created datasets, and the graph info would allow us to trace clones.\r\n\r\nMore info:  http://json-ld.org\r\n\r\nTODO:\r\n\r\n- cache metadata\r\n- retrieve cached metadata for query, instead of reparse\r\n"},"comment":{"url":"https://api.github.com/repos/datalad/datalad/issues/comments/244747222","html_url":"https://github.com/datalad/datalad/pull/682#issuecomment-244747222","issue_url":"https://api.github.com/repos/datalad/datalad/issues/682","id":244747222,"user":{"login":"coveralls","id":2354108,"avatar_url":"https://avatars.githubusercontent.com/u/2354108?v=3","gravatar_id":"","url":"https://api.github.com/users/coveralls","html_url":"https://github.com/coveralls","followers_url":"https://api.github.com/users/coveralls/followers","following_url":"https://api.github.com/users/coveralls/following{/other_user}","gists_url":"https://api.github.com/users/coveralls/gists{/gist_id}","starred_url":"https://api.github.com/users/coveralls/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/coveralls/subscriptions","organizations_url":"https://api.github.com/users/coveralls/orgs","repos_url":"https://api.github.com/users/coveralls/repos","events_url":"https://api.github.com/users/coveralls/events{/privacy}","received_events_url":"https://api.github.com/users/coveralls/received_events","type":"User","site_admin":false},"created_at":"2016-09-05T13:29:23Z","updated_at":"2016-09-05T13:29:23Z","body":"\n[![Coverage Status](https://coveralls.io/builds/7748123/badge)](https://coveralls.io/builds/7748123)\n\nCoverage decreased (-2.4%) to 86.459% when pulling **62b4f7c80b7d9842291c4318a6232916aaff1eb9 on mih:nf-metaparser** into **e4673ff7b256f9d0c621a9a2cd026710281b3ce5 on datalad:master**.\n"}}
{"actor":{"display_login":"coveralls"}}