{"action":"created","issue":{"url":"https://api.github.com/repos/mediachain/aleph/issues/44","repository_url":"https://api.github.com/repos/mediachain/aleph","labels_url":"https://api.github.com/repos/mediachain/aleph/issues/44/labels{/name}","comments_url":"https://api.github.com/repos/mediachain/aleph/issues/44/comments","events_url":"https://api.github.com/repos/mediachain/aleph/issues/44/events","html_url":"https://github.com/mediachain/aleph/issues/44","id":185429296,"number":44,"title":"Schema versioning and translation","user":{"login":"yusefnapora","id":678715,"avatar_url":"https://avatars.githubusercontent.com/u/678715?v=3","gravatar_id":"","url":"https://api.github.com/users/yusefnapora","html_url":"https://github.com/yusefnapora","followers_url":"https://api.github.com/users/yusefnapora/followers","following_url":"https://api.github.com/users/yusefnapora/following{/other_user}","gists_url":"https://api.github.com/users/yusefnapora/gists{/gist_id}","starred_url":"https://api.github.com/users/yusefnapora/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yusefnapora/subscriptions","organizations_url":"https://api.github.com/users/yusefnapora/orgs","repos_url":"https://api.github.com/users/yusefnapora/repos","events_url":"https://api.github.com/users/yusefnapora/events{/privacy}","received_events_url":"https://api.github.com/users/yusefnapora/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":6,"created_at":"2016-10-26T15:32:46Z","updated_at":"2016-10-26T16:50:08Z","closed_at":null,"body":"Opening this to discuss and plan the schema semantics, especially versioning and translation.\r\nBelow are some notes that describe my current thinking:\r\n\r\n## Versioning\r\n- Schemas are stored as mediachain records in jsonschema format in a well-known namespace (e.g `mediachain.schemas` or something). \r\n- Schemas have human-readble names + semantic versions, which combined form the WKI used to look them up.  e.g. `SELECT * FROM mediachain.schemas WHERE wki = io.mediachain.indexer/image/jsonschema/1-0-0`\r\n- A schema is \"self-describing\", in that it contains a json object that states its name, version, etc.\r\n- Examples here use the snowplow [self-describing schema format](https://github.com/snowplow/iglu/wiki/Self-describing-JSON-Schemas) of `vendor/schema-name/format/version`, where version is a [SchemaVer](https://github.com/snowplow/iglu/wiki/SchemaVer) version string\r\n- Mediachain records should contain a link to the schema that they conform to.\r\n- Link format is up for discussion; actual IPLD-compatible multiaddrs don't support custom protocol codes, so linking directly to a mediachain object blob by multihash would probably need something else.\r\n- I think we should probably have a \"link object\" that has, at minimum, the WKI of the schema + a multihash of the schema blob.  It could also have e.g. http links as a fallback.\r\n\r\nPossible example of record with schema link:\r\n```\r\n{\r\n\t\"schema\": \r\n\t{\r\n\t\t\"wki\": \"io.mediachain.indexer/image/jsonschema/1-0-0,\" \r\n\t\t\"object\": \"QmF00...\", \r\n\t\t\"http\": \"http://indexer.mediachain.io/schemas/image/jsonschema/1-0-0\", \r\n\t\t\"ipfs\": {\"/\": \"QmF00\"}\r\n\t},\r\n\t\"data\":\r\n\t{\r\n\t\t\"id\": \"foo\",\r\n\t\t\"description\": \"actual record that conforms to schema above\"\r\n\t}\r\n}\r\n```\r\n\r\nThe example above uses the [snowplow convention](https://github.com/snowplow/iglu/wiki/Self-describing-JSONs) of wrapping the actual payload in a `data` object, which lets us avoid having to designate a special \"schema key\" that you're not allowed to use in your actual data payload.\r\n\r\nWhen a record is published, the schema object will be listed in the `deps` field of the statement, so that when you fetch the record you'll also get its schema.  This should ensure that we don't end up with \"orphan\" records whose schema can't be retrieved.  It might also be useful to put the WKI of the schema into the statement's `tags` field, so that users can filter by objects they know how to work with.\r\n\r\n## Translation\r\nA \"translator\" is a [jq filter](https://stedolan.github.io/jq/manual/#Basicfilters) that will massage data from your schema into another.\r\n\r\n- A translator is published as a mediachain record; it can be a json object with links to the source and destination schemas, plus the jq filter string (and version of jq used!).\r\n- translators are \"one way\", and may be lossy; there's no guarantee that the translated output can be re-translated back into the original schema\r\n- However, a translated record should contain a link to the original record and the translator used.  This could be another top-level field like `schema` and `data` above\r\n- In theory, translation can happen \"on the fly\" at read time, instead of a node republishing translated records.  Whether that's a good idea will depend on read volumes, etc.\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/mediachain/aleph/issues/comments/256409650","html_url":"https://github.com/mediachain/aleph/issues/44#issuecomment-256409650","issue_url":"https://api.github.com/repos/mediachain/aleph/issues/44","id":256409650,"user":{"login":"autoencoder","id":19392619,"avatar_url":"https://avatars.githubusercontent.com/u/19392619?v=3","gravatar_id":"","url":"https://api.github.com/users/autoencoder","html_url":"https://github.com/autoencoder","followers_url":"https://api.github.com/users/autoencoder/followers","following_url":"https://api.github.com/users/autoencoder/following{/other_user}","gists_url":"https://api.github.com/users/autoencoder/gists{/gist_id}","starred_url":"https://api.github.com/users/autoencoder/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/autoencoder/subscriptions","organizations_url":"https://api.github.com/users/autoencoder/orgs","repos_url":"https://api.github.com/users/autoencoder/repos","events_url":"https://api.github.com/users/autoencoder/events{/privacy}","received_events_url":"https://api.github.com/users/autoencoder/received_events","type":"User","site_admin":false},"created_at":"2016-10-26T16:50:08Z","updated_at":"2016-10-26T16:50:08Z","body":"One alternative to JSON-defined schema to consider is a flat schema definition, e.g:\r\n\r\n- [Flattening (shredding) and reassembling nested JSON schemas -- Section 4 of the Dremel Paper](http://static.googleusercontent.com/media/research.google.com/en//pubs/archive/36632.pdf)\r\n\r\n- [SQL-like schema definition via REPEATED -- which internally uses the Dremel method](https://eventql.io/documentation/tables/datatypes/) and [see also](https://github.com/eventql/eventql/blob/master/doc/getting-started/concepts.md).\r\n\r\nBenefits:\r\n\r\n- Can be directly used to easily allow efficient storage of records in more traditional flat schema DBs, e.g. traditional columnar data warehouse DBs.\r\n\r\n- Makes it easy to do all kinds of reasoning about the schema with fewer LOC.\r\n"}}
{"actor":{"display_login":"autoencoder"}}