{"action":"created","issue":{"url":"https://api.github.com/repos/mozilla/mentat/issues/756","repository_url":"https://api.github.com/repos/mozilla/mentat","labels_url":"https://api.github.com/repos/mozilla/mentat/issues/756/labels{/name}","comments_url":"https://api.github.com/repos/mozilla/mentat/issues/756/comments","events_url":"https://api.github.com/repos/mozilla/mentat/issues/756/events","html_url":"https://github.com/mozilla/mentat/issues/756","id":334673298,"node_id":"MDU6SXNzdWUzMzQ2NzMyOTg=","number":756,"title":"Core schema seems insufficient","user":{"login":"grigoryk","id":465422,"node_id":"MDQ6VXNlcjQ2NTQyMg==","avatar_url":"https://avatars3.githubusercontent.com/u/465422?v=4","gravatar_id":"","url":"https://api.github.com/users/grigoryk","html_url":"https://github.com/grigoryk","followers_url":"https://api.github.com/users/grigoryk/followers","following_url":"https://api.github.com/users/grigoryk/following{/other_user}","gists_url":"https://api.github.com/users/grigoryk/gists{/gist_id}","starred_url":"https://api.github.com/users/grigoryk/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/grigoryk/subscriptions","organizations_url":"https://api.github.com/users/grigoryk/orgs","repos_url":"https://api.github.com/users/grigoryk/repos","events_url":"https://api.github.com/users/grigoryk/events{/privacy}","received_events_url":"https://api.github.com/users/grigoryk/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-06-21T22:00:40Z","updated_at":"2018-07-08T23:45:20Z","closed_at":null,"author_association":"COLLABORATOR","body":"If core schema is viewed through a lens of multiple mentats figuring out if they're compatible with each other, it seems insufficient. Currently, if two instances agree on the core schema, their transactions are not necessarily compatible with each other.\r\n\r\nQuoting @ncalexan, \"[when defining vocabularies] we can say a thing needs “to be” an enum (`:db.type/ref`) but we can’t say “and these are the valid enum cases `:db.cardinality/*`“.  (And the transactor doesn’t handle that value restriction at all.)\"\r\n\r\n@rnewman you've added the core schema initially; do you think it's reasonable to expand it going forward? Do you think having a subset of the bootstrap transaction defined as a special thing is valuable?"},"comment":{"url":"https://api.github.com/repos/mozilla/mentat/issues/comments/403326529","html_url":"https://github.com/mozilla/mentat/issues/756#issuecomment-403326529","issue_url":"https://api.github.com/repos/mozilla/mentat/issues/756","id":403326529,"node_id":"MDEyOklzc3VlQ29tbWVudDQwMzMyNjUyOQ==","user":{"login":"rnewman","id":91722,"node_id":"MDQ6VXNlcjkxNzIy","avatar_url":"https://avatars2.githubusercontent.com/u/91722?v=4","gravatar_id":"","url":"https://api.github.com/users/rnewman","html_url":"https://github.com/rnewman","followers_url":"https://api.github.com/users/rnewman/followers","following_url":"https://api.github.com/users/rnewman/following{/other_user}","gists_url":"https://api.github.com/users/rnewman/gists{/gist_id}","starred_url":"https://api.github.com/users/rnewman/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rnewman/subscriptions","organizations_url":"https://api.github.com/users/rnewman/orgs","repos_url":"https://api.github.com/users/rnewman/repos","events_url":"https://api.github.com/users/rnewman/events{/privacy}","received_events_url":"https://api.github.com/users/rnewman/received_events","type":"User","site_admin":false},"created_at":"2018-07-08T23:45:20Z","updated_at":"2018-07-08T23:45:20Z","author_association":"COLLABORATOR","body":"Before I dig into this, one minor clarification: `:db.type/ref` doesn't mean enum. Some ref attributes are used as an enum-like thing, but not all, and enums aren't even necessarily named. Mentat doesn't provide a way to enforce that specific kind of enum, and doing so is actually not as simple as it looks, because you would need to also restrict the space of operations that can be performed on `:db/ident`. But I digress.\r\n\r\nLet me restate, see if I understand what you're asking.\r\n\r\nYou're saying that:\r\n\r\n- A certain subset of attributes (_e.g._, `:db/cardinality`) are `db.type/ref`, and some of those attributes require their ref values to be entities with `:db/ident` (\"enums\").\r\n- Some enum attributes expect the range of their `:db/ident` attributes to be a restricted set (_e.g._, `{:db.cardinality/one, :db.cardinality/many}`).\r\n- Those restricted sets might well change over time (_e.g._, adding `{:db.cardinality/list}`).\r\n- Those restricted sets are not captured explicitly in the core schema.\r\n- Ergo, we cannot easily detect an important change.\r\n\r\nYou are correct, but it kinda doesn't matter.\r\n\r\nThere will always be cases in which the meaning of a schema changes over time; we (and Datomic) have picked/inherited a number of common axes that we directly support (cardinality, doc, uniqueness, _etc_.), and there are others that we don't (\"metaness\", permanence/transience, _etc_.).\r\n\r\nEven within the set of properties we model in the core schema there are domain-level concepts that can change in a way we don't formally describe.\r\n\r\nThere are three ways to represent that in the vocabulary system:\r\n\r\n1. If the change is backward-compatible (in a scenario as outlined above, typically this will be because the restricted set isn't so restricted after all — imagine a property like `:monkey/species`, where the introduction of `:species/callicebus_miltoni` in 2015 is A-OK), we can simply begin using the new 'enum case', and older clients will probably behave correctly.\r\n2. If the change is not backward-compatible, but only _narrows_ the semantics of the property (`:height/very-tall` is a subset of `:height/tall`), then we can add a second attribute and keep writing the first. This might well be a data modeling error — we should have recorded `:person/height` instead!\r\n3. If the change is not backward-compatible, and (a) there's no way to phrase new data in a way that old clients can handle it safely, and (b) data old clients write should not be missing the new aspect when read by new clients, then we should bump the version number of the schema.\r\n\r\nThe version number of the schema exists precisely to model this kind of exclusion, where the schema cannot only be _implicitly extended_, but needs to be _replaced_.\r\n\r\nI don't think it's all that feasible to model every possible restriction in the schema language itself: after all, there are:\r\n\r\n- Arbitrary restrictions on values: \"account numbers must be 10 digits… oh, actually, now we're prefixing them all with 0000\".\r\n- Arbitrary restrictions on expected structure: \"the range of `:person/residence` is the domain of entities that have a `:country/country_code`… oh, but people can live in places that [don't have country calling codes](https://en.wikipedia.org/wiki/List_of_country_calling_codes#Locations_with_no_country_code)\".\r\n- Higher-level restrictions: attributes that must both appear together, cardinality restrictions that depend on other attributes (you can have as many passport numbers as countries of citizenship…), and so on.\r\n\r\nI used the vocabulary version number to allow developers to indicate that one of these constraints has changed.\r\n\r\n(This kind of sophistication is one reason why even complicated SQL databases support triggers and stored procedures to impose computed constraints!)\r\n\r\nTo return to your question:\r\n\r\nThe reason you're asking is that merging two databases which have the same core vocabulary version but allow different enum cases for a schema-related attribute (cardinality being one such) will break Mentat.\r\n\r\nThe weaker version of that scenario is that merging two databases which have the same non-core vocabulary version but use different enum cases for its non-schema attribute might break application code.\r\n\r\nYou have two choices here.\r\n\r\nThe first, which is the one I was taking, is to say: **don't do that**. If you have an attribute that has a limited range of acceptable entities, then when you add another such entity (you'll find yourself writing `[:db/add \"foo\" :db/ident :my/ident]`) you must do exactly the same thing you would do when you make a non-back-compat change to a vocabulary: bump the vocabulary version.\r\n\r\nThe second is to say that this is something we'll model in vocabulary and track in Mentat. Perhaps `:db/type :db.type/closed-enum`, and a way to write out the cases.\r\n\r\nIf you go this route you will need to validate in the transactor, record those enums before syncing values, decide whether the enum cases can shrink and/or grow… and **you still haven't solved the problem**, because if the enum set can't change backward-compatibly (and for cardinality it cannot), then the developer _still needs to bump the version_, or at least handle the case where Mentat complains that the remote timeline has a different enum set.\r\n\r\nUltimately you are bumping into the question of what to do when fundamental change occurs. My position is that we should support _indicating_, _migrating_, and _detecting_, but that we cannot transform all kinds of fundamental change into automatic change."}}
{"actor":{"display_login":"rnewman"}}