{"action":"created","issue":{"url":"https://api.github.com/repos/frictionlessdata/specs/issues/349","repository_url":"https://api.github.com/repos/frictionlessdata/specs","labels_url":"https://api.github.com/repos/frictionlessdata/specs/issues/349/labels{/name}","comments_url":"https://api.github.com/repos/frictionlessdata/specs/issues/349/comments","events_url":"https://api.github.com/repos/frictionlessdata/specs/issues/349/events","html_url":"https://github.com/frictionlessdata/specs/issues/349","id":196995908,"number":349,"title":"Normalized and Extended Forms for Specs","user":{"login":"rufuspollock","id":180658,"avatar_url":"https://avatars.githubusercontent.com/u/180658?v=3","gravatar_id":"","url":"https://api.github.com/users/rufuspollock","html_url":"https://github.com/rufuspollock","followers_url":"https://api.github.com/users/rufuspollock/followers","following_url":"https://api.github.com/users/rufuspollock/following{/other_user}","gists_url":"https://api.github.com/users/rufuspollock/gists{/gist_id}","starred_url":"https://api.github.com/users/rufuspollock/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rufuspollock/subscriptions","organizations_url":"https://api.github.com/users/rufuspollock/orgs","repos_url":"https://api.github.com/users/rufuspollock/repos","events_url":"https://api.github.com/users/rufuspollock/events{/privacy}","received_events_url":"https://api.github.com/users/rufuspollock/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2016-12-21T17:59:37Z","updated_at":"2016-12-22T01:32:50Z","closed_at":null,"body":"## Problem\r\n\r\nWe have a fair amount of \"optionality\" for properties in our specs e.g.\r\n\r\n* License can be a string **or** an object\r\n* Author **or** contributors\r\n* Author can be string **or** object \r\n* Resource path can be a single string **or** an array of strings\r\n* Resource path can be an absolute url **or** a relative path / url\r\n\r\nThis adds both cognitive and implementation complexity especially for consumers of data packages.\r\n\r\nIt is also a major blocker for statically typed languages and e.g. protocol buffers.\r\n\r\nAt the same time, the optionality is there to make it easier for publishers to create data packages, especially if creating by hand (but how often does this happen?).\r\n\r\n## Suggestion\r\n\r\nIntroduce concept of:\r\n\r\n* Normalized form: spec in form with all optionality removed e.g. author is part of contributors, license is an object, contributors are objects, resource path is an array etc\r\n* Extended form: version of spec where optionality is present.\r\n\r\nSpecs focus on **normalized form**.\r\n\r\nExtended form is an appendix (or even separate item). Tools can state whether they support extended form. Tools can also be explicit about the step of converting extended to normal form.\r\n\r\n**Aside**: we can also eliminate some of this optionality by simply removing it entirely the specs. We should consider this - and already are in e.g. #236 for licenses. Merging author into contributors (and maybe adding \"role\" as a property to contributors) seems another obvious item.\r\n\r\n/cc @Stiivi @pwalsh "},"comment":{"url":"https://api.github.com/repos/frictionlessdata/specs/issues/comments/268695096","html_url":"https://github.com/frictionlessdata/specs/issues/349#issuecomment-268695096","issue_url":"https://api.github.com/repos/frictionlessdata/specs/issues/349","id":268695096,"user":{"login":"Stiivi","id":71105,"avatar_url":"https://avatars.githubusercontent.com/u/71105?v=3","gravatar_id":"","url":"https://api.github.com/users/Stiivi","html_url":"https://github.com/Stiivi","followers_url":"https://api.github.com/users/Stiivi/followers","following_url":"https://api.github.com/users/Stiivi/following{/other_user}","gists_url":"https://api.github.com/users/Stiivi/gists{/gist_id}","starred_url":"https://api.github.com/users/Stiivi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Stiivi/subscriptions","organizations_url":"https://api.github.com/users/Stiivi/orgs","repos_url":"https://api.github.com/users/Stiivi/repos","events_url":"https://api.github.com/users/Stiivi/events{/privacy}","received_events_url":"https://api.github.com/users/Stiivi/received_events","type":"User","site_admin":false},"created_at":"2016-12-22T01:32:50Z","updated_at":"2016-12-22T01:32:50Z","body":"> Strings as a shorthand for objects\r\n> One pattern in the specs is strings as shorthand for objects. Author is the most obvious example of this pattern. This is probably a direct result of the npm influence at some stage in the history of the specs and the early implementations in javascript.\r\n> If we were writing these specs from scratch, I doubt we'd have this pattern, and instead just have objects in these cases. I'm happy to move to this for v1.\r\n\r\nIn this case we have the best opportunity to not to support an anti-pattern and do it better. In dynamically typed languages it adds unnecessary manual type checking, which might be forgotten by spec implementors or might serve as a place for potential error.\r\n\r\nIn statically typed languages it adds a complication by needing a sum-type to contain the value or to do unnecessary conversion.\r\n\r\n> 2/ URIs as pointers, or directly inlining a payload\r\n> schema is the most obvious example - provide a pointer to the thing, or the thing itself. I think this is a good and suitable pattern for our use cases.\r\n\r\nNot normalized data – another place for potential error. Which one is to be considered/has priority? Why?\r\n\r\nSee my note below about normalization.\r\n\r\n> 3/ URIs as a single path property, instead of url and path as distinct properties with distinct meanings\r\n> path as any URI is a really simple pattern for publishers, and it is common enough that implementors can negotiate different URIs and then handle accordingly.\r\n> I think those 4 cases are the ones we have for cases where implementors need to know the type for a given property, before handling the property.\r\n\r\nAgain – very unnecessary sum-type and is non-trivial for implementors. 4 cases is a lot - 4 here, two there ... it adds up.\r\n\r\n\"Implementors need to know\"... they don't. Type-checking of this kind in code is also unnecessary and it is another opportunity for introducing errors (similar to the previous one). In this case I need to check for 4 types of one property and test it.\r\n\r\n> I always want to optimise for publishers. I think it is part of our scrappy mode of operation, and I think it is why we are getting (small scale, true) adoption even when the specs have been far from stable until now. But I'd like to see specific examples of how we are creating difficulty for implementors here. \r\n\r\nIt is great to have optimisation for publishers, however if this is a specification for something to be consumed, then attention should be paid for the consumers, as they need to _correctly_ consume the data. It is way much easier to fix/change publishers, as they are at the end of pipelines. The opposite is true about the consumers - they are at the very beginning of the pipelines and might already have a lot dependencies (that might even assume incorrect data and when they get correct data they might cause issues) and potential workarounds.\r\n\r\nIf one wants to satisfy \"human data publishers\", then we should have a DSL (preferably some kind of SQL-like DML) or even better: an import from more convenient form, such as a spreadsheet (which is the most common tool of business analysts - people filling similar specs like this one, if I can oversimplify). JSON is terrible form for this purpose. That JSON is write-able and editable is same illusion as XML being editable.\r\n\r\n> We are implementing ourselves in Python and Javascript, and we have had 3rd parties implementing in Ruby, R, and Java in the last 6 months. I've heard various useful comments, mostly around ambiguity in the specs, but none to date around fundamental difficulty to implement code to handle the specifications.\r\n\r\nThere are lots of examples with difficulties for implementors: non-normalized model, complex data types resulting in manual type checking. It is not about _not being able_ to implement the spec, it is rather about not being able to implement it _correctly_, _reliably_ and with assurance that the data consumed is going to be correct.\r\n\r\n> Can we have some concrete examples of things are that not fun/very brittle in a statically typed language (presumably due to the strong influence of javascript and python in the way we write and implement the specs)?\r\n\r\nJavascript is inconsistent language \"by design\" (just look at the weird implicit casting rules). I would not get any inspiration from JS for projects where I need to assure quality. I consider all data projects to require quality assurance. Python is dynamically typed language – if one wants to assure quality, one has to do work and write tests that basically say \"it compiles\". We are writing specification which describes how a structure and content of data looks like prior to the data processing yet we get inspiration from languages that do exactly the opposite – determine types as they go.\r\n\r\nIn this case what is brittle is the dynamicaly typed language. Statically typed language here will do fine, just have way much more code that might be avoided. However, the following still applies no matter which language you use: more code = more maintenance, reduced readability, increased complexity → more places for errors. So we are basically negatively affecting both types of languages here: dynamic by requiring type checking, static by requiring more custom data types.\r\n\r\n> Normalized and extended form: I'm a -1 on that. We need one tight form. Let's drop what we can drop right now, and queue up some other points for post v1 refinements.\r\n\r\nNormalization is very basic engineering practice and I do not think that we have to discuss whether to apply this practice at something as important as specification for data exchange format. We are talking about data and we expect certain quality of the data to be useable, yet we don't even assure quality of the metadata describing the data?\r\n\r\nNot normalized metadata also goes against data provenance. Data provenance is a hard problem to be solved in the first place, now we add even more problems to it. Makes diagnosis difficult.\r\n\r\nIf the basic practices are not followed at the very lowest level of a stack, which in this case is a data infrastructure, it makes the whole stack susceptible to errors with many layers where the errors can be amplified – intentional or not. The errors will be non-trivial to remove in complex, established data environments.\r\n\r\nThere are things that can't be done post-v1, especially those that go against core engineering practices. They should be polished as soon as possible. Otherwise we are risking that they will never be removed due to established dependencies in systems using the spec. I can change the stack on my machine, for my own purposes, but I can not change stack in a large organization with heterogenous ecosystem with lots of daily (or more frequent) recurrent loadings.\r\n\r\n# Summary\r\n\r\nMy main concern is about data quality and data issue diagnostic. The more branches/control flow paths we can remove, the better – the less places for errors to happen, the less places to check, the less places to think about. It is very important that the lowest levels of the stack are correct and are easy to verify that they are correct.  \r\n\r\nIf I can't trust that the consumer implemented the specification correctly, I can't trust data. If I can't trust the data, there is no point to work with that data at all.\r\n\r\nIf one wants to use the data provided by the specification in a serious data environment, they have to convert it into simplified and normalized form anyway to assure data quality. Question is, whether the spec is for toy data projects or for serious consumption from which trustworthy insights can be derived?"}}
{"actor":{"display_login":"Stiivi"}}