{"action":"created","issue":{"url":"https://api.github.com/repos/aodn/aodn-netcdf-tools/issues/14","repository_url":"https://api.github.com/repos/aodn/aodn-netcdf-tools","labels_url":"https://api.github.com/repos/aodn/aodn-netcdf-tools/issues/14/labels{/name}","comments_url":"https://api.github.com/repos/aodn/aodn-netcdf-tools/issues/14/comments","events_url":"https://api.github.com/repos/aodn/aodn-netcdf-tools/issues/14/events","html_url":"https://github.com/aodn/aodn-netcdf-tools/issues/14","id":368960718,"node_id":"MDU6SXNzdWUzNjg5NjA3MTg=","number":14,"title":"Appropriate validation at different stages of dataset creation","user":{"login":"mhidas","id":3579859,"node_id":"MDQ6VXNlcjM1Nzk4NTk=","avatar_url":"https://avatars1.githubusercontent.com/u/3579859?v=4","gravatar_id":"","url":"https://api.github.com/users/mhidas","html_url":"https://github.com/mhidas","followers_url":"https://api.github.com/users/mhidas/followers","following_url":"https://api.github.com/users/mhidas/following{/other_user}","gists_url":"https://api.github.com/users/mhidas/gists{/gist_id}","starred_url":"https://api.github.com/users/mhidas/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mhidas/subscriptions","organizations_url":"https://api.github.com/users/mhidas/orgs","repos_url":"https://api.github.com/users/mhidas/repos","events_url":"https://api.github.com/users/mhidas/events{/privacy}","received_events_url":"https://api.github.com/users/mhidas/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/aodn/aodn-netcdf-tools/milestones/1","html_url":"https://github.com/aodn/aodn-netcdf-tools/milestone/1","labels_url":"https://api.github.com/repos/aodn/aodn-netcdf-tools/milestones/1/labels","id":3729344,"node_id":"MDk6TWlsZXN0b25lMzcyOTM0NA==","number":1,"title":"v0.1","description":"Minimum work necessary build the package and start testing in RC","creator":{"login":"mhidas","id":3579859,"node_id":"MDQ6VXNlcjM1Nzk4NTk=","avatar_url":"https://avatars1.githubusercontent.com/u/3579859?v=4","gravatar_id":"","url":"https://api.github.com/users/mhidas","html_url":"https://github.com/mhidas","followers_url":"https://api.github.com/users/mhidas/followers","following_url":"https://api.github.com/users/mhidas/following{/other_user}","gists_url":"https://api.github.com/users/mhidas/gists{/gist_id}","starred_url":"https://api.github.com/users/mhidas/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mhidas/subscriptions","organizations_url":"https://api.github.com/users/mhidas/orgs","repos_url":"https://api.github.com/users/mhidas/repos","events_url":"https://api.github.com/users/mhidas/events{/privacy}","received_events_url":"https://api.github.com/users/mhidas/received_events","type":"User","site_admin":false},"open_issues":3,"closed_issues":4,"state":"open","created_at":"2018-10-11T06:17:22Z","updated_at":"2018-11-14T06:31:24Z","due_on":"2018-10-19T07:00:00Z","closed_at":null},"comments":2,"created_at":"2018-10-11T05:42:09Z","updated_at":"2018-11-15T01:05:54Z","closed_at":null,"author_association":"MEMBER","body":"There are different levels of validation we can apply at different stages:\r\n* **Schema validation**: making sure each individual dictionary (_dimensions, variables_ and _attributes_) has items that makes sense for its purpose. This is what's currently done with `jsonschema` at template  creation time. It can't be too restrictive because it may not make sense to fully specify all dataset elements in a JSON template (e.g. a generic template could specify the attributes of a TEMP variable, without saying what dimensions it should have).\r\n* **Schema completeness**: Similar to validation, but this level should insist on having _all_ the information required to create a valid netCDF4 object. This could be done using a more strict set of json schemas.\r\n* **Consistency**: This is to ensure that the three dictionaries are consistent with each other; in particular, that the dimensions of variables (and their data arrays) are consistent with what is specified in the _dimensions_ dictionary.\r\n\r\nI think validation can be applied at the beginning, but checking for completeness and consistency can be left until just before creating the file. The code is doing something close to this already, but  we need to make sure it's doing the right amount of validation at each stage."},"comment":{"url":"https://api.github.com/repos/aodn/aodn-netcdf-tools/issues/comments/438877417","html_url":"https://github.com/aodn/aodn-netcdf-tools/issues/14#issuecomment-438877417","issue_url":"https://api.github.com/repos/aodn/aodn-netcdf-tools/issues/14","id":438877417,"node_id":"MDEyOklzc3VlQ29tbWVudDQzODg3NzQxNw==","user":{"login":"mhidas","id":3579859,"node_id":"MDQ6VXNlcjM1Nzk4NTk=","avatar_url":"https://avatars1.githubusercontent.com/u/3579859?v=4","gravatar_id":"","url":"https://api.github.com/users/mhidas","html_url":"https://github.com/mhidas","followers_url":"https://api.github.com/users/mhidas/followers","following_url":"https://api.github.com/users/mhidas/following{/other_user}","gists_url":"https://api.github.com/users/mhidas/gists{/gist_id}","starred_url":"https://api.github.com/users/mhidas/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mhidas/subscriptions","organizations_url":"https://api.github.com/users/mhidas/orgs","repos_url":"https://api.github.com/users/mhidas/repos","events_url":"https://api.github.com/users/mhidas/events{/privacy}","received_events_url":"https://api.github.com/users/mhidas/received_events","type":"User","site_admin":false},"created_at":"2018-11-15T01:05:54Z","updated_at":"2018-11-15T01:05:54Z","author_association":"MEMBER","body":"Just to get it clear in my head before I start messing around in the code, here's what I'm thinking\r\n\r\n**At template creation time**, just validate the dictionaries (or JSON) provided against the schema we have already specified. Checks include\r\n* All attribute and variable names are valid; \r\n* Only _allowed_ (but at this stage not _required_) special keys are present (e.g. \"_dimensions\", \"_datatype\");\r\n* All specified attributes have valid types.\r\n\r\n**At netCDF writing time** (just before attempting to write the file)\r\n1. Validate the schema again, as above;\r\n1. Ensure that _all_ the necessary special keys are present, automatically filling in missing information if possible (e.g. set \"_datatype\" attribute from type of the array assgined to \"_data\").\r\n1. Ensure consistency\r\n    * between \"_dimensions\" of variables and those defined for the dataset (auto update their sizes from the data arrays if not set in template); and\r\n    * between \"_datatype\"  and type of \"_data\" for each variable."}}
{"actor":{"display_login":"mhidas"}}