{"action":"created","issue":{"url":"https://api.github.com/repos/frictionlessdata/data-quality-cli/issues/13","repository_url":"https://api.github.com/repos/frictionlessdata/data-quality-cli","labels_url":"https://api.github.com/repos/frictionlessdata/data-quality-cli/issues/13/labels{/name}","comments_url":"https://api.github.com/repos/frictionlessdata/data-quality-cli/issues/13/comments","events_url":"https://api.github.com/repos/frictionlessdata/data-quality-cli/issues/13/events","html_url":"https://github.com/frictionlessdata/data-quality-cli/issues/13","id":160292384,"number":13,"title":"Refactoring \"timeliness\" as a dimension for assessment","user":{"login":"pwalsh","id":791678,"avatar_url":"https://avatars.githubusercontent.com/u/791678?v=3","gravatar_id":"","url":"https://api.github.com/users/pwalsh","html_url":"https://github.com/pwalsh","followers_url":"https://api.github.com/users/pwalsh/followers","following_url":"https://api.github.com/users/pwalsh/following{/other_user}","gists_url":"https://api.github.com/users/pwalsh/gists{/gist_id}","starred_url":"https://api.github.com/users/pwalsh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pwalsh/subscriptions","organizations_url":"https://api.github.com/users/pwalsh/orgs","repos_url":"https://api.github.com/users/pwalsh/repos","events_url":"https://api.github.com/users/pwalsh/events{/privacy}","received_events_url":"https://api.github.com/users/pwalsh/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/frictionlessdata/data-quality-cli/labels/question","name":"question","color":"cc317c"}],"state":"open","locked":false,"assignee":{"login":"georgiana-b","id":8862002,"avatar_url":"https://avatars.githubusercontent.com/u/8862002?v=3","gravatar_id":"","url":"https://api.github.com/users/georgiana-b","html_url":"https://github.com/georgiana-b","followers_url":"https://api.github.com/users/georgiana-b/followers","following_url":"https://api.github.com/users/georgiana-b/following{/other_user}","gists_url":"https://api.github.com/users/georgiana-b/gists{/gist_id}","starred_url":"https://api.github.com/users/georgiana-b/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/georgiana-b/subscriptions","organizations_url":"https://api.github.com/users/georgiana-b/orgs","repos_url":"https://api.github.com/users/georgiana-b/repos","events_url":"https://api.github.com/users/georgiana-b/events{/privacy}","received_events_url":"https://api.github.com/users/georgiana-b/received_events","type":"User","site_admin":false},"assignees":[{"login":"georgiana-b","id":8862002,"avatar_url":"https://avatars.githubusercontent.com/u/8862002?v=3","gravatar_id":"","url":"https://api.github.com/users/georgiana-b","html_url":"https://github.com/georgiana-b","followers_url":"https://api.github.com/users/georgiana-b/followers","following_url":"https://api.github.com/users/georgiana-b/following{/other_user}","gists_url":"https://api.github.com/users/georgiana-b/gists{/gist_id}","starred_url":"https://api.github.com/users/georgiana-b/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/georgiana-b/subscriptions","organizations_url":"https://api.github.com/users/georgiana-b/orgs","repos_url":"https://api.github.com/users/georgiana-b/repos","events_url":"https://api.github.com/users/georgiana-b/events{/privacy}","received_events_url":"https://api.github.com/users/georgiana-b/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/frictionlessdata/data-quality-cli/milestones/5","html_url":"https://github.com/frictionlessdata/data-quality-cli/milestones/Current","labels_url":"https://api.github.com/repos/frictionlessdata/data-quality-cli/milestones/5/labels","id":1076990,"number":5,"title":"Current","description":"","creator":{"login":"pwalsh","id":791678,"avatar_url":"https://avatars.githubusercontent.com/u/791678?v=3","gravatar_id":"","url":"https://api.github.com/users/pwalsh","html_url":"https://github.com/pwalsh","followers_url":"https://api.github.com/users/pwalsh/followers","following_url":"https://api.github.com/users/pwalsh/following{/other_user}","gists_url":"https://api.github.com/users/pwalsh/gists{/gist_id}","starred_url":"https://api.github.com/users/pwalsh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pwalsh/subscriptions","organizations_url":"https://api.github.com/users/pwalsh/orgs","repos_url":"https://api.github.com/users/pwalsh/repos","events_url":"https://api.github.com/users/pwalsh/events{/privacy}","received_events_url":"https://api.github.com/users/pwalsh/received_events","type":"User","site_admin":false},"open_issues":3,"closed_issues":8,"state":"open","created_at":"2015-04-21T12:09:07Z","updated_at":"2016-07-12T09:15:28Z","due_on":"2016-07-16T21:00:00Z","closed_at":null},"comments":2,"created_at":"2016-06-14T21:49:02Z","updated_at":"2016-07-12T23:03:39Z","closed_at":null,"body":"# Description\r\n\r\nTimeliness of publication, along with Structure and Schema of the source files, is a dimension used in the assessment of data quality in the first dashboard we've developed for UK25k spend data https://github.com/okfn/data-quality-uk-25k-spend\r\n\r\nHowever, timeliness is really hard to a) detect from the data available for a given data file, and b) to generalise for use over a range of different datasets we'd want to assess for quality.\r\n\r\nUsing the ideas we developed for the uk25k spend data, we need to address two points:\r\n\r\n- Configuration of \"timeliness\" as a dimension for assessment\r\n- Strategies for detecting \"timely publication\" for a given file\r\n\r\n# Tasks\r\n\r\n- [ ] Timeliness configuration (on `dq.json`)\r\n  - [ ] Assess timeliness [ `assess_timeliness: False` ]: boolean. Whether to include timeliness as a dimension of the data quality assessment. Default to false.\r\n  - [ ] Timeliness period [ `timeliness_period`: 1 ]: integer. The period of time in which a file publication can be considered timely. In months. Default 1.\r\n  - [ ] Timeliness strategy [ `timeliness_strategy` ]: 'parse_filename'. The strategy to use to detect the period the data file applies to. One of `parse_filename`, `parse_property` (still TBD in detail)\r\n    - [ ] Implement the parse filename strategy which should be able to be extracted from the uk25k spend codebase\r\n- [ ] Apply the changes here to the uk25k spend data\r\n- [ ] Tests"},"comment":{"url":"https://api.github.com/repos/frictionlessdata/data-quality-cli/issues/comments/232208051","html_url":"https://github.com/frictionlessdata/data-quality-cli/issues/13#issuecomment-232208051","issue_url":"https://api.github.com/repos/frictionlessdata/data-quality-cli/issues/13","id":232208051,"user":{"login":"georgiana-b","id":8862002,"avatar_url":"https://avatars.githubusercontent.com/u/8862002?v=3","gravatar_id":"","url":"https://api.github.com/users/georgiana-b","html_url":"https://github.com/georgiana-b","followers_url":"https://api.github.com/users/georgiana-b/followers","following_url":"https://api.github.com/users/georgiana-b/following{/other_user}","gists_url":"https://api.github.com/users/georgiana-b/gists{/gist_id}","starred_url":"https://api.github.com/users/georgiana-b/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/georgiana-b/subscriptions","organizations_url":"https://api.github.com/users/georgiana-b/orgs","repos_url":"https://api.github.com/users/georgiana-b/repos","events_url":"https://api.github.com/users/georgiana-b/events{/privacy}","received_events_url":"https://api.github.com/users/georgiana-b/received_events","type":"User","site_admin":false},"created_at":"2016-07-12T23:03:39Z","updated_at":"2016-07-12T23:03:39Z","body":"I agree with the workflow described in your checklist, except for `timeliness_strategy`. As far as I've seen in CKAN, most of the times there are multiple fields that can contain `period_id`. In `data-quality-uk-spend` [we use `title` (+ `description`) and `data`](https://github.com/okfn/data-quality-uk-25k-spend/blob/master/scripts/id_data.py#L240). So I think a safer way to go is have the `timeliness_strategy` be an array of multiple fields that should be analyzed, in order of priority. \r\nFor example `timeliness_strategy: ['title', 'data']`  would mean:  \r\n\r\n* Parse  `title` \r\n    * if you find something use it and stop\r\n    * otherwise move to `data`.  \r\n* If no `period_id` is found in any of the fields, I think we should use the strategy you described before:\r\n    * Count the ones that don't have `period_id` and use `created` instead\r\n    * If sources with empty `period_id` make 10% or more of the total:\r\n\r\n        a. Tell the user about it, and ask if they want to continue anyway in a prompt and continue/stop based on the input.\r\n        b. Raise and error asking them to make `assess_timeliness: False` because it's unreliable or add other fields that could contain it. \r\n    * If they don't, move on\r\n\r\nFor the \"move on\" part, it's unclear to me if I should incorporate this in the current scoring algorithm or just set up  the functionality described above and move on to #5. If I should incorporate it in the current one, how would it influence the score?"}}
{"actor":{"display_login":"georgiana-b"}}