{"action":"created","issue":{"url":"https://api.github.com/repos/openaq/openaq-data-format/issues/16","repository_url":"https://api.github.com/repos/openaq/openaq-data-format","labels_url":"https://api.github.com/repos/openaq/openaq-data-format/issues/16/labels{/name}","comments_url":"https://api.github.com/repos/openaq/openaq-data-format/issues/16/comments","events_url":"https://api.github.com/repos/openaq/openaq-data-format/issues/16/events","html_url":"https://github.com/openaq/openaq-data-format/issues/16","id":187704046,"number":16,"title":"Data format to differentiate real-time captured-data and QA/QC'ed.","user":{"login":"RocketD0g","id":13404290,"avatar_url":"https://avatars.githubusercontent.com/u/13404290?v=3","gravatar_id":"","url":"https://api.github.com/users/RocketD0g","html_url":"https://github.com/RocketD0g","followers_url":"https://api.github.com/users/RocketD0g/followers","following_url":"https://api.github.com/users/RocketD0g/following{/other_user}","gists_url":"https://api.github.com/users/RocketD0g/gists{/gist_id}","starred_url":"https://api.github.com/users/RocketD0g/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/RocketD0g/subscriptions","organizations_url":"https://api.github.com/users/RocketD0g/orgs","repos_url":"https://api.github.com/users/RocketD0g/repos","events_url":"https://api.github.com/users/RocketD0g/events{/privacy}","received_events_url":"https://api.github.com/users/RocketD0g/received_events","type":"User","site_admin":false},"labels":[{"id":278165651,"url":"https://api.github.com/repos/openaq/openaq-data-format/labels/help%20wanted","name":"help wanted","color":"159818","default":true},{"id":337743267,"url":"https://api.github.com/repos/openaq/openaq-data-format/labels/v2","name":"v2","color":"0052cc","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2016-11-07T12:51:53Z","updated_at":"2016-11-07T12:59:38Z","closed_at":null,"body":"Below, @masalmon mentioned differences she noticed between real-time captured US Emb data from Delhi and data reported on the Emb site later. OpenAQ currently *only* captures real-time data for government sources, and does not allow the insertion of a QA/QC layer. \r\n\r\nFrom @masalmon: \r\n\r\n> Sarath asked me to do a graph comparing Beijing and Delhi and for that I decided to use data from the embassy that is not in OpenAQ, for having a whole year for Delhi. I use my usaqmindia Github repo for that. It was a great occasion to compare both datasets for the common times: OpenAQ vs embassy data. So it's interesting to notice that there are differences. They seem to be: sometimes OpenAQ has -999 while the dataset from the embassy website has more credible values. \r\n> \r\n> \r\n> ![rplot](https://cloud.githubusercontent.com/assets/13404290/20058341/7e42824e-a4be-11e6-8471-ab0f21038613.png)\r\n> \r\n> \r\n> Here you see the negative measures, I'll make a summary of some sort for the repeated measures.\r\n> \r\n> So  a small summary of the issue. I'm looking at 5463 non missing measures between 2015-12-12 03:00:00 and 2016-08-01 00:00:00. 145 are different between the embassy website and OpenAQ. So not a lot but still interesting, do you want to add the \"right\" value with a corresponding flag as suggested by @christa (real time vs not)? (edited)\r\n> \r\n> \r\n> I've just looked at the dates at which non concordant values appear, and it's not at a given period, last time was in July this year."},"comment":{"url":"https://api.github.com/repos/openaq/openaq-data-format/issues/comments/258829215","html_url":"https://github.com/openaq/openaq-data-format/issues/16#issuecomment-258829215","issue_url":"https://api.github.com/repos/openaq/openaq-data-format/issues/16","id":258829215,"user":{"login":"RocketD0g","id":13404290,"avatar_url":"https://avatars.githubusercontent.com/u/13404290?v=3","gravatar_id":"","url":"https://api.github.com/users/RocketD0g","html_url":"https://github.com/RocketD0g","followers_url":"https://api.github.com/users/RocketD0g/followers","following_url":"https://api.github.com/users/RocketD0g/following{/other_user}","gists_url":"https://api.github.com/users/RocketD0g/gists{/gist_id}","starred_url":"https://api.github.com/users/RocketD0g/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/RocketD0g/subscriptions","organizations_url":"https://api.github.com/users/RocketD0g/orgs","repos_url":"https://api.github.com/users/RocketD0g/repos","events_url":"https://api.github.com/users/RocketD0g/events{/privacy}","received_events_url":"https://api.github.com/users/RocketD0g/received_events","type":"User","site_admin":false},"created_at":"2016-11-07T12:59:37Z","updated_at":"2016-11-07T12:59:37Z","body":"This is something we would like to do in the future - have a QA/QC'ed layer in addition to the real-time gov layer. Current constraints: (1) Time it would take to re-format existing data points in system, and (2) Financial: this could, in theory, greatly expand our database with another layer of data (from sources/locations) already in our system. There were would be significant costs associated with this if we were to duplicate existing databases that have this information. One question might be: in the mean time, rather than to duplicate these databases (again due to time + financial constraints), how do we systematically point people do these databases, in addition to tagging the source? \r\n\r\nOn the specific differences @masalmon noticed: it's odd to me that '-999' got taken out and replaced with other numbers. '-999' is the symbol for 'my instrument isn't reporting data.' Measurements may drift systematically, meaning QA/QC'ed numbers may be adjusted, but I don't see how '-999' numbers are adjusted. Perhaps the instrument itself was truly measuring numbers but the measurements weren't making it out or something?\r\n\r\ncc: @jflasher @masalmon"}}
{"actor":{"display_login":"RocketD0g"}}