{"action":"created","issue":{"url":"https://api.github.com/repos/cosmir/openmic-2018/issues/21","repository_url":"https://api.github.com/repos/cosmir/openmic-2018","labels_url":"https://api.github.com/repos/cosmir/openmic-2018/issues/21/labels{/name}","comments_url":"https://api.github.com/repos/cosmir/openmic-2018/issues/21/comments","events_url":"https://api.github.com/repos/cosmir/openmic-2018/issues/21/events","html_url":"https://github.com/cosmir/openmic-2018/issues/21","id":354858618,"node_id":"MDU6SXNzdWUzNTQ4NTg2MTg=","number":21,"title":"Aggregated vs raw labels","user":{"login":"ejhumphrey","id":926379,"node_id":"MDQ6VXNlcjkyNjM3OQ==","avatar_url":"https://avatars0.githubusercontent.com/u/926379?v=4","gravatar_id":"","url":"https://api.github.com/users/ejhumphrey","html_url":"https://github.com/ejhumphrey","followers_url":"https://api.github.com/users/ejhumphrey/followers","following_url":"https://api.github.com/users/ejhumphrey/following{/other_user}","gists_url":"https://api.github.com/users/ejhumphrey/gists{/gist_id}","starred_url":"https://api.github.com/users/ejhumphrey/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ejhumphrey/subscriptions","organizations_url":"https://api.github.com/users/ejhumphrey/orgs","repos_url":"https://api.github.com/users/ejhumphrey/repos","events_url":"https://api.github.com/users/ejhumphrey/events{/privacy}","received_events_url":"https://api.github.com/users/ejhumphrey/received_events","type":"User","site_admin":false},"labels":[{"id":1026381051,"node_id":"MDU6TGFiZWwxMDI2MzgxMDUx","url":"https://api.github.com/repos/cosmir/openmic-2018/labels/question","name":"question","color":"d876e3","default":true}],"state":"open","locked":false,"assignee":{"login":"bmcfee","id":1190540,"node_id":"MDQ6VXNlcjExOTA1NDA=","avatar_url":"https://avatars3.githubusercontent.com/u/1190540?v=4","gravatar_id":"","url":"https://api.github.com/users/bmcfee","html_url":"https://github.com/bmcfee","followers_url":"https://api.github.com/users/bmcfee/followers","following_url":"https://api.github.com/users/bmcfee/following{/other_user}","gists_url":"https://api.github.com/users/bmcfee/gists{/gist_id}","starred_url":"https://api.github.com/users/bmcfee/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bmcfee/subscriptions","organizations_url":"https://api.github.com/users/bmcfee/orgs","repos_url":"https://api.github.com/users/bmcfee/repos","events_url":"https://api.github.com/users/bmcfee/events{/privacy}","received_events_url":"https://api.github.com/users/bmcfee/received_events","type":"User","site_admin":false},"assignees":[{"login":"bmcfee","id":1190540,"node_id":"MDQ6VXNlcjExOTA1NDA=","avatar_url":"https://avatars3.githubusercontent.com/u/1190540?v=4","gravatar_id":"","url":"https://api.github.com/users/bmcfee","html_url":"https://github.com/bmcfee","followers_url":"https://api.github.com/users/bmcfee/followers","following_url":"https://api.github.com/users/bmcfee/following{/other_user}","gists_url":"https://api.github.com/users/bmcfee/gists{/gist_id}","starred_url":"https://api.github.com/users/bmcfee/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bmcfee/subscriptions","organizations_url":"https://api.github.com/users/bmcfee/orgs","repos_url":"https://api.github.com/users/bmcfee/repos","events_url":"https://api.github.com/users/bmcfee/events{/privacy}","received_events_url":"https://api.github.com/users/bmcfee/received_events","type":"User","site_admin":false}],"milestone":null,"comments":2,"created_at":"2018-08-28T18:52:47Z","updated_at":"2018-08-29T11:46:44Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"I was just working through some dataset integrity checks, and noticed that the confidence / relevance score reported from CrowdFlower isn't _only_ a function of `num_responses`. \r\n\r\n![image](https://user-images.githubusercontent.com/926379/44741069-0f344680-aaca-11e8-99b3-f630d7f28251.png)\r\n\r\nLooking at the raw data, it could be a mix of trusted / untrusted judgments, e.g. when an annotator dropped below a confidence level and was removed from the job, or if confidence is weighted by `trust` (whether this is intra- or inter-task, I'm not sure). \r\n\r\nThis raises at least two questions:\r\n- Are we happy to proceed with the confidence / relevance scores provided by CF?\r\n- Is there value in _also_ sharing a more granular dataset of annotator responses? This table's columns could look like `[sample_key, annotator_id, trust, instrument, response]`"},"comment":{"url":"https://api.github.com/repos/cosmir/openmic-2018/issues/comments/416923830","html_url":"https://github.com/cosmir/openmic-2018/issues/21#issuecomment-416923830","issue_url":"https://api.github.com/repos/cosmir/openmic-2018/issues/21","id":416923830,"node_id":"MDEyOklzc3VlQ29tbWVudDQxNjkyMzgzMA==","user":{"login":"julian-urbano","id":6097978,"node_id":"MDQ6VXNlcjYwOTc5Nzg=","avatar_url":"https://avatars0.githubusercontent.com/u/6097978?v=4","gravatar_id":"","url":"https://api.github.com/users/julian-urbano","html_url":"https://github.com/julian-urbano","followers_url":"https://api.github.com/users/julian-urbano/followers","following_url":"https://api.github.com/users/julian-urbano/following{/other_user}","gists_url":"https://api.github.com/users/julian-urbano/gists{/gist_id}","starred_url":"https://api.github.com/users/julian-urbano/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/julian-urbano/subscriptions","organizations_url":"https://api.github.com/users/julian-urbano/orgs","repos_url":"https://api.github.com/users/julian-urbano/repos","events_url":"https://api.github.com/users/julian-urbano/events{/privacy}","received_events_url":"https://api.github.com/users/julian-urbano/received_events","type":"User","site_admin":false},"created_at":"2018-08-29T11:46:44Z","updated_at":"2018-08-29T11:46:44Z","author_association":"NONE","body":"Last time I used CF (maybe 2013?) they provided separate files for workers' info and for their answers to the units, like [this](https://github.com/julian-urbano/ismir2012-significance/tree/master/data). I'd fine it very useful to have all that if possible, but if not, what you proposed looks fine."}}
{"actor":{"display_login":"julian-urbano"}}