{"action":"created","issue":{"url":"https://api.github.com/repos/openimages/dataset/issues/8","repository_url":"https://api.github.com/repos/openimages/dataset","labels_url":"https://api.github.com/repos/openimages/dataset/issues/8/labels{/name}","comments_url":"https://api.github.com/repos/openimages/dataset/issues/8/comments","events_url":"https://api.github.com/repos/openimages/dataset/issues/8/events","html_url":"https://github.com/openimages/dataset/issues/8","id":182436020,"number":8,"title":"Question about Human image-level annotations (Validation Set)","user":{"login":"pengpaiSH","id":3724085,"avatar_url":"https://avatars.githubusercontent.com/u/3724085?v=3","gravatar_id":"","url":"https://api.github.com/users/pengpaiSH","html_url":"https://github.com/pengpaiSH","followers_url":"https://api.github.com/users/pengpaiSH/followers","following_url":"https://api.github.com/users/pengpaiSH/following{/other_user}","gists_url":"https://api.github.com/users/pengpaiSH/gists{/gist_id}","starred_url":"https://api.github.com/users/pengpaiSH/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pengpaiSH/subscriptions","organizations_url":"https://api.github.com/users/pengpaiSH/orgs","repos_url":"https://api.github.com/users/pengpaiSH/repos","events_url":"https://api.github.com/users/pengpaiSH/events{/privacy}","received_events_url":"https://api.github.com/users/pengpaiSH/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2016-10-12T05:05:04Z","updated_at":"2016-10-12T06:39:42Z","closed_at":null,"body":"@gkrasin Thank you for publishing such awesome Open Images Dataset. In the Google research bog, it was mentioned that \"_For the validation set, we had human raters verify these automated labels to find and remove false positives._\" And you also provided a 9MB file \"Human image-level annotations (validation set)\". It is interesting to know will human raters annotate the labels that your previous vision model missed to predict? In other words, the human label file will certainly reduce number of labels (**due to removing false positives**). Meanwhile, will it increase the number of labels (those **missing ones** that the vision model used to predict). Thank you for you attention and looking forward to your reply!"},"comment":{"url":"https://api.github.com/repos/openimages/dataset/issues/comments/253131903","html_url":"https://github.com/openimages/dataset/issues/8#issuecomment-253131903","issue_url":"https://api.github.com/repos/openimages/dataset/issues/8","id":253131903,"user":{"login":"pengpaiSH","id":3724085,"avatar_url":"https://avatars.githubusercontent.com/u/3724085?v=3","gravatar_id":"","url":"https://api.github.com/users/pengpaiSH","html_url":"https://github.com/pengpaiSH","followers_url":"https://api.github.com/users/pengpaiSH/followers","following_url":"https://api.github.com/users/pengpaiSH/following{/other_user}","gists_url":"https://api.github.com/users/pengpaiSH/gists{/gist_id}","starred_url":"https://api.github.com/users/pengpaiSH/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pengpaiSH/subscriptions","organizations_url":"https://api.github.com/users/pengpaiSH/orgs","repos_url":"https://api.github.com/users/pengpaiSH/repos","events_url":"https://api.github.com/users/pengpaiSH/events{/privacy}","received_events_url":"https://api.github.com/users/pengpaiSH/received_events","type":"User","site_admin":false},"created_at":"2016-10-12T06:39:42Z","updated_at":"2016-10-12T06:39:42Z","body":"@gkrasin Thank you for your quick response! I think you might miss my points. Let me give a toy example. Say for an image **I** with ground-truth labels \"_dog, girl, beach, sea, sky_\" (as you may imagine the beautiful scene), the **visual classifier** (as you mentioned above) predicts the labels: \"dog, **boy**, beach, sea, sky\" and with each unnormalized score \"0.9, **0.2**, 0.9, 0.9, 0.9\". Then, according to you above suggestion, the human rater will receive an image-label pair (**I**, boy) and the rater will justify it was a **false-positive** prediction by classifier, right? Thus, the label list now becomes \"dog, beach, sea, sky\" (without \"boy\", removed by rater). Now my question comes: will the rater add \"girl\" to the ultimate label file published to us? Or, the final label for this image is \"dog, beach, sea, sky\", even verified by human raters as claimed?"}}
{"actor":{"display_login":"pengpaiSH"}}