{"action":"created","issue":{"url":"https://api.github.com/repos/Netflix/vmaf/issues/222","repository_url":"https://api.github.com/repos/Netflix/vmaf","labels_url":"https://api.github.com/repos/Netflix/vmaf/issues/222/labels{/name}","comments_url":"https://api.github.com/repos/Netflix/vmaf/issues/222/comments","events_url":"https://api.github.com/repos/Netflix/vmaf/issues/222/events","html_url":"https://github.com/Netflix/vmaf/issues/222","id":356354356,"node_id":"MDU6SXNzdWUzNTYzNTQzNTY=","number":222,"title":"How to reject subjective score?","user":{"login":"liuxj901103","id":40554507,"node_id":"MDQ6VXNlcjQwNTU0NTA3","avatar_url":"https://avatars1.githubusercontent.com/u/40554507?v=4","gravatar_id":"","url":"https://api.github.com/users/liuxj901103","html_url":"https://github.com/liuxj901103","followers_url":"https://api.github.com/users/liuxj901103/followers","following_url":"https://api.github.com/users/liuxj901103/following{/other_user}","gists_url":"https://api.github.com/users/liuxj901103/gists{/gist_id}","starred_url":"https://api.github.com/users/liuxj901103/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/liuxj901103/subscriptions","organizations_url":"https://api.github.com/users/liuxj901103/orgs","repos_url":"https://api.github.com/users/liuxj901103/repos","events_url":"https://api.github.com/users/liuxj901103/events{/privacy}","received_events_url":"https://api.github.com/users/liuxj901103/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-09-03T04:27:50Z","updated_at":"2018-09-03T06:32:03Z","closed_at":null,"author_association":"NONE","body":"Hi, I have got some raw subjective scores by my subjective tests , but some of the scores were nosiy. With the algorithm of subjective rejection(SR_DMOS), I could not reject any observer. Also I Tried MLE, which could get Subject bias, Subject inconsistency, Content ambiguity,Outliers. I could not figuer out what was the meaning of the above parameters.  I still don't know how to recover quality scores from noisy scores. Thanks! @pablete @jspears @moustaki @zkwentz @fabiokung "},"comment":{"url":"https://api.github.com/repos/Netflix/vmaf/issues/comments/418013655","html_url":"https://github.com/Netflix/vmaf/issues/222#issuecomment-418013655","issue_url":"https://api.github.com/repos/Netflix/vmaf/issues/222","id":418013655,"node_id":"MDEyOklzc3VlQ29tbWVudDQxODAxMzY1NQ==","user":{"login":"slhck","id":582444,"node_id":"MDQ6VXNlcjU4MjQ0NA==","avatar_url":"https://avatars3.githubusercontent.com/u/582444?v=4","gravatar_id":"","url":"https://api.github.com/users/slhck","html_url":"https://github.com/slhck","followers_url":"https://api.github.com/users/slhck/followers","following_url":"https://api.github.com/users/slhck/following{/other_user}","gists_url":"https://api.github.com/users/slhck/gists{/gist_id}","starred_url":"https://api.github.com/users/slhck/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/slhck/subscriptions","organizations_url":"https://api.github.com/users/slhck/orgs","repos_url":"https://api.github.com/users/slhck/repos","events_url":"https://api.github.com/users/slhck/events{/privacy}","received_events_url":"https://api.github.com/users/slhck/received_events","type":"User","site_admin":false},"created_at":"2018-09-03T06:32:03Z","updated_at":"2018-09-03T06:32:03Z","author_association":"CONTRIBUTOR","body":"> but some of the scores were nosiy\r\n\r\nHow do you know that some of your scores were noisy?\r\n\r\n> I could not figuer out what was the meaning of the above parameters\r\n\r\nYou can find an explanation [in the paper](https://github.com/Netflix/sureal/blob/master/resource/doc/dcc17v3.pdf):\r\n\r\n> - Subject bias. The notion of quality is highly subjective and test subjects are entitled to rate the videos in their own opinions. For example, more picky viewers tend to be biased toward lower scores, and vice versa. Also, not every subject has “golden eyes” – their sensitivity to impairments varies.\r\n> - Subject inconsistency. Subjective testing is a laborious process, and not every viewer can maintain attentiveness throughout. Some tend to rate more consistently than others.\r\n> - Content ambiguity. Some contents tend to be more difficult to be rated than others. For example, water surface with ripples in the dark is more ambiguous than a bright blue sky.\r\n> - Outliers. Last but not least, some raw scores are simply outliers – viewers may just not pay attention. Software issues may also render scores meaningless\r\n\r\n> I still don't know how to recover quality scores from noisy scores\r\n\r\nRunning the sureal software should do exactly that.\r\n\r\nIf you want to be strict about subject rejection, you could also simply look at the correlations between individual ratings and mean opinion scores and remove all subjects under a given  threshold."}}
{"actor":{"display_login":"slhck"}}