{"action":"opened","issue":{"url":"https://api.github.com/repos/jpauwels/MusOOEvaluator/issues/6","repository_url":"https://api.github.com/repos/jpauwels/MusOOEvaluator","labels_url":"https://api.github.com/repos/jpauwels/MusOOEvaluator/issues/6/labels{/name}","comments_url":"https://api.github.com/repos/jpauwels/MusOOEvaluator/issues/6/comments","events_url":"https://api.github.com/repos/jpauwels/MusOOEvaluator/issues/6/events","html_url":"https://github.com/jpauwels/MusOOEvaluator/issues/6","id":304112277,"number":6,"title":"Are my WCSR scores biased when assuming that estimated start and end times are the same as the ground truth?","user":{"login":"CKKOY","id":13520810,"avatar_url":"https://avatars2.githubusercontent.com/u/13520810?v=4","gravatar_id":"","url":"https://api.github.com/users/CKKOY","html_url":"https://github.com/CKKOY","followers_url":"https://api.github.com/users/CKKOY/followers","following_url":"https://api.github.com/users/CKKOY/following{/other_user}","gists_url":"https://api.github.com/users/CKKOY/gists{/gist_id}","starred_url":"https://api.github.com/users/CKKOY/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/CKKOY/subscriptions","organizations_url":"https://api.github.com/users/CKKOY/orgs","repos_url":"https://api.github.com/users/CKKOY/repos","events_url":"https://api.github.com/users/CKKOY/events{/privacy}","received_events_url":"https://api.github.com/users/CKKOY/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-03-10T22:45:37Z","updated_at":"2018-03-10T22:45:37Z","closed_at":null,"author_association":"NONE","body":"Hi, @jpauwels. This is to ask for clarification about the way evaluation works when assuming that the estimated start and end times are the same as the ground truth (gt) start and end times.\r\n\r\nI'd like to compare my WCSR scores to [Junqi Deng's](https://github.com/tangkk/phd-thesis-junqi-deng/blob/master/junqi-thesis-hku.pdf). (In his experiments, he _predicts_ start and end times and doesn't make the assumption that I'm making). \r\n\r\nWCSR is defined as follows:\r\n![tmp_wcsr](https://user-images.githubusercontent.com/13520810/37247265-8774ca20-24b0-11e8-8dba-6162b94a023c.PNG), \r\nwhere CSR is defined as follows: \r\n![tmp_csr](https://user-images.githubusercontent.com/13520810/37247269-9c40899e-24b0-11e8-85a8-bc93f1a5442e.PNG). \r\nIn his [thesis](https://github.com/tangkk/phd-thesis-junqi-deng/blob/master/junqi-thesis-hku.pdf), Junqi gives a verbal definition of CSR as: \r\n![tmp_csr_in_words](https://user-images.githubusercontent.com/13520810/37247273-b04841ca-24b0-11e8-9d41-2978683bca5f.PNG)\r\n\r\nLet's assume that we're dealing with a single song. Then my understanding of these definitions is that the intersection of S and S* contains all segments (corresponding lines in the reference and test files) for which:\r\n\r\n1. There's overlap between the gt start-end window and the estimated start-end window, AND\r\n2. The gt and estimated chord annotations are equal.\r\n\r\n**_Is the above correct?_** Also, point 1 above suggests that, if the estimated start-end window for a segment has no overlap with the corresponding gt start-end window, then (even if the chord annotations are equal) this segment contributes nothing to the CSR score. _**Is this correct?**_\r\n\r\nFinally, my understanding is that the phrase 'total duration' mentioned in the verbal definition of CSR refers to the total duration of the gt segment (in both the numerator and denominator of CSR), not the total duration of the estimated segment. _**Is this correct?**_ \r\n\r\nIf my understanding so far is correct, then I think that, by using the gt start-end times as the start-end times for my test files, I'm giving myself a higher chance of getting a good WCSR score (because the segments in my test files have greater chance of overlapping with the gt segments), **_right?_**. And so, to compare my WCSR scores with Junqi's, I shouldn't be assuming that estimated start-end times are the same as gt start-end times (but instead my test files should perhaps use the start-end windows that Junqi predicts in his experiments). (Note that I'm using exactly the same data and gt files he uses. The only difference is that I'm not predicting start-end times; I could use his code to produce the predicted start-end times he used, if necessary). \r\n\r\nThank you in advance!"}}
{"actor":{"display_login":"CKKOY"}}