{"action":"created","issue":{"url":"https://api.github.com/repos/psychoinformatics-de/remodnav/issues/2","repository_url":"https://api.github.com/repos/psychoinformatics-de/remodnav","labels_url":"https://api.github.com/repos/psychoinformatics-de/remodnav/issues/2/labels{/name}","comments_url":"https://api.github.com/repos/psychoinformatics-de/remodnav/issues/2/comments","events_url":"https://api.github.com/repos/psychoinformatics-de/remodnav/issues/2/events","html_url":"https://github.com/psychoinformatics-de/remodnav/pull/2","id":359405428,"node_id":"MDExOlB1bGxSZXF1ZXN0MjE0OTAzNjU5","number":2,"title":"Script to reproduce basic stats reported in Anderson et al/ compare algorithm against them","user":{"login":"mih","id":136479,"node_id":"MDQ6VXNlcjEzNjQ3OQ==","avatar_url":"https://avatars1.githubusercontent.com/u/136479?v=4","gravatar_id":"","url":"https://api.github.com/users/mih","html_url":"https://github.com/mih","followers_url":"https://api.github.com/users/mih/followers","following_url":"https://api.github.com/users/mih/following{/other_user}","gists_url":"https://api.github.com/users/mih/gists{/gist_id}","starred_url":"https://api.github.com/users/mih/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mih/subscriptions","organizations_url":"https://api.github.com/users/mih/orgs","repos_url":"https://api.github.com/users/mih/repos","events_url":"https://api.github.com/users/mih/events{/privacy}","received_events_url":"https://api.github.com/users/mih/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2018-09-12T10:01:26Z","updated_at":"2018-09-12T20:40:16Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/psychoinformatics-de/remodnav/pulls/2","html_url":"https://github.com/psychoinformatics-de/remodnav/pull/2","diff_url":"https://github.com/psychoinformatics-de/remodnav/pull/2.diff","patch_url":"https://github.com/psychoinformatics-de/remodnav/pull/2.patch"},"body":"ATM we cannot reproduce (reason is subject to research). Here is a summary of the differences: duration stats for fixations, saccades, PSOs and pursuits. The first value is our's, the one in parenthesis is the one reported in the paper. Note, our values are not own detection results, but stats computed from their released data. Subjectively substantial deviations are in **BOLD**, although there really should be no deviations at all, and the code to compute all stats is the same for all events (see PR).\r\n\r\nConclusion: We get what is in the paper for saccades and PSOs, but something substantially different for number of fixations.\r\n\r\n### Fixation durations\r\nCoder | IMG-Mean | IMG-SD | IMG-No | VID-Mean | VID-SD | VID-No\r\n--- | --- | --- | --- | --- | --- | ---\r\nMN | 252 (248) | 285 (271) | **403 (380)** | 304 (318) | 277 (289) | **82 (67)**\r\nRA | 247 (242) | 288 (273) | **391 (369)** | 232 (240) | 177 (189) | **81 (67)**\r\n\r\n### Saccade durations\r\nCoder | IMG-Mean | IMG-SD | IMG-No | VID-Mean | VID-SD | VID-No\r\n--- | --- | --- | --- | --- | --- | ---\r\nMN | 29 (30) | 17 (17) | 377 (376) | 26 (26) | 13 (13) | 117 (116)\r\nRA | 31 (31) | 15 (15) | 374 (372) | 25 (25) | 12 (12) | 127 (126)\r\n\r\n### PSO durations\r\nCoder | IMG-Mean | IMG-SD | IMG-No | VID-Mean | VID-SD | VID-No\r\n--- | --- | --- | --- | --- | --- | ---\r\nMN | 21 (21) | 11 (11) | 313 (312) | 20 (20) | 11 (11) | 97 (97)\r\nRA | 21 (21) | 9 (9) | 310 (309) | 17 (17) | 8 (8) | 89 (89)\r\n\r\n### Pursuit durations\r\nCoder | IMG-Mean | IMG-SD | IMG-No | VID-Mean | VID-SD | VID-No\r\n--- | --- | --- | --- | --- | --- | ---\r\nMN | 363 (363) | **153 (187)** | 3 (3) | 528 (521) | 344 (347) | 51 (50)\r\nRA | 299 (305) | 175 (184) | 17 (16) | 481 (472) | 317 (319) | 70 (68)\r\n\r\n## Confusions\r\n\r\nAssuming we make no mistakes extracting the Anderson labels, here is how our algorithm performs re confusions. \r\n\r\n### First without pursuit events (as in the paper):\r\n\r\n#### Compared to coder MN\r\n![image](https://user-images.githubusercontent.com/136479/45426454-97a31180-b69c-11e8-94b4-84641b84d5e5.png)\r\n\r\n#### Compared to coder RA\r\n![image](https://user-images.githubusercontent.com/136479/45426414-7c380680-b69c-11e8-9f46-9bf322831854.png)\r\n\r\n### With pursuit events:\r\n\r\n#### Compared to coder MN\r\n![image](https://user-images.githubusercontent.com/136479/45426562-e355bb00-b69c-11e8-91c4-59b03cc2ff57.png)\r\n\r\n#### Compared to coder RA\r\n![image](https://user-images.githubusercontent.com/136479/45426611-01232000-b69d-11e8-847d-a3ace1430587.png)\r\n\r\n\r\n## Interim conclusion\r\n\r\nPerformance looks good. Without pursuit this looks better than the stats in the paper (although I am not 100% confident that we compute things the exact same way). Confusion patterns with and without pursuit look sensible.\r\n\r\nCritical feedback appreciated!"},"comment":{"url":"https://api.github.com/repos/psychoinformatics-de/remodnav/issues/comments/420789767","html_url":"https://github.com/psychoinformatics-de/remodnav/pull/2#issuecomment-420789767","issue_url":"https://api.github.com/repos/psychoinformatics-de/remodnav/issues/2","id":420789767,"node_id":"MDEyOklzc3VlQ29tbWVudDQyMDc4OTc2Nw==","user":{"login":"AdinaWagner","id":29738718,"node_id":"MDQ6VXNlcjI5NzM4NzE4","avatar_url":"https://avatars1.githubusercontent.com/u/29738718?v=4","gravatar_id":"","url":"https://api.github.com/users/AdinaWagner","html_url":"https://github.com/AdinaWagner","followers_url":"https://api.github.com/users/AdinaWagner/followers","following_url":"https://api.github.com/users/AdinaWagner/following{/other_user}","gists_url":"https://api.github.com/users/AdinaWagner/gists{/gist_id}","starred_url":"https://api.github.com/users/AdinaWagner/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AdinaWagner/subscriptions","organizations_url":"https://api.github.com/users/AdinaWagner/orgs","repos_url":"https://api.github.com/users/AdinaWagner/repos","events_url":"https://api.github.com/users/AdinaWagner/events{/privacy}","received_events_url":"https://api.github.com/users/AdinaWagner/received_events","type":"User","site_admin":false},"created_at":"2018-09-12T20:40:16Z","updated_at":"2018-09-12T20:40:16Z","author_association":"COLLABORATOR","body":"These results look neat. Thanks a lot for all the work! \r\n\r\nI have not succeeded in finding an definite explanation for the weird occasional differences in fixation and pursuit duration. However, their paper was published online/accepted in 2016 -- the main analysis code was published on Github in Aug 2017 with the commit message 'Added code for processing data extracted from algorithms, including latest(?) version of extracted data. More code to follow.' Maybe these 'latest(?)' versions of extracted data bear the differences between the results reported in the paper and their more recent code. It at least suggests that anything more recent than something previous has been shared a year after the paper was accepted.\r\n\r\nAnd to quote their readme:\r\n\r\n> \"Some values in the scripts may have been interactively changed during the analysis, so it should not be a interpreted as run-once-for-complete-results code.\"\r\nas well as\r\n> \"Some of the matlab code used for the publication\" \r\nwith an emphasis on *some* -I wasn't able to find some functions anywhere, e.g. their function simpleAgreement (in mainDetection, used for calculation of proportion of correct classifications), so in my opinion it seems impossible or at least unfeasible to figure out the exact way of their computations.\r\n\r\nThe way you computed the results looks perfectly reasonable to me regardless of whether this may or may not be a 100% replication of their analysis. Its cool that the performance without pursuits exceeds their stats. For the results with added pursuits the results are still showing good performance of remodnav, and its understandable that adding a classification label to the confusion matrix is more likely to decrease overall classification accuracy compared to the papers \"only Top 3 events\" approach."}}
{"actor":{"display_login":"AdinaWagner"}}