{"action":"created","issue":{"url":"https://api.github.com/repos/gcp/leela-zero/issues/2047","repository_url":"https://api.github.com/repos/gcp/leela-zero","labels_url":"https://api.github.com/repos/gcp/leela-zero/issues/2047/labels{/name}","comments_url":"https://api.github.com/repos/gcp/leela-zero/issues/2047/comments","events_url":"https://api.github.com/repos/gcp/leela-zero/issues/2047/events","html_url":"https://github.com/gcp/leela-zero/issues/2047","id":384440229,"node_id":"MDU6SXNzdWUzODQ0NDAyMjk=","number":2047,"title":"Problem with quasi-identical match games","user":{"login":"Friday9i","id":37338822,"node_id":"MDQ6VXNlcjM3MzM4ODIy","avatar_url":"https://avatars0.githubusercontent.com/u/37338822?v=4","gravatar_id":"","url":"https://api.github.com/users/Friday9i","html_url":"https://github.com/Friday9i","followers_url":"https://api.github.com/users/Friday9i/followers","following_url":"https://api.github.com/users/Friday9i/following{/other_user}","gists_url":"https://api.github.com/users/Friday9i/gists{/gist_id}","starred_url":"https://api.github.com/users/Friday9i/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Friday9i/subscriptions","organizations_url":"https://api.github.com/users/Friday9i/orgs","repos_url":"https://api.github.com/users/Friday9i/repos","events_url":"https://api.github.com/users/Friday9i/events{/privacy}","received_events_url":"https://api.github.com/users/Friday9i/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":33,"created_at":"2018-11-26T17:43:54Z","updated_at":"2018-11-30T07:51:11Z","closed_at":null,"author_association":"NONE","body":"**Synthesis (after two updates on 27/11/2018):** \r\n**ALL match games** tested since yesterday (~80 games from 3 tested nets against LZ191) **have exactly the same opening for the first 30 moves!** For this morning net (1:11am) at move 50, already well in the \"middle game\", 25 out of the 40 games analyzed had the same 2 variations: still very little diversity at move 50...\r\nTo make it short, it seems to be a big issue: current matches do not reflect the \"general strength\" of tested nets, these matches just reflect the strength of tested nets _only in these specific openings_. That's interesting but extremely biased. The good news is that the training is not biased (thanks to the noise in training games) but clearly match games are biased, which may hinder selection process and hence the rythm of progression of LZ nets. JMHO. \r\n**Note:** I'll do some tests tonight to see the difference in match results with & without noise. If results are strongly correlated (ie \"good\" nets with noise are also good without noise), that's not so much of an issue for LZ. On the opposite, if results are badly correlated (which I fear), it means we are selecting nets more or less randomly, which for sure is a pity and inefficient.\r\n\r\n**Now the details of the approach:**\r\nI did a short analysis of some match games, and it seems we’ve got a real problem with quasi-identical games: I’ll investigate more, but here is a first view of the problem.\r\n\r\n1) Symmetries\r\n\r\nFirst, to get a clear view, symmetries have to be taken into account. For that:\r\n-\tThe traditional Go \"etiquette\"/convention is to play the first move on top right corner, on or below the diagonal\r\n-\tIf the first move is on the diagonal, a (not-so-used) convention is to play the second move on or below the diagonal including the top right corner\r\n-\tIf the second move is on the diagonal, repeat (ie play on or below the diagonal)\r\n-\tOnce a move is played below the diagonal, no more symmetries: following moves can of course be played freely anywhere. \r\n\r\nSo I implemented it to convert sgf files to that convention (in Excel), which allows a relatively easy comparison of games. Converted like that, it is easy to see the point where they really diverge\r\n\r\n2) First quick analysis of match games \r\n\r\nI used that Excel tool to compare the first 10 games of the last promotion match (http://zero.sjeng.org/match-games/5bfbd48c469ad81ed787dc39)\r\nConclusion:\r\n-\tthe 10 games are identical up to move 30!!!\r\n-\t4 games are still identical up to move 66\r\n-\tWe have to wait for move 82 to get 10 different games…\r\n\r\n3) First conclusion\r\n\r\nI only analyze 10 games (as my tool is not very automatic yet), but it seems already clear that we have a problem of diversity, with most of the games being identical for a long time. And this problem may plague our promotion process: a \"good\" net can be rejected because it fails the most common opening(s) or on the opposite, a \"bad\" net can PASS just because he is lucky and win the most common opening(s)\r\n\r\n4) Next steps\r\n\r\nI’ll extend the analysis to more match games, to confirm (or not) the trend.\r\nIf it is confirmed, we should change the match settings to introduce more diversity (how should we do is the following question : -)"},"comment":{"url":"https://api.github.com/repos/gcp/leela-zero/issues/comments/443120677","html_url":"https://github.com/gcp/leela-zero/issues/2047#issuecomment-443120677","issue_url":"https://api.github.com/repos/gcp/leela-zero/issues/2047","id":443120677,"node_id":"MDEyOklzc3VlQ29tbWVudDQ0MzEyMDY3Nw==","user":{"login":"Friday9i","id":37338822,"node_id":"MDQ6VXNlcjM3MzM4ODIy","avatar_url":"https://avatars0.githubusercontent.com/u/37338822?v=4","gravatar_id":"","url":"https://api.github.com/users/Friday9i","html_url":"https://github.com/Friday9i","followers_url":"https://api.github.com/users/Friday9i/followers","following_url":"https://api.github.com/users/Friday9i/following{/other_user}","gists_url":"https://api.github.com/users/Friday9i/gists{/gist_id}","starred_url":"https://api.github.com/users/Friday9i/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Friday9i/subscriptions","organizations_url":"https://api.github.com/users/Friday9i/orgs","repos_url":"https://api.github.com/users/Friday9i/repos","events_url":"https://api.github.com/users/Friday9i/events{/privacy}","received_events_url":"https://api.github.com/users/Friday9i/received_events","type":"User","site_admin":false},"created_at":"2018-11-30T07:51:11Z","updated_at":"2018-11-30T07:51:11Z","author_association":"NONE","body":"@alreadydone Yeah we should try what you suggest.\r\nI did 2 tests at 100 visits and with 100 games with and 100 games without randomness (using -t 4 or -t 5 for tests without randomness, in order to avoid 99% of duplicate games ; -). Apparently winrates change quite significantly with/without randomness, but the difference may be due to luck (as typical error after 100 games is quite high, at ~5 point). I should test more, but I had not enough time yet. \r\nI'll also try to use your branch #2046 in the coming days, that's very nice, thanks! But I'm not a developer, hope I'll manage to use it..."}}
{"actor":{"display_login":"Friday9i"}}