{"action":"created","issue":{"url":"https://api.github.com/repos/gcp/leela-zero/issues/908","repository_url":"https://api.github.com/repos/gcp/leela-zero","labels_url":"https://api.github.com/repos/gcp/leela-zero/issues/908/labels{/name}","comments_url":"https://api.github.com/repos/gcp/leela-zero/issues/908/comments","events_url":"https://api.github.com/repos/gcp/leela-zero/issues/908/events","html_url":"https://github.com/gcp/leela-zero/issues/908","id":299586347,"number":908,"title":"Maybe I found why the \"Hybrid\" weight is stronger than original weight","user":{"login":"pangafu","id":10149168,"avatar_url":"https://avatars2.githubusercontent.com/u/10149168?v=4","gravatar_id":"","url":"https://api.github.com/users/pangafu","html_url":"https://github.com/pangafu","followers_url":"https://api.github.com/users/pangafu/followers","following_url":"https://api.github.com/users/pangafu/following{/other_user}","gists_url":"https://api.github.com/users/pangafu/gists{/gist_id}","starred_url":"https://api.github.com/users/pangafu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pangafu/subscriptions","organizations_url":"https://api.github.com/users/pangafu/orgs","repos_url":"https://api.github.com/users/pangafu/repos","events_url":"https://api.github.com/users/pangafu/events{/privacy}","received_events_url":"https://api.github.com/users/pangafu/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":30,"created_at":"2018-02-23T03:19:36Z","updated_at":"2018-03-14T14:20:47Z","closed_at":null,"author_association":"NONE","body":"As we mentioned in #814  #867 , I download 2 weights:\r\n72ea669da5f491458cb5dfc44a80f0e760a9df71a8d0690b185822de49e00cb3.txt\r\naf3c6e330b932c97b0e517ac7db544d37fe1d18184f101d4b0340ff02f51ce88.txt\r\n\r\nAnd I made a \"Hybrid\" weight af3-72e_1-1.txt, it average parameters of upper 2 weight.\r\n\r\n\r\nThen I open a new game, make some move, and see the heatmap of 3 weights in same situation\r\nThe test command is (playout 1):  leelaz.exe -g --noponder -t1 -p 1 -w weight.txt\r\n![broad](https://user-images.githubusercontent.com/10149168/36576109-89bd51cc-1889-11e8-8d11-d3846c33ea8f.png)\r\n\r\nI found the hybrid weight output is very closed to 2 original weights output's average, the result can see bellow.\r\n![hybrid](https://user-images.githubusercontent.com/10149168/36576110-8a1e88c0-1889-11e8-9ac2-91d078078e7b.png)\r\n\r\n\r\nSo in my opinion, because the network is linear( or close to linear in small distance), we average the network's parameter, just equal to average several network's outputs. So \"Hybrid\" weight maybe equal to assemble and average many network's output, this method had been widely used to make predict more accurate."},"comment":{"url":"https://api.github.com/repos/gcp/leela-zero/issues/comments/373037757","html_url":"https://github.com/gcp/leela-zero/issues/908#issuecomment-373037757","issue_url":"https://api.github.com/repos/gcp/leela-zero/issues/908","id":373037757,"user":{"login":"pheasant75","id":18132237,"avatar_url":"https://avatars3.githubusercontent.com/u/18132237?v=4","gravatar_id":"","url":"https://api.github.com/users/pheasant75","html_url":"https://github.com/pheasant75","followers_url":"https://api.github.com/users/pheasant75/followers","following_url":"https://api.github.com/users/pheasant75/following{/other_user}","gists_url":"https://api.github.com/users/pheasant75/gists{/gist_id}","starred_url":"https://api.github.com/users/pheasant75/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pheasant75/subscriptions","organizations_url":"https://api.github.com/users/pheasant75/orgs","repos_url":"https://api.github.com/users/pheasant75/repos","events_url":"https://api.github.com/users/pheasant75/events{/privacy}","received_events_url":"https://api.github.com/users/pheasant75/received_events","type":"User","site_admin":false},"created_at":"2018-03-14T14:20:47Z","updated_at":"2018-03-14T14:20:47Z","author_association":"NONE","body":"@ThorAvaTahr Model ensembling is a common and popular method to improve accuracy in machine learning.  See some explanations and variants of ensembling strategies.\r\nhttps://mlwave.com/kaggle-ensembling-guide/\r\n\r\nCurrently what's done here are average and some manual test of different weights.  Technically, you can train another NN to come up with the best ensembling or even dynamically choose result from different network during a game.  Of course, that is only worth trying after we reach our training limit (when everything is stalled)."}}
{"actor":{"display_login":"pheasant75"}}