{"action":"created","issue":{"url":"https://api.github.com/repos/gcp/leela-zero/issues/1569","repository_url":"https://api.github.com/repos/gcp/leela-zero","labels_url":"https://api.github.com/repos/gcp/leela-zero/issues/1569/labels{/name}","comments_url":"https://api.github.com/repos/gcp/leela-zero/issues/1569/comments","events_url":"https://api.github.com/repos/gcp/leela-zero/issues/1569/events","html_url":"https://github.com/gcp/leela-zero/issues/1569","id":333310111,"node_id":"MDU6SXNzdWUzMzMzMTAxMTE=","number":1569,"title":"8M games - training progress note","user":{"login":"gcp","id":1104290,"node_id":"MDQ6VXNlcjExMDQyOTA=","avatar_url":"https://avatars2.githubusercontent.com/u/1104290?v=4","gravatar_id":"","url":"https://api.github.com/users/gcp","html_url":"https://github.com/gcp","followers_url":"https://api.github.com/users/gcp/followers","following_url":"https://api.github.com/users/gcp/following{/other_user}","gists_url":"https://api.github.com/users/gcp/gists{/gist_id}","starred_url":"https://api.github.com/users/gcp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gcp/subscriptions","organizations_url":"https://api.github.com/users/gcp/orgs","repos_url":"https://api.github.com/users/gcp/repos","events_url":"https://api.github.com/users/gcp/events{/privacy}","received_events_url":"https://api.github.com/users/gcp/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":8,"created_at":"2018-06-18T15:06:07Z","updated_at":"2018-06-19T01:53:43Z","closed_at":null,"author_association":"OWNER","body":"Small note because I know many people will start \"panicking\" as we have 200k games with no notable strength progress (i.e. PASS).\r\n\r\nOne reason why progress dropped a bit is that the ELF games left the training window. This wasn't intentional. I tried a quick fix (I was traveling) of increasing the window to 800k but the result wasn't particularly good. I somewhat rewrote the dumping code now to correctly deal with this and the next run should be closer to what was intended  (~230k ELF + 250k regular). We'll see if this improves things. There'll be some delay before the next bunch of networks as I was fiddling with the training setup to get it debugged.\r\n\r\nIf these fixes don't produce progress, the next step will be a lowering of the learning rate. Because we're using SWA this might not necessarily produce a jump either.\r\n\r\nIt would be good to be able to start training and comparing 256x20 now, to figure out if network size is the issue (it sounds likely, but we do need confirmation in the form of a stronger 256x20 without which we can't do anything anyway!). It is really **extremely** unfortunate that we have the problem in #167 now. If this does not get resolved in a few days I will see if I can get some other hosting set up. (Data loss is not an issue, I have 2 full backups of the database...)"},"comment":{"url":"https://api.github.com/repos/gcp/leela-zero/issues/comments/398249220","html_url":"https://github.com/gcp/leela-zero/issues/1569#issuecomment-398249220","issue_url":"https://api.github.com/repos/gcp/leela-zero/issues/1569","id":398249220,"node_id":"MDEyOklzc3VlQ29tbWVudDM5ODI0OTIyMA==","user":{"login":"bjiyxo","id":30645842,"node_id":"MDQ6VXNlcjMwNjQ1ODQy","avatar_url":"https://avatars2.githubusercontent.com/u/30645842?v=4","gravatar_id":"","url":"https://api.github.com/users/bjiyxo","html_url":"https://github.com/bjiyxo","followers_url":"https://api.github.com/users/bjiyxo/followers","following_url":"https://api.github.com/users/bjiyxo/following{/other_user}","gists_url":"https://api.github.com/users/bjiyxo/gists{/gist_id}","starred_url":"https://api.github.com/users/bjiyxo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bjiyxo/subscriptions","organizations_url":"https://api.github.com/users/bjiyxo/orgs","repos_url":"https://api.github.com/users/bjiyxo/repos","events_url":"https://api.github.com/users/bjiyxo/events{/privacy}","received_events_url":"https://api.github.com/users/bjiyxo/received_events","type":"User","site_admin":false},"created_at":"2018-06-19T01:53:43Z","updated_at":"2018-06-19T01:53:43Z","author_association":"NONE","body":"@gcp Can you post the new raw training data (including 147, 148, and ELF) on Google Drive or Dropbox? Then I can train a new 20b catching up the latest 15b."}}
{"actor":{"display_login":"bjiyxo"}}