{"action":"created","issue":{"url":"https://api.github.com/repos/tensorflow/tensor2tensor/issues/529","repository_url":"https://api.github.com/repos/tensorflow/tensor2tensor","labels_url":"https://api.github.com/repos/tensorflow/tensor2tensor/issues/529/labels{/name}","comments_url":"https://api.github.com/repos/tensorflow/tensor2tensor/issues/529/comments","events_url":"https://api.github.com/repos/tensorflow/tensor2tensor/issues/529/events","html_url":"https://github.com/tensorflow/tensor2tensor/issues/529","id":290065713,"number":529,"title":"Bug introduced in v1.3.0 causing training divergence","user":{"login":"martinpopel","id":724617,"avatar_url":"https://avatars1.githubusercontent.com/u/724617?v=4","gravatar_id":"","url":"https://api.github.com/users/martinpopel","html_url":"https://github.com/martinpopel","followers_url":"https://api.github.com/users/martinpopel/followers","following_url":"https://api.github.com/users/martinpopel/following{/other_user}","gists_url":"https://api.github.com/users/martinpopel/gists{/gist_id}","starred_url":"https://api.github.com/users/martinpopel/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/martinpopel/subscriptions","organizations_url":"https://api.github.com/users/martinpopel/orgs","repos_url":"https://api.github.com/users/martinpopel/repos","events_url":"https://api.github.com/users/martinpopel/events{/privacy}","received_events_url":"https://api.github.com/users/martinpopel/received_events","type":"User","site_admin":false},"labels":[{"id":627478123,"url":"https://api.github.com/repos/tensorflow/tensor2tensor/labels/bug","name":"bug","color":"ee0701","default":true}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":16,"created_at":"2018-01-19T18:26:10Z","updated_at":"2018-02-09T18:53:58Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"I train with exactly the same setup (1GPU, transformer_big_single_gpu, batch_size=1500, en-cs dataset with 58M training sentences, checkpoints saved each hour) with T2T v1.2.9 and [v1.3.0](https://github.com/tensorflow/tensor2tensor/pull/449) (which is the version immediately following 1.2.9). While in v1.2.9 everything is OK, **in v1.3.0 the training diverges after about one hour of training** and BLEU drops to almost zero after about 5 hours of training. See the learning curves below (orange=v1.2.9, magenta=1.3.0, y-axis is dev-set BLEU as measured by `t2t-bleu`, x-axis is training time in hours):\r\n![v129-vs-v130](https://user-images.githubusercontent.com/724617/35154688-745e8c20-fd2b-11e7-8b3f-f67f41d99e2d.png)\r\n\r\nI attach also training loss curves for these two experiments (but now orange=v1.3.0, gray=v1.2.9, so *the colors don't match the previous graph*, x-axis is now in steps):\r\n![v129-vs-v130-train-loss](https://user-images.githubusercontent.com/724617/35165125-612f6d64-fd4d-11e7-81c2-7a1bba8bfd45.png)\r\n\r\n* This confirms the problem: the training loss in v1.3.0 starts slowly growing after 1h (10k steps) of training. **So the bug is already in training, not in decoding.**\r\n* I am not sure if it is significant, but v1.3.0 has lower training loss than v1.2.9 between 2k and 8k steps. This could mean that it is not a bug, but just faster training, which needs different optimal hyperparameters to prevent the divergence (e.g. higher a bigger batch size or more warmup steps) than v1.2.9. But until it is proven, I will call it a bug/problem.\r\n* I tried all versions between 1.3.1 and 1.4.2 and they show the same problem as 1.3.0.\r\n* Other users (e.g. in #525) report similar problems with different datasets (en-de, en-fr).\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/tensorflow/tensor2tensor/issues/comments/364524682","html_url":"https://github.com/tensorflow/tensor2tensor/issues/529#issuecomment-364524682","issue_url":"https://api.github.com/repos/tensorflow/tensor2tensor/issues/529","id":364524682,"user":{"login":"rsepassi","id":1115739,"avatar_url":"https://avatars3.githubusercontent.com/u/1115739?v=4","gravatar_id":"","url":"https://api.github.com/users/rsepassi","html_url":"https://github.com/rsepassi","followers_url":"https://api.github.com/users/rsepassi/followers","following_url":"https://api.github.com/users/rsepassi/following{/other_user}","gists_url":"https://api.github.com/users/rsepassi/gists{/gist_id}","starred_url":"https://api.github.com/users/rsepassi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rsepassi/subscriptions","organizations_url":"https://api.github.com/users/rsepassi/orgs","repos_url":"https://api.github.com/users/rsepassi/repos","events_url":"https://api.github.com/users/rsepassi/events{/privacy}","received_events_url":"https://api.github.com/users/rsepassi/received_events","type":"User","site_admin":false},"created_at":"2018-02-09T18:53:58Z","updated_at":"2018-02-09T18:53:58Z","author_association":"MEMBER","body":"Thanks @vince62s. That's good news. \r\n\r\nIt may be that we need to retune some of the hparams sets and update them.\r\n\r\nIn general it seems to be very difficult to maintain reproducibility without freezing a codebase/codepath entirely.\r\n\r\nWhat we may want to consider doing is a hard fork of the entire `T2TModel.estimator_model_fn` and hparams set for every publication. Copy in all the code, put the date in the model name and hparams set, and then continue development on the main model separately.\r\n\r\nOpen to suggestions and discussion here. This is a hard problem. "}}
{"actor":{"display_login":"rsepassi"}}