{"action":"created","issue":{"url":"https://api.github.com/repos/eriklindernoren/Keras-GAN/issues/87","repository_url":"https://api.github.com/repos/eriklindernoren/Keras-GAN","labels_url":"https://api.github.com/repos/eriklindernoren/Keras-GAN/issues/87/labels{/name}","comments_url":"https://api.github.com/repos/eriklindernoren/Keras-GAN/issues/87/comments","events_url":"https://api.github.com/repos/eriklindernoren/Keras-GAN/issues/87/events","html_url":"https://github.com/eriklindernoren/Keras-GAN/issues/87","id":369975939,"node_id":"MDU6SXNzdWUzNjk5NzU5Mzk=","number":87,"title":"nan during training in CycleGAN","user":{"login":"yhgon","id":2494562,"node_id":"MDQ6VXNlcjI0OTQ1NjI=","avatar_url":"https://avatars3.githubusercontent.com/u/2494562?v=4","gravatar_id":"","url":"https://api.github.com/users/yhgon","html_url":"https://github.com/yhgon","followers_url":"https://api.github.com/users/yhgon/followers","following_url":"https://api.github.com/users/yhgon/following{/other_user}","gists_url":"https://api.github.com/users/yhgon/gists{/gist_id}","starred_url":"https://api.github.com/users/yhgon/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yhgon/subscriptions","organizations_url":"https://api.github.com/users/yhgon/orgs","repos_url":"https://api.github.com/users/yhgon/repos","events_url":"https://api.github.com/users/yhgon/events{/privacy}","received_events_url":"https://api.github.com/users/yhgon/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2018-10-15T02:30:09Z","updated_at":"2019-06-24T13:56:29Z","closed_at":null,"author_association":"NONE","body":"Did you see NaN in the training?\r\nconfiguration\r\ntensorflow : 1.11.0\r\nkeras : 2.1.6\r\nGPU : K80 \r\nmodel : CycleGAN\r\ndataset : apple2orange \r\n\r\n\r\nI've test multiple models.  **aggan**, **wgan**,  **dcgan** , **pix2pix** works well in 1,000 epochs in my environment. \r\nwhen I try to test **cycleGAN**, it train well.. but I got NaN in epoch 67. \r\n\r\nI've modified the code to use BatchNormalization with  batch(32) instead of Instance Normalization  \r\n\r\ngan.train(epochs=200, batch_size=64, sample_interval=15) \r\n\r\nin Epoch 66, I got the result \r\n```\r\n/usr/local/lib/python3.6/dist-packages/scipy/misc/pilutil.py:482: FutureWarning: Conversion of the second argument of issubdtype from `int` to `np.signedinteger` is deprecated. In future, it will be treated as `np.int64 == np.dtype(int).type`.\r\n  if issubdtype(ts, int):\r\n/usr/local/lib/python3.6/dist-packages/scipy/misc/pilutil.py:485: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\r\n  elif issubdtype(type(size), float):\r\n/usr/local/lib/python3.6/dist-packages/keras/engine/training.py:975: UserWarning: Discrepancy between trainable weights and collected trainable weights, did you set `model.trainable` without calling `model.compile` after ?\r\n  'Discrepancy between trainable weights and collected trainable'\r\n[Epoch 0/200] [Batch 0/15] [D loss: 10.340113, acc:  16%] [G loss: 36.504208, adv: 9.286882, recon: 0.813415, id: 1.059065] time: 0:00:46.796445 \r\n[Epoch 0/200] [Batch 1/15] [D loss: 10.501231, acc:  14%] [G loss: 26.232471, adv: 4.194573, recon: 0.792024, id: 1.146999] time: 0:00:51.253706 \r\n[Epoch 0/200] [Batch 2/15] [D loss: 7.842362, acc:  17%] [G loss: 22.555479, adv: 3.907498, recon: 0.631787, id: 1.116496] time: 0:00:55.256927 \r\n[Epoch 0/200] [Batch 3/15] [D loss: 3.473462, acc:  21%] [G loss: 17.249832, adv: 2.483031, recon: 0.506841, id: 1.100872] time: 0:00:59.269601\r\n```\r\n```\r\n[Epoch 67/200] [Batch 7/15] [D loss: 0.071113, acc:  95%] [G loss: 1.885691, adv: 0.042987, recon: 0.077898, id: 0.101703] time: 1:05:19.633317 \r\n[Epoch 67/200] [Batch 8/15] [D loss: 0.096040, acc:  91%] [G loss: 1.872224, adv: 0.072618, recon: 0.075492, id: 0.099593] time: 1:05:23.721828 \r\n[Epoch 67/200] [Batch 9/15] [D loss: 0.151695, acc:  80%] [G loss: 2.235736, adv: 0.179553, recon: 0.082687, id: 0.102930] time: 1:05:27.831364 \r\n[Epoch 67/200] [Batch 10/15] [D loss: 0.140757, acc:  83%] [G loss: 1.851864, adv: 0.066634, recon: 0.074924, id: 0.101068] time: 1:05:31.932562 \r\n[Epoch 67/200] [Batch 11/15] [D loss: 0.093928, acc:  89%] [G loss: 2.103337, adv: 0.081708, recon: 0.085628, id: 0.117779] time: 1:05:36.041804 \r\n[Epoch 67/200] [Batch 12/15] [D loss: nan, acc:  60%] [G loss: 00nan, adv: 00nan, recon: 0.075524, id: 0.119893] time: 1:05:40.103866 \r\n[Epoch 67/200] [Batch 13/15] [D loss: nan, acc:  25%] [G loss: 00nan, adv: 00nan, recon: 00nan, id: 0.092526] time: 1:05:44.104335 \r\n[Epoch 68/200] [Batch 0/15] [D loss: nan, acc:  12%] [G loss: 00nan, adv: 00nan, recon: 00nan, id: 00nan] time: 1:05:48.023691 \r\n/usr/local/lib/python3.6/dist-packages/numpy/core/_methods.py:29: RuntimeWarning: invalid value encountered in reduce\r\n  return umr_minimum(a, axis, None, out, keepdims)\r\n[Epoch 68/200] [Batch 1/15] [D loss: nan, acc:   0%] [G loss: 00nan, adv: 00nan, recon: 00nan, id: 00nan] time: 1:05:52.220977 \r\n[Epoch 68/200] [Batch 2/15] [D loss: nan, acc:   0%] [G loss: 00nan, adv: 00nan, recon: 00nan, id: 00nan] time: 1:05:56.135995 \r\n[Epoch 68/200] [Batch 3/15] [D loss: nan, acc:   0%] [G loss: 00nan, adv: 00nan, recon: 00nan, id: 00nan] time: 1:06:00.045664 \r\n[Epoch 68/200] [Batch 4/15] [D loss: nan, acc:   0%] [G loss: 00nan, adv: 00nan, recon: 00nan, id: 00nan] time: 1:06:03.959619\r\n```"},"comment":{"url":"https://api.github.com/repos/eriklindernoren/Keras-GAN/issues/comments/505021088","html_url":"https://github.com/eriklindernoren/Keras-GAN/issues/87#issuecomment-505021088","issue_url":"https://api.github.com/repos/eriklindernoren/Keras-GAN/issues/87","id":505021088,"node_id":"MDEyOklzc3VlQ29tbWVudDUwNTAyMTA4OA==","user":{"login":"Feiyu-Zhang","id":34130100,"node_id":"MDQ6VXNlcjM0MTMwMTAw","avatar_url":"https://avatars1.githubusercontent.com/u/34130100?v=4","gravatar_id":"","url":"https://api.github.com/users/Feiyu-Zhang","html_url":"https://github.com/Feiyu-Zhang","followers_url":"https://api.github.com/users/Feiyu-Zhang/followers","following_url":"https://api.github.com/users/Feiyu-Zhang/following{/other_user}","gists_url":"https://api.github.com/users/Feiyu-Zhang/gists{/gist_id}","starred_url":"https://api.github.com/users/Feiyu-Zhang/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Feiyu-Zhang/subscriptions","organizations_url":"https://api.github.com/users/Feiyu-Zhang/orgs","repos_url":"https://api.github.com/users/Feiyu-Zhang/repos","events_url":"https://api.github.com/users/Feiyu-Zhang/events{/privacy}","received_events_url":"https://api.github.com/users/Feiyu-Zhang/received_events","type":"User","site_admin":false},"created_at":"2019-06-24T13:56:29Z","updated_at":"2019-06-24T13:56:29Z","author_association":"NONE","body":"> I've modified the code to use BatchNormalization with batch(32) instead of Instance Normalization\r\n> \r\n> below is my code the difference for generator.\r\n> \r\n> ```\r\n>             d = BatchNormalization()(d) #d = InstanceNormalization()(d)  \r\n>             u = BatchNormalization()(u) # u = InstanceNormalization()(u)\r\n> ```\r\n> \r\n> for discriminator\r\n> \r\n> ```\r\n>             if normalization:\r\n>                 d = BatchNormalization()(d)  # d = InstanceNormalization()(d)\r\n>             return d\r\n> ```\r\n\r\nPlease have you found the reason?"}}
{"actor":{"display_login":"Feiyu-Zhang"}}