{"action":"created","issue":{"url":"https://api.github.com/repos/fizyr/keras-retinanet/issues/322","repository_url":"https://api.github.com/repos/fizyr/keras-retinanet","labels_url":"https://api.github.com/repos/fizyr/keras-retinanet/issues/322/labels{/name}","comments_url":"https://api.github.com/repos/fizyr/keras-retinanet/issues/322/comments","events_url":"https://api.github.com/repos/fizyr/keras-retinanet/issues/322/events","html_url":"https://github.com/fizyr/keras-retinanet/issues/322","id":303836185,"number":322,"title":"Training slows down","user":{"login":"lvaleriu","id":927999,"avatar_url":"https://avatars0.githubusercontent.com/u/927999?v=4","gravatar_id":"","url":"https://api.github.com/users/lvaleriu","html_url":"https://github.com/lvaleriu","followers_url":"https://api.github.com/users/lvaleriu/followers","following_url":"https://api.github.com/users/lvaleriu/following{/other_user}","gists_url":"https://api.github.com/users/lvaleriu/gists{/gist_id}","starred_url":"https://api.github.com/users/lvaleriu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lvaleriu/subscriptions","organizations_url":"https://api.github.com/users/lvaleriu/orgs","repos_url":"https://api.github.com/users/lvaleriu/repos","events_url":"https://api.github.com/users/lvaleriu/events{/privacy}","received_events_url":"https://api.github.com/users/lvaleriu/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":31,"created_at":"2018-03-09T13:10:46Z","updated_at":"2018-04-23T14:59:56Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"I'm training mobilenet on coco with default parameters (batch_size=1, etc, etc, as specified here: https://github.com/fizyr/keras-retinanet/issues/300).\r\n\r\nI've seen that from time to time (every 2, 3 epochs) an epoch takes much more time and i dont understand why. \r\n\r\nNormal epoch speed: 2939s 294ms/step\r\nSlow epoch speed: 28566s 3s/step \r\n\r\nI've tried to stop the training and restart it as soon as i see this and then the speed goes normal.\r\nI've also tried to benchmark the generator when the training speed gets low but there was no disk IO reading speed issue.\r\n\r\nIt is like after 2, 3 epochs the training is slowed down by something and dont know what...\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/fizyr/keras-retinanet/issues/comments/383606638","html_url":"https://github.com/fizyr/keras-retinanet/issues/322#issuecomment-383606638","issue_url":"https://api.github.com/repos/fizyr/keras-retinanet/issues/322","id":383606638,"user":{"login":"hgaiser","id":716138,"avatar_url":"https://avatars1.githubusercontent.com/u/716138?v=4","gravatar_id":"","url":"https://api.github.com/users/hgaiser","html_url":"https://github.com/hgaiser","followers_url":"https://api.github.com/users/hgaiser/followers","following_url":"https://api.github.com/users/hgaiser/following{/other_user}","gists_url":"https://api.github.com/users/hgaiser/gists{/gist_id}","starred_url":"https://api.github.com/users/hgaiser/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hgaiser/subscriptions","organizations_url":"https://api.github.com/users/hgaiser/orgs","repos_url":"https://api.github.com/users/hgaiser/repos","events_url":"https://api.github.com/users/hgaiser/events{/privacy}","received_events_url":"https://api.github.com/users/hgaiser/received_events","type":"User","site_admin":false},"created_at":"2018-04-23T14:59:56Z","updated_at":"2018-04-23T14:59:56Z","author_association":"OWNER","body":"Alright. I'm assuming you rebooted after updating the driver?\r\n\r\nAlso, I never tested mobilenet yet (intend to do so soon), do you only see this slowdown on mobilenet ?"}}
{"actor":{"display_login":"hgaiser"}}