{"action":"created","issue":{"url":"https://api.github.com/repos/Microsoft/AutonomousDrivingCookbook/issues/43","repository_url":"https://api.github.com/repos/Microsoft/AutonomousDrivingCookbook","labels_url":"https://api.github.com/repos/Microsoft/AutonomousDrivingCookbook/issues/43/labels{/name}","comments_url":"https://api.github.com/repos/Microsoft/AutonomousDrivingCookbook/issues/43/comments","events_url":"https://api.github.com/repos/Microsoft/AutonomousDrivingCookbook/issues/43/events","html_url":"https://github.com/Microsoft/AutonomousDrivingCookbook/issues/43","id":323125687,"number":43,"title":"Model Training early stopping","user":{"login":"cl3m3nt","id":8882133,"avatar_url":"https://avatars1.githubusercontent.com/u/8882133?v=4","gravatar_id":"","url":"https://api.github.com/users/cl3m3nt","html_url":"https://github.com/cl3m3nt","followers_url":"https://api.github.com/users/cl3m3nt/followers","following_url":"https://api.github.com/users/cl3m3nt/following{/other_user}","gists_url":"https://api.github.com/users/cl3m3nt/gists{/gist_id}","starred_url":"https://api.github.com/users/cl3m3nt/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cl3m3nt/subscriptions","organizations_url":"https://api.github.com/users/cl3m3nt/orgs","repos_url":"https://api.github.com/users/cl3m3nt/repos","events_url":"https://api.github.com/users/cl3m3nt/events{/privacy}","received_events_url":"https://api.github.com/users/cl3m3nt/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2018-05-15T08:58:20Z","updated_at":"2018-05-15T17:11:34Z","closed_at":null,"author_association":"NONE","body":"Hi, can you provide a feedback to following issue ?\r\nI appreciate your help\r\n\r\n## Problem description\r\nThere is an early stopping of GPU processing activity during\r\n\r\n## Problem details\r\nThe issue raised when launching following training activity:\r\nhistory = model.fit_generator(train_generator, steps_per_epoch=num_train_examples//batch_size, epochs=500, callbacks=callbacks,\\\r\n                   validation_data=eval_generator, validation_steps=num_eval_examples//batch_size, verbose=2)\r\nThe output message is the following:\r\nEpoch 32/500\r\nEpoch 00032: val_loss did not improve\r\n - 49s - loss: 0.0022 - val_loss: 3.7090e-04\r\nEpoch 00032: early stopping\r\n\r\nThis matches only 6% training activity:\r\nTraining 6% 32/500 [33:26<6:26:38, 49.57s/it]\r\n\r\n## Exepriment/Environment details\r\n* Tutorial used: For example, AirSimE2EDeepLearning\r\n* Environment used: DSVM with K80 GPU on NC6 Azure VM\r\n* Versions of artifacts used (if applicable): *(For example, Python 3.6, Keras 2.1.2, Tensorflow backend\r\n"},"comment":{"url":"https://api.github.com/repos/Microsoft/AutonomousDrivingCookbook/issues/comments/389243258","html_url":"https://github.com/Microsoft/AutonomousDrivingCookbook/issues/43#issuecomment-389243258","issue_url":"https://api.github.com/repos/Microsoft/AutonomousDrivingCookbook/issues/43","id":389243258,"user":{"login":"mitchellspryn","id":20287427,"avatar_url":"https://avatars3.githubusercontent.com/u/20287427?v=4","gravatar_id":"","url":"https://api.github.com/users/mitchellspryn","html_url":"https://github.com/mitchellspryn","followers_url":"https://api.github.com/users/mitchellspryn/followers","following_url":"https://api.github.com/users/mitchellspryn/following{/other_user}","gists_url":"https://api.github.com/users/mitchellspryn/gists{/gist_id}","starred_url":"https://api.github.com/users/mitchellspryn/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mitchellspryn/subscriptions","organizations_url":"https://api.github.com/users/mitchellspryn/orgs","repos_url":"https://api.github.com/users/mitchellspryn/repos","events_url":"https://api.github.com/users/mitchellspryn/events{/privacy}","received_events_url":"https://api.github.com/users/mitchellspryn/received_events","type":"User","site_admin":false},"created_at":"2018-05-15T17:11:34Z","updated_at":"2018-05-15T17:11:34Z","author_association":"COLLABORATOR","body":"This is expected and desired. Notice how your validation loss is not improving - this means that if you continue to train, your model will overfit. You can see the effects of that by removing the [EarlyStoppingCallback](https://keras.io/callbacks/#earlystopping) from your code and retraining - you will most likely get poor results.\r\n\r\nHow does the trained model perform? Does it seem to predict the correct angles (in the last cell of the notebook)? How far does the car go before crashing? Note that since we don't have a very large dataset, it's possible that the model converges to a suboptimal minimum. Retraining will help in that case."}}
{"actor":{"display_login":"mitchellspryn"}}