{"action":"created","issue":{"url":"https://api.github.com/repos/tensorflow/models/issues/5165","repository_url":"https://api.github.com/repos/tensorflow/models","labels_url":"https://api.github.com/repos/tensorflow/models/issues/5165/labels{/name}","comments_url":"https://api.github.com/repos/tensorflow/models/issues/5165/comments","events_url":"https://api.github.com/repos/tensorflow/models/issues/5165/events","html_url":"https://github.com/tensorflow/models/issues/5165","id":352977500,"node_id":"MDU6SXNzdWUzNTI5Nzc1MDA=","number":5165,"title":"Loss increases suddenly after training the model nicely for ~30 mins ","user":{"login":"harshilpatel312","id":25410696,"node_id":"MDQ6VXNlcjI1NDEwNjk2","avatar_url":"https://avatars2.githubusercontent.com/u/25410696?v=4","gravatar_id":"","url":"https://api.github.com/users/harshilpatel312","html_url":"https://github.com/harshilpatel312","followers_url":"https://api.github.com/users/harshilpatel312/followers","following_url":"https://api.github.com/users/harshilpatel312/following{/other_user}","gists_url":"https://api.github.com/users/harshilpatel312/gists{/gist_id}","starred_url":"https://api.github.com/users/harshilpatel312/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/harshilpatel312/subscriptions","organizations_url":"https://api.github.com/users/harshilpatel312/orgs","repos_url":"https://api.github.com/users/harshilpatel312/repos","events_url":"https://api.github.com/users/harshilpatel312/events{/privacy}","received_events_url":"https://api.github.com/users/harshilpatel312/received_events","type":"User","site_admin":false},"labels":[{"id":387356495,"node_id":"MDU6TGFiZWwzODczNTY0OTU=","url":"https://api.github.com/repos/tensorflow/models/labels/stat:awaiting%20response","name":"stat:awaiting response","color":"5319e7","default":false}],"state":"open","locked":false,"assignee":{"login":"k-w-w","id":31663267,"node_id":"MDQ6VXNlcjMxNjYzMjY3","avatar_url":"https://avatars2.githubusercontent.com/u/31663267?v=4","gravatar_id":"","url":"https://api.github.com/users/k-w-w","html_url":"https://github.com/k-w-w","followers_url":"https://api.github.com/users/k-w-w/followers","following_url":"https://api.github.com/users/k-w-w/following{/other_user}","gists_url":"https://api.github.com/users/k-w-w/gists{/gist_id}","starred_url":"https://api.github.com/users/k-w-w/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/k-w-w/subscriptions","organizations_url":"https://api.github.com/users/k-w-w/orgs","repos_url":"https://api.github.com/users/k-w-w/repos","events_url":"https://api.github.com/users/k-w-w/events{/privacy}","received_events_url":"https://api.github.com/users/k-w-w/received_events","type":"User","site_admin":false},"assignees":[{"login":"k-w-w","id":31663267,"node_id":"MDQ6VXNlcjMxNjYzMjY3","avatar_url":"https://avatars2.githubusercontent.com/u/31663267?v=4","gravatar_id":"","url":"https://api.github.com/users/k-w-w","html_url":"https://github.com/k-w-w","followers_url":"https://api.github.com/users/k-w-w/followers","following_url":"https://api.github.com/users/k-w-w/following{/other_user}","gists_url":"https://api.github.com/users/k-w-w/gists{/gist_id}","starred_url":"https://api.github.com/users/k-w-w/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/k-w-w/subscriptions","organizations_url":"https://api.github.com/users/k-w-w/orgs","repos_url":"https://api.github.com/users/k-w-w/repos","events_url":"https://api.github.com/users/k-w-w/events{/privacy}","received_events_url":"https://api.github.com/users/k-w-w/received_events","type":"User","site_admin":false}],"milestone":null,"comments":6,"created_at":"2018-08-22T14:24:23Z","updated_at":"2018-08-23T18:32:23Z","closed_at":null,"author_association":"NONE","body":"### System information\r\n- **What is the top-level directory of the model you are using**: models/research/object-detection\r\n- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: Ubuntu 16.04.4 LTS\r\n- **TensorFlow installed from (source or binary)**: pip install tensorflow-gpu\r\n- **TensorFlow version (use command below)**: 1.8.0    \r\n- **GPU model and memory**: GeForce GTX 980 Ti\r\n- **CUDA/cuDNN version**: N/A\r\n- **Bazel version**: N/A\r\n- **Have I written custom code**: N/A\r\n- **Exact command to reproduce**: python train.py --logtostderr --train_dir=training/ --pipeline_config_path=training/faster_rcnn_resnet101_coco.config --num_clones=2 --ps_tasks=1\r\n\r\n### Describe the problem\r\nI gathered labelled data for object detection using my own camera, trained the model, ran predictions, everything works okay. Then I decided to supplement the data with labelled data from Open Images Dataset: cleaned up data, added zero padding to resize it to 1920x1080, and trained on it. The loss decreased steadily, as expected, for ~30 mins, after which it suddenly increases and the model never converges after that (see attached TotalLoss log plots).\r\n\r\nCould someone tell me what's wrong? I'm not sure if it is a bug or if I'm doing anything wrong.\r\n\r\n\r\n**ZOOMED IN :**\r\n\r\n![Zoomed In](https://user-images.githubusercontent.com/25410696/44469230-0557a280-a5f5-11e8-88e7-f42a4874f096.jpg)\r\n\r\n**ZOOMED OUT :**\r\n\r\n![Zoomed Out](https://user-images.githubusercontent.com/25410696/44469258-12749180-a5f5-11e8-8def-2e69a6b80f61.jpg)\r\n   "},"comment":{"url":"https://api.github.com/repos/tensorflow/models/issues/comments/415522899","html_url":"https://github.com/tensorflow/models/issues/5165#issuecomment-415522899","issue_url":"https://api.github.com/repos/tensorflow/models/issues/5165","id":415522899,"node_id":"MDEyOklzc3VlQ29tbWVudDQxNTUyMjg5OQ==","user":{"login":"harshilpatel312","id":25410696,"node_id":"MDQ6VXNlcjI1NDEwNjk2","avatar_url":"https://avatars2.githubusercontent.com/u/25410696?v=4","gravatar_id":"","url":"https://api.github.com/users/harshilpatel312","html_url":"https://github.com/harshilpatel312","followers_url":"https://api.github.com/users/harshilpatel312/followers","following_url":"https://api.github.com/users/harshilpatel312/following{/other_user}","gists_url":"https://api.github.com/users/harshilpatel312/gists{/gist_id}","starred_url":"https://api.github.com/users/harshilpatel312/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/harshilpatel312/subscriptions","organizations_url":"https://api.github.com/users/harshilpatel312/orgs","repos_url":"https://api.github.com/users/harshilpatel312/repos","events_url":"https://api.github.com/users/harshilpatel312/events{/privacy}","received_events_url":"https://api.github.com/users/harshilpatel312/received_events","type":"User","site_admin":false},"created_at":"2018-08-23T18:32:23Z","updated_at":"2018-08-23T18:32:23Z","author_association":"NONE","body":"Yeah, generating combined dataset from scratch was next up on my list. I will revert back with results soon.\r\n\r\nI'm sorry, I didn't get your question about the name. If you thought I mispelled \"oid\" for \"old\", then no; \"oid' stands for Open Images Dataset.."}}
{"actor":{"display_login":"harshilpatel312"}}