{"action":"created","issue":{"url":"https://api.github.com/repos/mrharicot/monodepth/issues/107","repository_url":"https://api.github.com/repos/mrharicot/monodepth","labels_url":"https://api.github.com/repos/mrharicot/monodepth/issues/107/labels{/name}","comments_url":"https://api.github.com/repos/mrharicot/monodepth/issues/107/comments","events_url":"https://api.github.com/repos/mrharicot/monodepth/issues/107/events","html_url":"https://github.com/mrharicot/monodepth/issues/107","id":294990199,"node_id":"MDU6SXNzdWUyOTQ5OTAxOTk=","number":107,"title":"(Fine-tune) Train on my own dataset, but the loss values are not convergence.","user":{"login":"keishatsai","id":8175070,"node_id":"MDQ6VXNlcjgxNzUwNzA=","avatar_url":"https://avatars1.githubusercontent.com/u/8175070?v=4","gravatar_id":"","url":"https://api.github.com/users/keishatsai","html_url":"https://github.com/keishatsai","followers_url":"https://api.github.com/users/keishatsai/followers","following_url":"https://api.github.com/users/keishatsai/following{/other_user}","gists_url":"https://api.github.com/users/keishatsai/gists{/gist_id}","starred_url":"https://api.github.com/users/keishatsai/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/keishatsai/subscriptions","organizations_url":"https://api.github.com/users/keishatsai/orgs","repos_url":"https://api.github.com/users/keishatsai/repos","events_url":"https://api.github.com/users/keishatsai/events{/privacy}","received_events_url":"https://api.github.com/users/keishatsai/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":7,"created_at":"2018-02-07T03:00:10Z","updated_at":"2018-11-01T06:11:28Z","closed_at":"2018-03-11T16:44:57Z","author_association":"NONE","body":"Hello, I just wondering that why the loss value became higher during batch 44000, and at the first time, it went well to convergence. Is it because of my batch_size is too small? Currently, I am using `batch_size = 4` to avoid the memory crash, and there are about 5500 samples on my own dataset.\r\n\r\n`model-287200` is trained on cityscapes datasets with batch_size=4, and all the other settings remained the same. \r\nHere is my command line:\r\n`$python monodepth_main.py --mode train --batch_size 4 --checkpoint_path ~/wonderland/monodepth-tmp/model20180130/model20180130/model-287200 --retrain --encoder resnet50 --model_name model20180206city2my--data_path ~/wonderland/ --filenames_file ~/wonderland/paper_code/monodepth-master/utils/filenames/my_train_files.txt --log_directory ~/wonderland/monodepth-tmp/model20180206 --full_summary`\r\n\r\ntotal number of samples: 5500\r\ntotal number of steps: 68750\r\n\r\nbatch    100 | examples/s: 8.42 | loss: 0.36922 | time elapsed: 0.02h | time left: 11.92h\r\nbatch    200 | examples/s: 8.26 | loss: 0.41477 | time elapsed: 0.03h | time left: 10.46h\r\nbatch    300 | examples/s: 8.14 | loss: 0.33602 | time elapsed: 0.04h | time left: 9.94h\r\nbatch    400 | examples/s: 8.66 | loss: 0.38731 | time elapsed: 0.06h | time left: 9.71h\r\nbatch    500 | examples/s: 8.64 | loss: 0.43357 | time elapsed: 0.07h | time left: 9.55h\r\nbatch   1000 | examples/s: 8.52 | loss: 0.34376 | time elapsed: 0.14h | time left: 9.21h\r\nbatch   2000 | examples/s: 8.13 | loss: 0.30938 | time elapsed: 0.27h | time left: 8.96h\r\nbatch   3000 | examples/s: 8.38 | loss: 0.41769 | time elapsed: 0.40h | time left: 8.82h\r\n...\r\nbatch  44100 | examples/s: 9.16 | loss: 3.75232 | time elapsed: 5.82h | time left: 3.25h\r\nbatch  45300 | examples/s: 9.20 | loss: 1.19604 | time elapsed: 5.97h | time left: 3.09h\r\nbatch  45400 | examples/s: 9.20 | loss: 1.20178 | time elapsed: 5.98h | time left: 3.08h\r\nbatch  45500 | examples/s: 9.19 | loss: 1.47934 | time elapsed: 5.99h | time left: 3.06h\r\nbatch  45600 | examples/s: 8.76 | loss: 1.23994 | time elapsed: 6.01h | time left: 3.05h\r\nbatch  45700 | examples/s: 8.63 | loss: 1.43722 | time elapsed: 6.02h | time left: 3.04h\r\n...\r\nbatch  68700 | examples/s: 8.99 | loss: 1.28485 | time elapsed: 8.89h | time left: 0.01h\r\n\r\n`Another question is that I tried to add flag full_summary , and it seems to the summary doesn't have  been saved. `\r\n\r\nDo you have any idea why I encounter these problems? \r\nThank you.\r\nKeisha"},"comment":{"url":"https://api.github.com/repos/mrharicot/monodepth/issues/comments/434941594","html_url":"https://github.com/mrharicot/monodepth/issues/107#issuecomment-434941594","issue_url":"https://api.github.com/repos/mrharicot/monodepth/issues/107","id":434941594,"node_id":"MDEyOklzc3VlQ29tbWVudDQzNDk0MTU5NA==","user":{"login":"Usernamezhx","id":14010625,"node_id":"MDQ6VXNlcjE0MDEwNjI1","avatar_url":"https://avatars3.githubusercontent.com/u/14010625?v=4","gravatar_id":"","url":"https://api.github.com/users/Usernamezhx","html_url":"https://github.com/Usernamezhx","followers_url":"https://api.github.com/users/Usernamezhx/followers","following_url":"https://api.github.com/users/Usernamezhx/following{/other_user}","gists_url":"https://api.github.com/users/Usernamezhx/gists{/gist_id}","starred_url":"https://api.github.com/users/Usernamezhx/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Usernamezhx/subscriptions","organizations_url":"https://api.github.com/users/Usernamezhx/orgs","repos_url":"https://api.github.com/users/Usernamezhx/repos","events_url":"https://api.github.com/users/Usernamezhx/events{/privacy}","received_events_url":"https://api.github.com/users/Usernamezhx/received_events","type":"User","site_admin":false},"created_at":"2018-11-01T06:11:28Z","updated_at":"2018-11-01T06:11:28Z","author_association":"NONE","body":"@keishatsai  hi you said \r\n\r\n>  I also fine tune your pretrained model with my own dataset (batch size 4), but the result is convergence\r\nmeans that the **model_kitti: Our main model trained on the kitti split** model？？？"}}
{"actor":{"display_login":"Usernamezhx"}}