{"action":"created","issue":{"url":"https://api.github.com/repos/pjreddie/darknet/issues/460","repository_url":"https://api.github.com/repos/pjreddie/darknet","labels_url":"https://api.github.com/repos/pjreddie/darknet/issues/460/labels{/name}","comments_url":"https://api.github.com/repos/pjreddie/darknet/issues/460/comments","events_url":"https://api.github.com/repos/pjreddie/darknet/issues/460/events","html_url":"https://github.com/pjreddie/darknet/issues/460","id":295557034,"number":460,"title":"Training yolo.2.0.cfg returns NaN for count = 0, even though image is annotated","user":{"login":"clockworkkiwi","id":36270888,"avatar_url":"https://avatars0.githubusercontent.com/u/36270888?v=4","gravatar_id":"","url":"https://api.github.com/users/clockworkkiwi","html_url":"https://github.com/clockworkkiwi","followers_url":"https://api.github.com/users/clockworkkiwi/followers","following_url":"https://api.github.com/users/clockworkkiwi/following{/other_user}","gists_url":"https://api.github.com/users/clockworkkiwi/gists{/gist_id}","starred_url":"https://api.github.com/users/clockworkkiwi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/clockworkkiwi/subscriptions","organizations_url":"https://api.github.com/users/clockworkkiwi/orgs","repos_url":"https://api.github.com/users/clockworkkiwi/repos","events_url":"https://api.github.com/users/clockworkkiwi/events{/privacy}","received_events_url":"https://api.github.com/users/clockworkkiwi/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":8,"created_at":"2018-02-08T15:35:11Z","updated_at":"2018-05-15T16:01:59Z","closed_at":null,"author_association":"NONE","body":"I am training yolo.2.0.cfg on a custom dataset and after some 100 Iterations I only get NaN like:\r\nRegion Avg IOU: nan, Class: nan, Obj: nan, No Obj: nan, Avg Recall: 0.000000,  count: 42\r\n\r\nI tried to reproduce the error on my CPU with batchsize 1 and only using 1 image. The image is annotated with 11 Objects, therefoe I thought that count should allways be 11. However it is sometimes 3,1, and 0 (see log below). When count is 0 I am getting NaN, probably because during calculation of IoU a division by 0 occurs.\r\n\r\nMy question is, is my concept of count wrong? And if not, why is it changing constantly?\r\nThe cfg and annotation file is provided below.\r\n\r\n./darknet detector train Training/cars.data Training/yolo.2.0_cars.cfg Training/darknet19_448.conv.23\r\nyolo\r\nlayer     filters    size              input                output\r\n    0 conv     32  3 x 3 / 1   608 x 608 x   3   ->   608 x 608 x  32\r\n    1 max          2 x 2 / 2   608 x 608 x  32   ->   304 x 304 x  32\r\n    2 conv     64  3 x 3 / 1   304 x 304 x  32   ->   304 x 304 x  64\r\n    3 max          2 x 2 / 2   304 x 304 x  64   ->   152 x 152 x  64\r\n    4 conv    128  3 x 3 / 1   152 x 152 x  64   ->   152 x 152 x 128\r\n    5 conv     64  1 x 1 / 1   152 x 152 x 128   ->   152 x 152 x  64\r\n    6 conv    128  3 x 3 / 1   152 x 152 x  64   ->   152 x 152 x 128\r\n    7 max          2 x 2 / 2   152 x 152 x 128   ->    76 x  76 x 128\r\n    8 conv    256  3 x 3 / 1    76 x  76 x 128   ->    76 x  76 x 256\r\n    9 conv    128  1 x 1 / 1    76 x  76 x 256   ->    76 x  76 x 128\r\n   10 conv    256  3 x 3 / 1    76 x  76 x 128   ->    76 x  76 x 256\r\n   11 max          2 x 2 / 2    76 x  76 x 256   ->    38 x  38 x 256\r\n   12 conv    512  3 x 3 / 1    38 x  38 x 256   ->    38 x  38 x 512\r\n   13 conv    256  1 x 1 / 1    38 x  38 x 512   ->    38 x  38 x 256\r\n   14 conv    512  3 x 3 / 1    38 x  38 x 256   ->    38 x  38 x 512\r\n   15 conv    256  1 x 1 / 1    38 x  38 x 512   ->    38 x  38 x 256\r\n   16 conv    512  3 x 3 / 1    38 x  38 x 256   ->    38 x  38 x 512\r\n   17 max          2 x 2 / 2    38 x  38 x 512   ->    19 x  19 x 512\r\n   18 conv   1024  3 x 3 / 1    19 x  19 x 512   ->    19 x  19 x1024\r\n   19 conv    512  1 x 1 / 1    19 x  19 x1024   ->    19 x  19 x 512\r\n   20 conv   1024  3 x 3 / 1    19 x  19 x 512   ->    19 x  19 x1024\r\n   21 conv    512  1 x 1 / 1    19 x  19 x1024   ->    19 x  19 x 512\r\n   22 conv   1024  3 x 3 / 1    19 x  19 x 512   ->    19 x  19 x1024\r\n   23 conv   1024  3 x 3 / 1    19 x  19 x1024   ->    19 x  19 x1024\r\n   24 conv   1024  3 x 3 / 1    19 x  19 x1024   ->    19 x  19 x1024\r\n   25 route  16\r\n   26 reorg              / 2    38 x  38 x 512   ->    19 x  19 x2048\r\n   27 route  26 24\r\n   28 conv   1024  3 x 3 / 1    19 x  19 x3072   ->    19 x  19 x1024\r\n   29 conv     30  1 x 1 / 1    19 x  19 x1024   ->    19 x  19 x  30\r\n   30 detection\r\nmask_scale: Using default '1.000000'\r\nLoading weights from Training/darknet19_448.conv.23...Done!\r\nLearning Rate: 0.001, Momentum: 0.9, Decay: 0.0005\r\nLoaded: 0.147676 seconds\r\nRegion Avg IOU: 0.237344, Class: 1.000000, Obj: 0.274930, No Obj: 0.443802, Avg Recall: 0.090909,  count: 11\r\n1: 576.071167, 576.071167 avg, 0.001000 rate, 71.343607 seconds, 1 images\r\nLoaded: 0.000104 seconds\r\nRegion Avg IOU: 0.077130, Class: 1.000000, Obj: 0.292493, No Obj: 0.446449, Avg Recall: 0.000000,  count: 11\r\n2: 702.093445, 588.673401 avg, 0.001000 rate, 69.007693 seconds, 2 images\r\nLoaded: 0.000073 seconds\r\nRegion Avg IOU: 0.130509, Class: 1.000000, Obj: 0.342454, No Obj: 0.444011, Avg Recall: 0.000000,  count: 11\r\n3: 576.471802, 587.453247 avg, 0.001000 rate, 69.223896 seconds, 3 images\r\nLoaded: 0.000078 seconds\r\nRegion Avg IOU: 0.048404, Class: 1.000000, Obj: 0.240457, No Obj: 0.440917, Avg Recall: 0.000000,  count: 3\r\n4: 555.401550, 584.248047 avg, 0.001000 rate, 68.168291 seconds, 4 images\r\nLoaded: 0.000082 seconds\r\nRegion Avg IOU: 0.062680, Class: 1.000000, Obj: 0.133529, No Obj: 0.450822, Avg Recall: 0.000000,  count: 11\r\n5: 647.937134, 590.616943 avg, 0.001000 rate, 67.656793 seconds, 5 images\r\nLoaded: 0.000079 seconds\r\nRegion Avg IOU: 0.065679, Class: 1.000000, Obj: 0.326323, No Obj: 0.441488, Avg Recall: 0.000000,  count: 3\r\n6: 475.536743, 579.108948 avg, 0.001000 rate, 66.304383 seconds, 6 images\r\nLoaded: 0.000087 seconds\r\n**Region Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.444047, Avg Recall: -nan,  count: 0**\r\n7: 438.538666, 565.051941 avg, 0.001000 rate, 67.867137 seconds, 7 images\r\nLoaded: 0.000070 seconds\r\nRegion Avg IOU: 0.132911, Class: 1.000000, Obj: 0.190774, No Obj: 0.443059, Avg Recall: 0.000000,  count: 11\r\n8: 619.727173, 570.519470 avg, 0.001000 rate, 66.372526 seconds, 8 images\r\nLoaded: 0.000081 seconds\r\nRegion Avg IOU: 0.252439, Class: 1.000000, Obj: 0.223981, No Obj: 0.443295, Avg Recall: 0.333333,  count: 3\r\n9: 460.461884, 559.513733 avg, 0.001000 rate, 68.189868 seconds, 9 images\r\nLoaded: 0.000087 seconds\r\nRegion Avg IOU: 0.142704, Class: 1.000000, Obj: 0.221254, No Obj: 0.443132, Avg Recall: 0.000000,  count: 11\r\n10: 569.588257, 560.521179 avg, 0.001000 rate, 66.707857 seconds, 10 images\r\nLoaded: 0.000085 seconds\r\nRegion Avg IOU: 0.024215, Class: 1.000000, Obj: 0.265335, No Obj: 0.443488, Avg Recall: 0.000000,  count: 1\r\n11: 446.488312, 549.117920 avg, 0.001000 rate, 65.911859 seconds, 11 images\r\nLoaded: 0.000075 seconds\r\nRegion Avg IOU: 0.136938, Class: 1.000000, Obj: 0.298591, No Obj: 0.442529, Avg Recall: 0.000000,  count: 11\r\n12: 619.259888, 556.132141 avg, 0.001000 rate, 67.203997 seconds, 12 images\r\nLoaded: 0.000087 seconds\r\nRegion Avg IOU: 0.128904, Class: 1.000000, Obj: 0.300296, No Obj: 0.449025, Avg Recall: 0.000000,  count: 11\r\n13: 537.905579, 554.309509 avg, 0.001000 rate, 75.953351 seconds, 13 images\r\nLoaded: 0.000117 seconds\r\nRegion Avg IOU: 0.219828, Class: 1.000000, Obj: 0.144575, No Obj: 0.442554, Avg Recall: 0.181818,  count: 11\r\n14: 587.459167, 557.624451 avg, 0.001000 rate, 69.909060 seconds, 14 images\r\nLoaded: 0.000088 seconds\r\nRegion Avg IOU: 0.118915, Class: 1.000000, Obj: 0.508260, No Obj: 0.442919, Avg Recall: 0.000000,  count: 1\r\n15: 449.169159, 546.778931 avg, 0.001000 rate, 66.654878 seconds, 15 images\r\nLoaded: 0.000085 seconds\r\nRegion Avg IOU: 0.228912, Class: 1.000000, Obj: 0.125506, No Obj: 0.442766, Avg Recall: 0.000000,  count: 1\r\n16: 443.907257, 536.491760 avg, 0.001000 rate, 70.434208 seconds, 16 images\r\nLoaded: 0.000077 seconds\r\n\r\n\r\n[1479502650254806942.txt](https://github.com/pjreddie/darknet/files/1707665/1479502650254806942.txt)\r\n\r\n[yolo.2.0_cars.cfg.txt](https://github.com/pjreddie/darknet/files/1707668/yolo.2.0_cars.cfg.txt)\r\n\r\nHelp is highly appreciated!"},"comment":{"url":"https://api.github.com/repos/pjreddie/darknet/issues/comments/389221328","html_url":"https://github.com/pjreddie/darknet/issues/460#issuecomment-389221328","issue_url":"https://api.github.com/repos/pjreddie/darknet/issues/460","id":389221328,"user":{"login":"chinmay5","id":16525717,"avatar_url":"https://avatars1.githubusercontent.com/u/16525717?v=4","gravatar_id":"","url":"https://api.github.com/users/chinmay5","html_url":"https://github.com/chinmay5","followers_url":"https://api.github.com/users/chinmay5/followers","following_url":"https://api.github.com/users/chinmay5/following{/other_user}","gists_url":"https://api.github.com/users/chinmay5/gists{/gist_id}","starred_url":"https://api.github.com/users/chinmay5/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/chinmay5/subscriptions","organizations_url":"https://api.github.com/users/chinmay5/orgs","repos_url":"https://api.github.com/users/chinmay5/repos","events_url":"https://api.github.com/users/chinmay5/events{/privacy}","received_events_url":"https://api.github.com/users/chinmay5/received_events","type":"User","site_admin":false},"created_at":"2018-05-15T16:01:59Z","updated_at":"2018-05-15T16:01:59Z","author_association":"NONE","body":"Is there some way to check if all my images have been annotated properly? Or is it just a shot in the dark?"}}
{"actor":{"display_login":"chinmay5"}}