{"action":"created","issue":{"url":"https://api.github.com/repos/davisking/dlib/issues/202","repository_url":"https://api.github.com/repos/davisking/dlib","labels_url":"https://api.github.com/repos/davisking/dlib/issues/202/labels{/name}","comments_url":"https://api.github.com/repos/davisking/dlib/issues/202/comments","events_url":"https://api.github.com/repos/davisking/dlib/issues/202/events","html_url":"https://github.com/davisking/dlib/issues/202","id":172903699,"number":202,"title":"train_object_detector example shows incorrect testing results","user":{"login":"e-fominov","id":13793158,"avatar_url":"https://avatars.githubusercontent.com/u/13793158?v=3","gravatar_id":"","url":"https://api.github.com/users/e-fominov","html_url":"https://github.com/e-fominov","followers_url":"https://api.github.com/users/e-fominov/followers","following_url":"https://api.github.com/users/e-fominov/following{/other_user}","gists_url":"https://api.github.com/users/e-fominov/gists{/gist_id}","starred_url":"https://api.github.com/users/e-fominov/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/e-fominov/subscriptions","organizations_url":"https://api.github.com/users/e-fominov/orgs","repos_url":"https://api.github.com/users/e-fominov/repos","events_url":"https://api.github.com/users/e-fominov/events{/privacy}","received_events_url":"https://api.github.com/users/e-fominov/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":9,"created_at":"2016-08-24T09:19:10Z","updated_at":"2016-08-24T13:50:09Z","closed_at":null,"body":"While training some object detectors I have found that when detector is loaded from .svm file and used in real-world scenario, its precision is different from the results shown at training process. Precision/Recall/AP results are different. And the reason is because my training/testing data has many overlapped objects\r\n\r\nI have found that when object detector is trained [in sample code](https://github.com/davisking/dlib/blob/master/examples/train_object_detector.cpp), line  291:\r\n` object_detector<image_scanner_type> detector = trainer.train(images, object_locations, ignore);\r\n`\r\nTrainer has no custom test_box_overlap initialized in its params and it will make automatic test_box_overlap training. When the process will be finished - detector will have some trained test_box_overlap parameters.\r\n\r\nBut when sample code is testing object detector (line 297) - it does call test_object_detector function with default argument test_box_overlap:\r\n\r\n`cout << \"Test detector (precision,recall,AP): \" << test_object_detection_function(detector, images, object_locations) << endl;`\r\n\r\nSo test_object_detector will not use trained test_box_overlap instance from detector, but will use some default-constructed one for making decisions about detector's precision. Really - threre will be two different overlap testers used - one when detector makes detections and other for calculating precision. \r\n\r\nIs this the correct way to calculate detector's precision? \r\n\r\nMay be we should update sample code to use detector's trained overlap tester when calling test_object_detector? \r\n\r\n`"},"comment":{"url":"https://api.github.com/repos/davisking/dlib/issues/comments/242070715","html_url":"https://github.com/davisking/dlib/issues/202#issuecomment-242070715","issue_url":"https://api.github.com/repos/davisking/dlib/issues/202","id":242070715,"user":{"login":"e-fominov","id":13793158,"avatar_url":"https://avatars.githubusercontent.com/u/13793158?v=3","gravatar_id":"","url":"https://api.github.com/users/e-fominov","html_url":"https://github.com/e-fominov","followers_url":"https://api.github.com/users/e-fominov/followers","following_url":"https://api.github.com/users/e-fominov/following{/other_user}","gists_url":"https://api.github.com/users/e-fominov/gists{/gist_id}","starred_url":"https://api.github.com/users/e-fominov/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/e-fominov/subscriptions","organizations_url":"https://api.github.com/users/e-fominov/orgs","repos_url":"https://api.github.com/users/e-fominov/repos","events_url":"https://api.github.com/users/e-fominov/events{/privacy}","received_events_url":"https://api.github.com/users/e-fominov/received_events","type":"User","site_admin":false},"created_at":"2016-08-24T13:50:09Z","updated_at":"2016-08-24T13:50:09Z","body":"test_object_detection_function calls this:\r\n  \r\n```\r\n for (unsigned long i = 0; i < images.size(); ++i)\r\n        {\r\n            std::vector<std::pair<double,rectangle> > hits; \r\n            detector(images[i], hits, adjust_threshold);\r\n            correct_hits += impl::number_of_truth_hits(truth_dets[i], ignore[i], hits, overlap_tester, all_dets, missing_detections);\r\n            total_true_targets += truth_dets[i].size();\r\n        }\r\n\r\n```\r\n\r\nwhere overlap_tester is default (4th argument). hits will contain false detections missed by detector's overlap tester and impl::number_of_truth_hits will ignore them:\r\n```\r\n            unsigned long count = 0;\r\n            std::vector<bool> used(boxes.size(),false);\r\n            for (unsigned long i = 0; i < truth_boxes.size(); ++i)\r\n            {\r\n                bool found_match = false;\r\n                // Find the first box that hits truth_boxes[i]\r\n                for (unsigned long j = 0; j < boxes.size(); ++j)\r\n                {\r\n                    if (used[j])\r\n                        continue;\r\n\r\n                    if (overlap_tester(truth_boxes[i].get_rect(), boxes[j].second))\r\n                    {\r\n                        used[j] = true;\r\n                        ++count;\r\n                        found_match = true;\r\n                        break;\r\n                    }\r\n                }\r\n\r\n                if (!found_match)\r\n                    ++missing_detections;\r\n            }\r\n```\r\n\r\n"}}
{"actor":{"display_login":"e-fominov"}}