<p><strong>Some of the preferred ways to improve <em>detection time</em> with already trained Yolov3 model are:</strong></p>
<ul>
<li>Quantisation: Run inference with INT8 instead of FP32. You can use this <a href="https://github.com/AlexeyAB/yolo2_light" rel="nofollow noreferrer">repo</a> for this purpose.</li>
<li>Use Inference accelerator such as <a href="https://developer.nvidia.com/tensorrt" rel="nofollow noreferrer">TensorRT</a> since you're using Nvidia's GPU. The tool includes good amount of inference oriented optimisations along with quantisation optimisations INT8 and FP16 to reduce detection time. This <a href="https://forums.developer.nvidia.com/t/yolov3-with-tensorrt-5/68705" rel="nofollow noreferrer">thread</a> talks about Yolov3 inference with TensorRT5. Use <a href="https://github.com/wang-xinyu/tensorrtx" rel="nofollow noreferrer">this</a> repo for Yolov3 on TensorRT7.</li>
<li>Use inference library such as <a href="https://github.com/ceccocats/tkDNN" rel="nofollow noreferrer">tkDNN</a>, which is a Deep Neural Network library built with cuDNN and tensorRT primitives, specifically thought to work on NVIDIA Jetson Boards.</li>
</ul>
<p><strong>If you're open to do the model training there are few more options other than the ones mentioned above:</strong></p>
<ul>
<li>You can train the models with tinier versions rather than full Yolo versions, of course this comes at the cost of drop in accuracy/mAP. You can train <a href="https://github.com/AlexeyAB/darknet/blob/master/cfg/yolov4-tiny-custom.cfg" rel="nofollow noreferrer">tiny-yolov4</a> (latest model) or train <a href="https://github.com/AlexeyAB/darknet/blob/master/cfg/yolov3-tiny_obj.cfg" rel="nofollow noreferrer">tiny-yolov3</a>.</li>
<li>Model Pruning - If you could rank the neurons in the network according to how much they contribute, you could then remove the low ranking neurons from the network, resulting in a smaller and faster network. Pruned yolov3 research <a href="https://arxiv.org/abs/1907.11093v1" rel="nofollow noreferrer">paper</a> and it's <a href="https://github.com/PengyiZhang/SlimYOLOv3" rel="nofollow noreferrer">implementation</a>. <a href="https://github.com/Lam1360/YOLOv3-model-pruning" rel="nofollow noreferrer">This</a> is another pruned Yolov3 implementation.</li>
</ul>
How to improve YOLOv3 detection time? (OpenCV + Python)
<p>I'm using YOLOv3 custom trained model with OpenCV 4.2.0 compiled with CUDA. When I'm testing code in Python I'm using OpenCV on GPU (GTX1050 Ti) but detection on single image (416px x 416px) takes 0.055 s (~20 FPS). My config file is set to small object detection, because I need to detect ~ 10px x 10px objects on 2500px x 2000px images so I split original image into 30 smaller pieces. My goal is to reach 0.013 s (~80 FPS) on 416px x 416px image. Is it possible in Python with OpenCV? If not, how to do it in proper way?</p>
<p>PS. Currently detection takes like 50% of CPU, 5GB RAM and 6% GPU.</p>