<p>This approach has some difficulties. First, in article you sent they use two-stage detection model with separate classification &quot;branches&quot;. In the same time YOLO is one-stage detector and is <strong>fullyconvolutional</strong>, that means there are no fullyconnected layers, and class predictions (1d) are taking from the whole 3d-tensor (see the image).
<a href="https://i.stack.imgur.com/0QlQV.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/0QlQV.png" alt="yolo tensors" /></a></p>
<p>You can take a look at <a href="https://arxiv.org/pdf/1612.08242.pdf" rel="nofollow noreferrer">YOLO9000</a> paper, the model was trained on detection and classification datasets at the same time - only loss function was changing.</p>
Object detector with multiple datasets
<p>I am interested in building a yolo detector with trained on multiple datasets where each dataset has it own detection head. It is a multi-task learning approach. I am not sure how to  convert the yolo detector architecture to support multiple head.</p>
<p>I came across the following projects, however I need your help to implement similar approach.</p>
<p><a href="https://github.com/xingyizhou/UniDet" rel="nofollow noreferrer">https://github.com/xingyizhou/UniDet</a></p>
<p><a href="https://link.springer.com/chapter/10.1007/978-981-16-6963-7_27" rel="nofollow noreferrer">https://link.springer.com/chapter/10.1007/978-981-16-6963-7_27</a></p>