<p>You can take a look at this <a href="https://arxiv.org/pdf/1611.10012.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/1611.10012.pdf</a> which contains comprehensive survey of recent detection architectures. Basically there are 3 meta architectures and all models fall into one of those categories:</p>

<ol>
<li>Faster-RCNN: Similar to the paper you have referenced, this is the improved version of fast-rcnn which did not use selective search and directly integrate proposal generation into the network known as region proposal network(rpn).</li>
<li>RFCN:similar in architecture to 1, except that roi pooling is performed differently, known as position sensitive roi pooling.</li>
<li>SSD:Modifies the rpn in Faster-rcnn to directly output class probabilities, eliminating the need for per-roi computation as is done in roi pooling. This is the fastest architecture type. Yolo falls into this architecture.</li>
</ol>

<p>I think based on my rough read through of the paper you have referenced, type 3 is the one you are looking for. However, in terms of implementation, it can be a little tricky to implement equation 3, i.e. you may need to stop backpropagating gradients to regions(or at least think about how it could effect the final results) that do not overlap with primary region as this architecture type computes probabilities for whole image.</p>

<p>I also note that there is in fact no primary/secondary "classifiers". The paper described primary/secondary "regions", the primary region is the region that contains the person(i.e use a person detector to find primary region first). And secondary regions are those that overlap with primary region. For activity classification <strong>there is only one classifier</strong> except that primary region carries more weight and secondary regions each contribute a little to the final prediction score.</p>
Ways to combine two deep learning based classifiers
<p>I want to have a primary CNN based classifier and a similar secondary classifier for image regions.</p>

<p>Both classifiers will be used on image regions. I need the first classifier to be used on a primary region while the secondary classifier to be used on assistive regions and will be used to support the decision made by the first classifier with further evidence.</p>

<p>Thus the primary image region and the assistive ones will be used to infer <strong>one class label at a time</strong>.</p>

<p>What other ways or architectures exist these days to perform such a task, instead of ROI Pooling?</p>

<p>Ideally, I would like to have a classifier scheme similar to the one of this paper but without the use of ROI Pooling.</p>

<p><a href="https://arxiv.org/pdf/1505.01197.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/1505.01197.pdf</a></p>