<p>Yaw Lin's answer contains a good amount of information, I'll just build on what he said in his last paragraph. I think the essence of what you want to do is not so much to process the person and the background independently and compare the results (that's clearly what you said you're doing), but to process the background first and infer from it the kinds of expectations you have for the primary region. Once you have some expectations, you can compare the primary region to the most significant expectations. </p>

<p>For example, from Figure 1 (b) in your Arxiv link, if you can process the background and determine that it's outdoors in a highly populated region, then you can focus a lot of the probability density function of what the person is doing in social outdoor activities, making jogging much more likely as a guess before you even process the figure you're interested in. In contrast, for Figure 1 (a), if you can process the background and tell that it's indoors and contains computers, then you can focus probability on lone indoor computer-based activities, skyrocketing the probability of "working on a computer".</p>
Ways to combine two deep learning based classifiers
<p>I want to have a primary CNN based classifier and a similar secondary classifier for image regions.</p>

<p>Both classifiers will be used on image regions. I need the first classifier to be used on a primary region while the secondary classifier to be used on assistive regions and will be used to support the decision made by the first classifier with further evidence.</p>

<p>Thus the primary image region and the assistive ones will be used to infer <strong>one class label at a time</strong>.</p>

<p>What other ways or architectures exist these days to perform such a task, instead of ROI Pooling?</p>

<p>Ideally, I would like to have a classifier scheme similar to the one of this paper but without the use of ROI Pooling.</p>

<p><a href="https://arxiv.org/pdf/1505.01197.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/1505.01197.pdf</a></p>