<p>This may be loosely related to tracking algorithms. Typically, you would use a LSTM or other algorithm coupled with a CNN to predict a human's behavior in time series images. </p>

<p>I don't see why you couldn't setup your dataset with target labels of phones vs no phones for the CNN to predict the class label. R-CNN or Yolo won't come out of the box like this so you would need to custom fit your algorithm and training set for this application. </p>

<p>Understanding human behavior is an important and active research topic for deep learning right now. Predicting behavior for a task like this is probably not as widely distributed in common libraries since these could be more domain specific tasks and the research is new, but that doesn't mean it's not possible. </p>

<p>This is a survey paper on this topic that may relate to your question: <a href="https://arxiv.org/pdf/1806.11230.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/1806.11230.pdf</a>. You may also want to look into the research going on with object tracking since it is a similar concept (but covers a wider scope than just detecting what someone is holding).</p>
Object detection using environment
<p>I'd like to ask a general question about DNN based object detection algorithms such as Yolo, SSD or R-CNN.</p>

<p>Assume I'd like to detect mobile phones on small images, where - consequently - the mobile devices themselves are super small, moreover, it's nearly impossible to detect them by only looking at those pixels which they appear on. For instance, looking at a 300x300 image, the mobile shows up on a 7x5 grid, so only by looking at the 7x5 picture no one can surely decide what can be seen there.</p>

<p>On the other hand, if we see a subway car on the picture, where a person has something black in her/his hand, we (human beings) are almost sure that the little, black 7x5 grid stands for a mobile device.</p>

<p><a href="https://i.stack.imgur.com/HncE0.jpg" rel="noreferrer"><img src="https://i.stack.imgur.com/HncE0.jpg" alt="man_with_phone"></a></p>

<p>Is my understanding right that the current state-of-the-art DNN algorithms cannot capture the environment as humans do, but they only detect objects by their physical appearance on the image? If not, can you suggest an algorithm that does not necessarily learn on a black pixel group only, but is able to capture a human being holding a black thing in her/his hand that is likely to be a phone?</p>

<p>Thanks.</p>