<p>The answer I want to give you is: <a href="https://arxiv.org/abs/1710.09829" rel="nofollow noreferrer">CapsNets</a></p>

<p>You should definately check out the paper, where you will be introduced to some short comings of CNNs and how they tried to fix them.</p>

<p>That said, I find it hard to believe that your architecture cannot solve the problem successfully when the perspective changes. Is your dataset extremely small? I'd expect the neural network to learn filters for the riffled edges, which can be seen from all perspectives. </p>

<p>If you're not limited to one camera you could try to train a "normal" classifier, which you feed multiple images in production and average the prediction. Or you could build an architecture that takes in multiple perspectives at once. You have to try for yourself, what works best.</p>

<p>Also, never underestimate the power of old school image preprocessing. If you have 3 different perspectives, you could take the one that comes closest to the "flat" perspective. This is probably as easy as using the image with the largest colored area, where <code>img.sum()</code> is the highest.</p>

<p>Another idea is to figure out the color through explicit programming, which should be fairly easy and then feed the network a grayscale image. Maybe your network is confused by the strong correlation of the color and ignores the shape altogether.</p>
Recognize objects while falling - viewpoint variation
<p>I have a problem statement to recognize 10 classes of different variations(variations in color and size) of same object (bottle cap) while falling taking into account the camera sees different viewpoint of the object. I have split this into sub-tasks</p>

<p>1) Trained a deep learning model to classify only the flat surface of the object and successful in this attempt.</p>

<p>Flat Faces of sample 2 class</p>

<p><a href="https://i.stack.imgur.com/cz3XS.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/cz3XS.png" alt="enter image description here"></a> <a href="https://i.stack.imgur.com/492S5.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/492S5.png" alt="enter image description here"></a></p>

<p>2) Instead of taking fall into account, trained a model for possible perspective changes - not successful.</p>

<p>Perception changes of sample 2 class</p>

<p><a href="https://i.stack.imgur.com/SPbQS.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/SPbQS.png" alt="enter image description here"></a> <a href="https://i.stack.imgur.com/CMRpn.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/CMRpn.png" alt="enter image description here"></a></p>

<p>What are the approaches to recognize the object even for perspective changes. I am not constrained to arrive with a single camera solution. Open to ideas in approaching towards this problem of variable perceptions.</p>

<p>Any help could be really appreciated, Thanks in advance!</p>