<p>Your task is related to object detection. The difference is, that you seem to have only one object in each of your images, whereas in detection there may be multiple objects or no object present. For object detection, there are networks such as YOLOv3 (<a href="https://pjreddie.com/media/files/papers/YOLOv3.pdf" rel="nofollow noreferrer">https://pjreddie.com/media/files/papers/YOLOv3.pdf</a>) or Single Shot Multibox Detector - SSD (<a href="https://arxiv.org/pdf/1512.02325.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/1512.02325.pdf</a>) but also ResNet can be trained as an object detection network (as in this paper: <a href="https://arxiv.org/pdf/1506.01497.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/1506.01497.pdf</a>)</p>

<p>I will shortly describe how YOLO solves the regression problem for bounding box x,y coordinates: </p>

<ul>
<li>YOLO uses a sigmoid activation function for x,y</li>
<li>It devides the image into grid cells and predicts offsets for a potential object in each grid-cell. This may be helpful in case you have large images or objects at multiple locations.</li>
<li>The original paper uses MSE as a loss function, but in my favorite keras-reimplementation they use crossentropy loss with the Adam optimizer.</li>
</ul>

<p>In principle your setup looks fine to me. But there are many things which could result in poor performance, since you don't tell about the domain of your dataset: Are you using a pretrained network or are you training from scratch? Is it a new category which you are to learning or an object category the network has seen before? etc. </p>

<p>Here are some ideas which you could try:</p>

<ul>
<li>change the optimizer (to SGD or Adam)</li>
<li>change the learning rate (better smaller than too large) </li>
<li>increase your dataset size. For retraining the network for a new object category my rule of thumb is to use about 500-1000 images. For retraining from scratch you need orders of magnitude more.</li>
<li>you may want to check out YOLO or SSD and modify those networks for your case</li>
</ul>

<p>I hope you find some inspiration for your solution.</p>
Can I use a Sigmoid activation for my output layer, even if my CNN model is doing a regression?
<p>Final objective: Object Midpoint calculation. </p>

<p>I have a small dataset (around 120 images), which has an object (the same in all cases), and the labels are the normalized x,y coordinates of the midpoint of the object in the image (always between 0 and 1)</p>

<p>e.g. x = image_005 ; y = (0.1, 0.15) for an image with the object placed near the bottom left corner</p>

<p>I am trying to use a ResNet architecture but customized for my image-size (all are identical images). Since the output values are always between 0 and 1, for both coordinates, I was wondering if it is possible to use Sigmoid activation in my last layer: </p>

<pre><code> X = Dense(2, activation='sigmoid', name='fc', kernel_initializer = glorot_uniform(seed=0))(X)
</code></pre>

<p>instead of a linear activation (as is advised often when you are trying to achieve a regression result)</p>

<p>For the loss function, I use MSE, with 'rmsprop' optimizer and in addition to accuracy and MSE, I have written a custom metric for telling me if the predicted points are off from the labels by more than 5%</p>

<pre><code>model.compile(optimizer='rmsprop', loss='mean_squared_error', metrics=['mse','acc',perc_midpoint_err])
</code></pre>

<p>I am not getting good results, after training the model on around 150 epochs (I experimented with different batch sizes too)</p>

<p>Should I change the activation layer to linear? Or is there a different modification I can do to my model? Or is ResNet completely unsuitable for this task?</p>