<p>This is simply how <code>Mask-RCNN</code> works, and is a known side effect. Nothing you can do implementation wise to make it not appear. <a href="https://arxiv.org/pdf/1912.08193.pdf" rel="nofollow noreferrer">PointRend</a> discusses the problem (proving that it's not just you), and also proposes their own algorithm (which is an extension to <code>Mask-RCNN</code>) to solve it. In the image below you can see an image from that paper. On the top left they ran Mask-RCNN on a 28x28 image, in which you can see the staircasing as well. The other images relate to how they solve it.</p>
<p>The bad news is of course that it's not easy to just slam the PointRend code onto Mask-RCNN, I at least don't know any great implementations. The PointRend <a href="https://github.com/facebookresearch/detectron2/tree/master/projects/PointRend" rel="nofollow noreferrer">github</a> itself also doesn't allow for any retraining on your own data.</p>
<p><a href="https://i.stack.imgur.com/N7kEe.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/N7kEe.png" alt="this" /></a></p>
<p>EDIT:</p>
<p>For postprocessing, there's a lot of approaches you can take. How about this one:</p>
<p><a href="https://i.stack.imgur.com/7E3Cz.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/7E3Cz.png" alt="enter image description here" /></a></p>
Inaccurate masks with Mask-RCNN: Stairs effect and sudden stops
<p>I've been using matterport's Mask R-CNN to train on a custom dataset. However, there seem to be some parameters that i failed to correctly define because on practically all of the images, the bottom or top of the object's mask is cut off:
<a href="https://i.stack.imgur.com/YiGbq.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/YiGbq.png" alt="innacurate prediction: mask is cut-off on the bottom" /></a></p>
<p>As you can see, the bounding box is fine since it covers the whole blade, but the mask seems to suddenly stop in a horizontal line on the bottom.</p>
<p>On another hand, there is a stair-like effect on masks of larger and curvier objects such as this one (in addition to the bottom and top cut-offs):
<a href="https://i.stack.imgur.com/riTf8.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/riTf8.png" alt="enter image description here" /></a></p>
<ul>
<li>The original images are downscaled to <code>IMAGE_MIN_DIM = IMAGE_MAX_DIM = 1024</code> using the &quot;square&quot; mode.</li>
<li><code>USE_MINI_MASK</code> is set to true with <code>MINI_MASK_SHAPE = (512, 512)</code> (somehow if i set it off, RAM gets filled and training chrashes).</li>
<li><code>RPN_ANCHOR_SCALES = (64, 128, 256, 512, 1024)</code> since the objects occupy a large space of the image.</li>
</ul>
<p>It doesn't feel like the problem comes from the amount of training. These two predictions come from 6 epochs of 7000 steps per epoch (took around 17 hours). And the problem appears from early stage and persists along all the epochs.</p>
<p>Any idea on what changes to make ?</p>