<p>I'm not sure what exactly you want. For example, would some pixels belong to two segments? If that is the case, then I'm relatively sure you have to do something on your own. Otherwise, the following might work:</p>

<h2>Opening and Closing</h2>

<p><a href="https://en.wikipedia.org/wiki/Opening_(morphology)" rel="nofollow noreferrer">Opening</a> and <a href="https://en.wikipedia.org/wiki/Closing_(morphology)" rel="nofollow noreferrer">closing</a> are two morphological operations which will smooth borders</p>

<h2>Clustering</h2>

<p>There are many clustering algorithms. They are what you want for non-semantic segmentation (for semantic segmentation, you might want to read my <a href="https://arxiv.org/pdf/1602.06541.pdf" rel="nofollow noreferrer">literature survey</a>). One example is</p>

<blockquote>
  <p>P. F. Felzenszwalb, <a href="http://cs.brown.edu/~pff/segment/" rel="nofollow noreferrer">“Graph based image segmentation.”</a></p>
</blockquote>

<p>I would simply give those algorithms a try and see if one directly works.</p>

<p>Other clustering algorithms:</p>

<ul>
<li>K-means</li>
<li>DB-SCAN</li>
<li>CLARANS</li>
<li><a href="https://stat.ethz.ch/R-manual/R-devel/library/cluster/html/agnes.html" rel="nofollow noreferrer">AGNES</a></li>
<li><a href="https://onlinelibrary.wiley.com/doi/book/10.1002/9780470316801" rel="nofollow noreferrer">DIANA</a></li>
</ul>
How to segment part of an image so that the edges are smooth?
<p>I have an input image as follows and wish to segment the parts into regions. I also want the segmented parts to not been just the pixels which contribute to the solid color but also the edge anti-aliasing between the edge of the region and the next region.</p>

<p>Does there exist any filter or method to segment the image in this way? The important part is that the end result segmented part must contain the edge anti-aliasing between it and the next regions. A correct solution is shown in yellow.</p>

<p>In these two images I zoomed the pixels to be large so the edge anti-aliasing between region edges can be seen clearly.</p>

<p><a href="https://i.stack.imgur.com/Lmakp.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/Lmakp.jpg" alt="enter image description here"></a></p>

<p>An example output that I want for the yellow region is shown.</p>

<p><a href="https://i.stack.imgur.com/VcMvv.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/VcMvv.jpg" alt="enter image description here"></a></p>

<p>For a definition of "edge anti-aliasing" see <a href="https://markpospesel.wordpress.com/2012/03/30/efficient-edge-antialiasing/" rel="nofollow noreferrer">https://markpospesel.wordpress.com/2012/03/30/efficient-edge-antialiasing/</a></p>