 <p>Judging on your "desired output" image, your usage of kmeans is wrong. The pixel coordinates should play no role in the clustering. You should only hand the color triplets to kmeans.</p> <pre><code>sample = cvCreateMat( image-&gt;height*image-&gt;width, 3, CV_32FC1 ); </code></pre> <p>…</p> <pre><code> for(j=0;j&lt;image-&gt;width;j++) { b = data[i*image-&gt;widthStep + j*image-&gt;nChannels +0]; g = data[i*image-&gt;widthStep + j*image-&gt;nChannels +1]; r = data[i*image-&gt;widthStep + j*image-&gt;nChannels +2]; cvSetReal2D( sample, k, 0, b); cvSetReal2D( sample, k, 1, g); cvSetReal2D( sample, k, 2, r); k++; } </code></pre> <p>Then you also have an indexing bug further down the road when setting data.</p>