<p>Indeed to do traditional cross validation with F1-score or V-Measure as scoring function you would need some labeled data as ground truth. But in this case you could just count the number of classes in the ground truth dataset and use it as your optimal value for K, hence no-need for cross-validation.</p>

<p>Alternatively you could use a <strong>cluster stability measure as unsupervised performance evaluation</strong> and do some kind of cross validation procedure for that. However this is not yet implemented in scikit-learn even though it's still on my personal todo list.</p>

<p>You can find additional info on this approach in the following <a href="http://metaoptimize.com/qa/questions/7105/quantifying-the-success-of-clustering#7111" rel="nofollow">answer on metaoptimize.com/qa</a>. In particular you should read <a href="http://arxiv.org/abs/1007.1075" rel="nofollow">Clustering Stability: An Overview by Ulrike von Luxburg</a>.</p>
k-fold Cross Validation for determining k in k-means?
<p>In a document clustering process, as a data pre-processing step, I first applied singular vector decomposition to obtain <code>U</code>, <code>S</code> and <code>Vt</code> and then by choosing a suitable number of eigen values I truncated <code>Vt</code>, which now gives me a good document-document correlation from what I read <a href="http://en.wikipedia.org/wiki/Latent_semantic_analysis" rel="nofollow noreferrer">here</a>. Now I am performing clustering on the columns of the matrix <code>Vt</code> to cluster similar documents together and for this I chose k-means and the initial results looked acceptable to me (with k = 10 clusters) but I wanted to dig a bit deeper on choosing the k value itself. To determine the number of clusters <code>k</code> in k-means, I was <a href="https://stackoverflow.com/questions/6615665/kmeans-without-knowing-the-number-of-clusters">suggested</a> to look at cross-validation. </p>

<p>Before implementing it I wanted to figure out if there is a built-in way to achieve it using numpy or scipy. Currently, the way I am performing <code>kmeans</code> is to simply use the function from scipy.</p>

<pre><code>import numpy, scipy

# Preprocess the data and compute svd
U, S, Vt = svd(A) # A is the TFIDF representation of the original term-document matrix

# Obtain the document-document correlations from Vt
# This 50 is the threshold obtained after examining a scree plot of S
docvectors = numpy.transpose(self.Vt[0:50, 0:]) 

# Prepare the data to run k-means
whitened = whiten(docvectors)
res, idx = kmeans2(whitened, 10, iter=20)
</code></pre>

<p>Assuming my methodology is correct so far (please correct me if I am missing some step), at this stage, what is the standard way of using the output to perform cross-validation? Any reference/implementations/suggestions on how this would be applied to k-means would be greatly appreciated.</p>