title: Efficiently Calculating a Euclidean Dist Matrix in Numpy?
body: <p>I have a large array (~20k entries) of two dimension data, and I want to calculate the pairwise Euclidean distance between all entries. I need the output to have standard square form. Multiple solutions for this problem have been proposed, but none of them seem to work efficiently for large arrays.</p>
  
  <p>The method using <a href="https://stackoverflow.com/questions/22720864/efficiently-calculating-a-euclidean-distance-matrix-using-numpy">complex transposing</a> fails for large arrays. </p>
  
  <p><a href="https://docs.scipy.org/doc/scipy/reference/generated/scipy.spatial.distance.pdist.html" rel="nofollow noreferrer">Scipy pdist</a> seems to be the most efficient method using numpy. However, using <a href="https://docs.scipy.org/doc/scipy-0.14.0/reference/generated/scipy.spatial.distance.squareform.html" rel="nofollow noreferrer">squareform</a> on the result to obtain a square matrix makes it very inefficient. </p>
  
  <p>So the best I could come up with is using <a href="https://docs.scipy.org/doc/scipy/reference/generated/scipy.spatial.distance.cdist.html" rel="nofollow noreferrer">Scipy cdist</a>, which is somewhat awkward, as it does calculate every pairwise distance twice. The provided time measurements show the advantage of pdist for the raw distance calculation.</p>
  
  <p>Complex: 49.605 s</p>
  
  <p>Cdist: 4.820 s </p>
  
  <p>Pdist 1.785 s</p>
  
  <p>Pdist with squareform 10.212 s</p>