What's the best way to index a large number of arrays for calculating the cosine similarity? <p>Goal: calculating the most relevant article using cosine similarity. </p> <p>I have a function that will produce an array of scores of different topics for each article. For example:</p> <p>[0.6, 0.7, 1] represent the article's scores in the [math, physics, bio] category.</p> <p>With a given article, I will find the most relevant article by calculating the cosine similarity between the array of given article against the rest of the arrays of other articles.</p> <p>I am thinking of using pickle to store my data and use numpy to calculate cosine similarity and it can find cosine similarity very fast with matrix manipulation. However I am running into a decision problem as of how to index each array with its corresponding article id. </p> <p>I have thought of storing each array into a dictionary with the key as the article id and the value as the array. However, i think that will make it hard for me to calculate cosine similarity among a large number of arrays.</p> <p>Any suggestions and ideas would be greatly appreciated, thanks!</p>