title: Computing cosine similarities on a large corpus in R using quanteda
body: <p>I am trying to work with a very large corpus of about 85,000 tweets that I'm trying to compare to dialog from television commercials. However, due to the size of my corpus, I am unable to process the cosine similarity measure without getting the "Error: cannot allocate vector of size n" message, (26 GB in my case). </p>
  
  <p>I am already running R 64 bit on a server with lots of memory. I've also tried using the AWS on the server with the most memory, (244 GB), but to no avail, (same error). </p>
  
  <p>Is there a way to use a package like fread to get around this memory limitation or do I just have to invent a way to break up my data? Thanks much for the help, I've appended the code below:</p>
  
  <pre><code>x &lt;- NULL
  y &lt;- NULL
  num &lt;- NULL
  z &lt;- NULL
  ad &lt;- NULL
  for (i in 1:nrow(ad.corp$documents)){
    num &lt;- i
    ad &lt;- paste("ad.num",num,sep="_")
    x &lt;- subset(ad.corp, ad.corp$documents$num== yoad)
    z &lt;- x + corp.all
    z$documents$texts &lt;- as.character(z$documents$texts)
    PolAdsDfm &lt;- dfm(z, ignoredFeatures = stopwords("english"), groups = "num",stem=TRUE, verbose=TRUE, removeTwitter=TRUE)
    PolAdsDfm &lt;- tfidf(PolAdsDfm)
    y &lt;- similarity(PolAdsDfm, ad, margin="documents",n=20, method = "cosine", normalize = T)
    y &lt;- sort(y, decreasing=T)
    if (y[1] &gt; .7){assign(paste(ad,x$documents$texts,sep="--"), y)}
    else {print(paste(ad,"didn't make the cut", sep="****"))}  
  }
  </code></pre>