title: Removing duplicates requires a transpose, but my dataframe is too large
body: <p>I had asked a question <a href="https://stackoverflow.com/questions/38649268/remove-duplicates-where-values-are-swapped-across-2-columns-in-r/38649301#38649301">here</a>.  I had a simple dataframe, for which I was attempting to remove duplicates. Very basic question.  </p>
  
  <p>Akrun gave a great answer, which was to use this line:  </p>
  
  <pre><code>df[!duplicated(data.frame(t(apply(df[1:2], 1, sort)), df$location)),]  
  </code></pre>
  
  <p>I went ahead and did this, which worked great on the dummy problem.  But I have 3.5 million records that I'm trying to filter.  </p>
  
  <p>In an attempt to see where the bottleneck is, I broke the code into steps.  </p>
  
  <pre><code>step1 &lt;- apply(df1[1:2], 1, sort)
  step2 &lt;- t(step1)
  step3 &lt;- data.frame(step2, df1$location)
  step4 &lt;- !duplicated(step3)
  final &lt;- df1[step4, ,]  
  </code></pre>
  
  <p>step 1 look quite a long time, but it wasn't the worst offender.<br>
  step 2, however, is clearly the culprit. </p>
  
  <p>So I'm in the unfortunate situation where I'm looking for a way to transpose 3.5 million rows in R.  (Or maybe not in R. Hopefully there is some way to do it somewhere).</p>
  
  <p>Looking around, I saw a few ideas</p>
  
  <ul>
  <li><p>install the WGCNA library, which has a <code>transposeBigData</code> function. Unfortunately this package is not longer being maintained, and I can't install all the dependencies.  </p></li>
  <li><p>write the data to a csv, then read it in line by line, and transpose each line one at a time. For me, even writing the file run overnight with no completion.  </p></li>
  </ul>
  
  <p>This is really strange. I just want to remove duplicates. For some reason, I have to transpose a dataframe in this process. But I can't transpose a dataframe this large.  </p>
  
  <p>So I need a better strategy for either removing duplicates, or for transposing.  Does anyone have any ideas on this?  </p>
  
  <p>By the way, I'm using Ubuntu 14.04, with 15.6 GiB RAM, for which <code>cat /proc/cpuinfo</code> returns  </p>
  
  <p>Intel(R) Core(TM) i7-3630QM CPU @ 2.40GHz<br>
  model name  : Intel(R) Core(TM) i7-3630QM CPU @ 2.40GHz<br>
  cpu MHz     : 1200.000<br>
  cache size  : 6144 KB  </p>
  
  <p>Thanks.</p>
  
  <pre><code>df &lt;- data.frame(id1 = c(1,2,3,4,9), id2 = c(2,1,4,5,10), location=c('Alaska', 'Alaska', 'California', 'Kansas', 'Alaska'), comment=c('cold', 'freezing!', 'nice', 'boring', 'cold'))
  </code></pre>