computing multiple fixed effects on large dataset <p>I'm trying to perform a fixed effects regression for two factor variables in a CSV dataset containing over 4000000 rows. These variables can respectively assume about 140000 and 50000 different integer values.</p> <p>I initially attempted to perform the regression using the biglm and ff packages for R as follows on a Linux machine with 8 Gb of memory; however, it seems that this requires too much memory because R complains about having to allocate a vector of a size greater than the maximum on my machine.</p> <pre><code>library(biglm) library(ff) d &lt;- read.csv.ffdf(file='data.csv', header=TRUE) model = y~factor(a)+factor(b)-1 out &lt;- biglm(model, data=d) </code></pre> <p>Some research online revealed that since factors are loaded into memory by ff, the latter will not significantly improve memory usage if many factor values are present.</p> <p>Is anyone aware of some other way to perform the aforementioned regression on a dataset of the magnitude I described without having to resort to a machine with significantly more memory?</p>