Take the 2-minute tour ×

I have a large amount of Agilent data (580 text files that are ~200 MB in size each) that I'd like to normalize. I've run into a problem in that my machine (with 8 GB of memory) still runs out of memory when I try to read the files using marray in order to geneerate an expression set for normalization. I don't need the expression set, just the normalized output--is there an analogous program for Agilent data like justRMA is used for Affy data?

share|improve this question
This question would be better suited to biostars since it is a completely technical problem related only to computers and not to the biology side of bioinformatics. That said, you could always run your R script in a loop, once for each file couldn't you? Something like for n in *.data; do Rscript script.R $n > $n.norm; done. –  terdon Jan 31 '13 at 18:03
See the "Large memory and out-of-memory data" section of cran.r-project.org/web/views/HighPerformanceComputing.html –  kmm Jan 31 '13 at 18:29
You could also perhaps ask here... stats.stackexchange.com –  Ben Jan 31 '13 at 19:20
@terdon: that wouldn't work, "the usual" array normalization methods pretty much require working on all of the data together (so it needs to be loaded in memory). That having been said, this question should definitely be asked on the bioconductor mailing-lsit, biostars.org. –  Steve Lianoglou Jan 31 '13 at 19:52
Would a "normalization of normals" work? Normalize 30 or so files at a time, then normalize the ~20 resulting normals. You may also be able to pick certain families of data points to normalize against each other, so you're only working with a fraction of the data at a time. Putting your data into a Hadoop (or similar) database is also an option. –  MattDMo Jan 31 '13 at 20:05
show 1 more comment

closed as off topic by terdon, jonsca, Rory M Feb 1 '13 at 16:03


1 Answer

You might want to consider using a one-sample-at-a-time normalization scheme like frozenRMA: http://www.bioconductor.org/packages/2.11/bioc/html/frmaTools.html

share|improve this answer
add comment
