title: How can I read selected rows from a large file using the R "readLines" command and write them to a data frame?
body: <p>I am engaged in data cleaning. I have a function that identifies bad rows in a large input file (too big to read at one go, given my ram size) and returns the row numbers of the bad rows as a vector <code>badRows</code>. This function seems to work. </p>
  
  <p>I am now trying to read just the bad rows into a data frame, so far unsuccessfully. </p>
  
  <p>My current approach is to use <code>read.table</code> on an open connection to my file, using a vector of the number of rows to skip between each row that is read. This number is zero for consecutive bad rows. </p>
  
  <p>I calculate <code>skipVec</code> as: </p>
  
  <pre><code>(badRowNumbers - c(0, badRowNumbers[1:(length(badRowNumbers-1]))-1
  </code></pre>
  
  <p>But for the moment I am just handing my function a <code>skipVec</code> vector of all zeros. </p>
  
  <p>If my logic is correct, this should return all the rows. It does not. Instead I get an error:</p>
  
  <blockquote>
    <p>"Error in read.table(con, skip = pass, nrow = 1, header = TRUE, sep =
    "") :    no lines available in input"</p>
  </blockquote>
  
  <p>My current function is loosely based on a function by Miron Kursa ("mbq"), which I found <a href="https://stackoverflow.com/questions/7156770/read-lines-by-number-from-a-large-file">here</a>.</p>
  
  <p>My question is somewhat duplicative of that one, but I assume his function works, so I have broken it somehow. I am still trying to understand the difference between opening a file and opening a connection to a file, and I suspect that the problem is there somewhere, or in my use of <code>lapply</code>. </p>
  
  <p>I am running R 3.0.1 under RStudio 0.97.551 on a cranky old Windows XP SP3 machine with 3gig of ram. Stone Age, I know. </p>
  
  <p>Here is the code that produces the error message above:</p>
  
  <pre><code># Make a small small test data frame, write it to a file, and read it back in 
  # a row at a time.
  testThis.DF &lt;- data.frame(nnn=c(2,3,5), fff=c("aa", "bb", "cc"))  
  testThis.DF 
  
  # This function will work only if the number of bad rows is not too big for memory
  write.table(testThis.DF, "testThis.DF")
  con&lt;-file("testThis.DF")
  open(con)
  skipVec &lt;- c(0,0,0)
  badRows.DF  &lt;- lapply(skipVec, FUN=function(pass){
    read.table(con, skip=pass, nrow=1, header=TRUE, sep="") })
  close(con)
  </code></pre>
  
  <p>The error occurs before the close command. If I yank the readLines command out of the lapply and the function and just stick it in by itself, I still get the same error.</p>