title: Import Large Unusual File To R
body: <p>First time poster here, so I'll try and make myself as clear as possible on the help I need. I'm fairly new to R, and this is my first real independent programming experience. </p>
  
  <p>I have stock tick data for about 2.5 years, each day has its own file. The files are .txt and consist of approximately 20-30 million rows, and averaging I guess 360mb each. I am working one file at a time for now. I don't need all the data these files contain, and I was hoping that I could use the programming to minimize my files a bit. </p>
  
  <p>Now my problem is that I am having some difficulties with writing the proper code so R understands what I need it to do.</p>
  
  <p>Let me first show you some of the data so you can get an idea of the formatting. </p>
  
  <pre><code>M977
  R 64266NRE1VEW107 FI0009653869 2EURXHEL 630 1
  R 64516SSA0B 80SHB SE0002798108 8SEKXSTO 40 1
  R 645730BBREEW750 FR0010734145 8EURXHEL 640 1
  R 64655OXS1C 900SWE SE0002800136 8SEKXSTO 40 1
  R 64663OXS1P 450SWE SE0002800219 8SEKXSTO 40 1
  R 64801SSIEGV LU0362355355 11EURXCSE 160 1
  M978
  </code></pre>
  
  <p>Another snip of data: </p>
  
  <pre><code>M732
  D 3547742
  A 3551497B 200000 67110 02800
  D 3550806
  D 3547743
  A 3551498S 250000 69228 09900
  </code></pre>
  
  <p>So as you can see each line begins with a letter. Each letter denotes what the line means. For instance <code>R</code> means order book directory message, <code>M</code> means milliseconds after last second, <code>H</code> means stock trading action message. There are 14 different letters used in total. </p>
  
  <p>I have used the <code>readLines</code> function to import the data into R. This however seems to take a very long time for R to process when I want to work with the data.</p>
  
  <p>Now I would like to write some sort of If function that says if the first letter is <code>R</code> then from offset 1 to 4 the code means Market Segment Identifier etc., and have R add columns to these so I can work with the data in a more structured fashion. </p>
  
  <p>What is the best way of importing such data, and also creating some form of structure - i.e. use unique ID information in the line of data to analyze 1 stock at a time for instance.</p>