title: Efficient data/string reading and copy from file (CSV) c
body: <p>I have read other posts regarding copying data from files. Let me show you why my case is different.
  With C I must read 43 Million input lines from a csv file. The entries have no errors and are in this form: </p>
  
  <pre><code>int, int , int , variable length string with only alphanumerics and spaces , int \n
  </code></pre>
  
  <p>Thing is I'm copying all data to memory in array and lists to do some very very simple averages on it and then outputting all processed data to a file, nothing fancy.
  There are three main areas where I need help:</p>
  
  <ol>
  <li><p>Regarding the string, (my BIGGEST problem is here) it is first read from the file, then is copied to an array and then passed to another functions which will copy it into dynamic memory only if a condition is met. Ex:</p>
  
  <pre><code>fileAnalizer(){
    while ( ! EOF ){
      char * s = function_to_read_string_from_file();
      data_to_array(s);
    }
    ....
    ....
    processData(arrays);
    dataToFiles(arrays);
  
  }
  
  void data_to_structures(char * s){
    if ( condition is met)
      char * aux = malloc((strlen(s)+1 )* sizeof(char));
      strcpy(aux,s);
    ....
    ...
  }
  </code></pre>
  
  <p>As you can see the string is traveled through 3 times. I'm needing a way to do this process more efficiently, traveling the string fewer times. I have tried reading char by char and counting the string length but the entire process becomes slower.</p></li>
  <li><p>Efficient reading of input: would you recommend first copying all data into a buffer? If so: what type of buffer, and in many blocks or just one? 
  This is my current reading program:</p>
  
  <pre><code>void
  csvReader(FILE* f){
      T_structCDT c;
      c.string = malloc(MAX_STRING_LENGHT*sizeof(char));
      while (fscanf(f,"%d,%d,%d,%[^,],%d\n",&amp;c.a, &amp;c.b, &amp;c.vivienda, c.c, &amp;c.d)==5 ){
          data_to_structures(c);
      }
  }
  </code></pre></li>
  <li><p>I then have nearly 500 hundred csv lines of processed data to dump in to other files. How would you recommend the dumping? Line by line or again sending the data to a buffer and then doing the dump? My code now looks something like this. </p>
  
  <pre><code>void dataToFiles(arrayOfStructs immenseAr1, arrayOfStructs immenseAr2){
    for (iteration over immenseAr1) {
        fprintf(f1, "%d,%d,%s\n", immenseAr1[i].a, immenseAr1[i].b, inmenseAr1[i].string);
    }
    for (iteration over immenseAr2) {
        fprintf(f2, "%d,%d,%s\n", inmenseAr2[i].a, inmenseAr2[i].b, inmenseAr2[i].string);
    }
  }
  </code></pre></li>
  </ol>
  
  <p>I must read all data before dumping it. Would you recommend a different method other than storing all data into memory and then analyzing it and then dumping all the analyzed data?
  With the 20 Million lines the program is currently taking more than 40 seconds.
  I really need to lower that time.</p>