title: Writing/reading large vectors of data to binary file in c++
body: <p>I have a c++ program that computes populations within a given radius by reading gridded population data from an ascii file into a large 8640x3432-element vector of doubles. Reading the ascii data into the vector takes ~30 seconds (looping over each column and each row), while the rest of the program only takes a few seconds. I was asked to speed up this process by writing the population data to a binary file, which would supposedly read in faster.</p>
  
  <p>The ascii data file has a few header rows that give some data specs like the number of columns and rows, followed by population data for each grid cell, which is formatted as 3432 rows of 8640 numbers, separated by spaces. The population data numbers are mixed formats and can be just 0, a decimal value (0.000685648), or a value in scientific notation (2.687768e-05).</p>
  
  <p>I found a few examples of reading/writing structs containing vectors to binary, and tried to implement something similar, but am running into problems. When I both write and read the vector to/from the binary file in the same program, it seems to work and gives me all the correct values, but then it ends with either a "segment fault: 11" or a memory allocation error that a "pointer being freed was not allocated". And if I try to just read the data in from the previously written binary file (without re-writing it in the same program run), then it gives me the header variables just fine but gives me a segfault before giving me the vector data.</p>
  
  <p>Any advice on what I might have done wrong, or on a better way to do this would be greatly appreciated! I am compiling and running on a mac, and I don't have boost or other non-standard libraries at present. (Note: I am extremely new at coding and am having to learn by jumping in the deep end, so I may be missing a lot of basic concepts and terminology -- sorry!).</p>
  
  <p>Here is the code I came up with:</p>
  
  <pre><code># include &lt;stdio.h&gt;
  # include &lt;stdlib.h&gt;
  # include &lt;string.h&gt;
  # include &lt;fstream&gt;
  # include &lt;iostream&gt;
  # include &lt;vector&gt;
  # include &lt;string.h&gt;
  
  using namespace std;
  
  //Define struct for population file data and initialize one struct variable for reading in ascii (A) and one for reading in binary (B)
  struct popFileData
  {
      int nRows, nCol;
      vector&lt; vector&lt;double&gt; &gt; popCount; //this will end up having 3432x8640 elements
  } popDataA, popDataB;
  
  int main() {
  
      string gridFname = "sample";
  
      double dum;
      vector&lt;double&gt; tempVector;
  
      //open ascii population grid file to stream
      ifstream gridFile;
      gridFile.open(gridFname + ".asc");
  
      int i = 0, j = 0;
  
      if (gridFile.is_open())
      {
          //read in header data from file
          string fileLine;
          gridFile &gt;&gt; fileLine &gt;&gt; popDataA.nCol;
          gridFile &gt;&gt; fileLine &gt;&gt; popDataA.nRows;
  
          popDataA.popCount.clear();
  
          //read in vector data, point-by-point
          for (i = 0; i &lt; popDataA.nRows; i++)
          {
              tempVector.clear();
  
              for (j = 0; j&lt;popDataA.nCol; j++)
              {
                  gridFile &gt;&gt; dum;
                  tempVector.push_back(dum);
              }
              popDataA.popCount.push_back(tempVector);
          }
          //close ascii grid file
          gridFile.close();
      }
      else
      {
          cout &lt;&lt; "Population file read failed!" &lt;&lt; endl;
      }
  
      //create/open binary file
      ofstream ofs(gridFname + ".bin", ios::trunc | ios::binary);
      if (ofs.is_open())
      {
          //write struct to binary file then close binary file
          ofs.write((char *)&amp;popDataA, sizeof(popDataA));
          ofs.close();
      }
      else cout &lt;&lt; "error writing to binary file" &lt;&lt; endl;
  
      //read data from binary file into popDataB struct
      ifstream ifs(gridFname + ".bin", ios::binary);
      if (ifs.is_open())
      {
          ifs.read((char *)&amp;popDataB, sizeof(popDataB));
          ifs.close();
      }
      else cout &lt;&lt; "error reading from binary file" &lt;&lt; endl;
  
      //compare results of reading in from the ascii file and reading in from the binary file
      cout &lt;&lt; "File Header Values:\n";
      cout &lt;&lt; "Columns (ascii vs binary): " &lt;&lt; popDataA.nCol &lt;&lt; " vs. " &lt;&lt; popDataB.nCol &lt;&lt; endl;
      cout &lt;&lt; "Rows (ascii vs binary):" &lt;&lt; popDataA.nRows &lt;&lt; " vs." &lt;&lt; popDataB.nRows &lt;&lt; endl;
  
      cout &lt;&lt; "Spot Check Vector Values: " &lt;&lt; endl;
      cout &lt;&lt; "Index 0,0: " &lt;&lt; popDataA.popCount[0][0] &lt;&lt; " vs. " &lt;&lt; popDataB.popCount[0][0] &lt;&lt; endl;
      cout &lt;&lt; "Index 3431,8639: " &lt;&lt; popDataA.popCount[3431][8639] &lt;&lt; " vs. " &lt;&lt; popDataB.popCount[3431][8639] &lt;&lt; endl;
      cout &lt;&lt; "Index 1600,4320: " &lt;&lt; popDataA.popCount[1600][4320] &lt;&lt; " vs. " &lt;&lt; popDataB.popCount[1600][4320] &lt;&lt; endl;
  
      return 0;
  }
  </code></pre>
  
  <p>Here is the output when I both write and read the binary file in the same run:</p>
  
  <pre><code>File Header Values:
  Columns (ascii vs binary): 8640 vs. 8640
  Rows (ascii vs binary):3432 vs.3432
  Spot Check Vector Values: 
  Index 0,0: 0 vs. 0
  Index 3431,8639: 0 vs. 0
  Index 1600,4320: 25.2184 vs. 25.2184
  a.out(11402,0x7fff77c25310) malloc: *** error for object 0x7fde9821c000: pointer being freed was not allocated
  *** set a breakpoint in malloc_error_break to debug
  Abort trap: 6
  </code></pre>
  
  <p>And here is the output I get if I just try to read from the pre-existing binary file:</p>
  
  <pre><code>File Header Values:
  Columns (binary): 8640
  Rows (binary):3432
  Spot Check Vector Values: 
  Segmentation fault: 11
  </code></pre>
  
  <p>Thanks in advance for any help!</p>