title: vector<string> together with map<string,int> possible failure at deallocation? wondering memory usage detail
body: <p>I am writing in c++ handling large amounts of streaming data containing information of millioins of nodes. I use vector to store each node's name and map for index.</p>
  
  <p>Now the problem is that vector is taking much more memory than expected and their destruction is unexplainable.</p>
  
  <p>Suppose somefile contains 1 million lines, each line more than 50 characters. Read them in twice and then check the memory usage of the process and the estimated memory usage by the vector. They differ in 60 MB. This is just a small projection of the bigger problem I have, and it may differ in GB scale.</p>
  
  <p>I use VS2010 SP1 on Windows7 SP1 Ultimate 64bit compiling the program with x86 settings.</p>
  
  <pre><code>#include &lt;iostream&gt;
  #include &lt;string&gt;
  #include &lt;vector&gt;
  #include &lt;map&gt;
  #include &lt;fstream&gt;
  #include &lt;Windows.h&gt;
  #include &lt;Psapi.h&gt;
  using namespace std;
  
  //#define COUNT 500000
  int COUNT = 0;
  
  vector&lt;string&gt; namesVector;
  map&lt;string,int&gt; namesMap;
  void ProcessStatistics()
  {   
      PROCESS_MEMORY_COUNTERS memCounter;
      GetProcessMemoryInfo(GetCurrentProcess(),&amp;memCounter,sizeof(memCounter));
      cout&lt;&lt;"Mem Usage by Process: "&lt;&lt;memCounter.WorkingSetSize * 1.0e-6f&lt;&lt;" MB."&lt;&lt;endl;
  }
  void VectorMemUsage()
  {   
      COUNT = namesVector.size();
      int overhead = StringOverhead();
      double mem = 0;
      mem += sizeof(vector&lt;string&gt;);
      mem += overhead*COUNT;
      for(int i=0; i&lt;COUNT; i++)
      {
          mem += namesVector[i].capacity();
      }
      cout&lt;&lt;"Calculated String Vector Usage: "&lt;&lt;mem * 1.0e-6f&lt;&lt;" MB of "&lt;&lt;COUNT&lt;&lt;" strings."&lt;&lt;endl;
  }
  int StringOverhead()
  {
      int overhead = sizeof(string);
      cout&lt;&lt;"String overhead: "&lt;&lt;overhead&lt;&lt;" Bytes."&lt;&lt;endl;   
      return overhead;
  }
  
  void main(){
      const std::string infile = "somefile";
      ifstream infstream(infile);
      string s;   
      while(getline(infstream,s) != NULL)
      {
          namesVector.push_back(s);
          //namesMap.insert(pair&lt;string,int&gt;(s,namesVector.size()));
      }
      infstream.clear();
      infstream.seekg(0,ios::beg);    
      while(getline(infstream,s) != NULL)
      {
          namesVector.push_back(s);
          //namesMap.insert(pair&lt;string,int&gt;(s,namesVector.size()));
      }
      //Check process and vector memory usage:
      ProcessStatistics();
      VectorMemUsage();
      System("pause");
  
      //Release the vector.
      cout&lt;&lt;"Now releasing the memory..."&lt;&lt;endl;        
      //vector&lt;string&gt;(namesVector).swap(namesVector);
      //vector&lt;string&gt;().swap(namesVector); //Deallocate Vector
  //map&lt;string,int&gt;().swap(namesMap);   //Deallocate Map
      cout&lt;&lt;"Capacity of vector "&lt;&lt;namesVector.capacity()&lt;&lt;endl;
      ProcessStatistics();
   }
  </code></pre>
  
  <p>The x86 version of the program output goes:</p>
  
  <pre><code>Mem Usage by Process: 336.523 MB.
  String overhead: 28 Bytes.
  Calculated String Vector Usage: 301.599 MB of 3385108 strings.
  Press any key to continue . . .
  Now releasing the memory...
  Mem Usage by Process: 7.64314 MB.
  </code></pre>
  
  <p>When I call namesVector.shrink_to_fit() or vector(namesVector).swap(namesVector) idiom on vector, the vector capacity really decreases, but the memory usage went high, any one has idea to fix this? Is the swap trick supposed to be pointer swap? why would it involve memory copy and all and cause this?</p>
  
  <pre><code>Mem Usage by Process: 336.536 MB.
  String overhead: 28 Bytes.
  Calculated String Usage: 301.599 MB of 3385108 strings.
  Vector Capacity is 3543306.
  Calculated String Vector Usage: 315.693 MB of 3385108 strings.
  Now releasing the memory...
  Capacity of vector 3385108
  Mem Usage by Process: 434.5 MB.
  </code></pre>
  
  <p>When I add a map for string index, unexpected behavior happened. When I call both vector().swap(namesVector) and map().swap(namesMap), the result is like this, which is pretty fine because the memory is released.</p>
  
  <pre><code>Mem Usage by Process: 534.778 MB.
  String overhead: 28 Bytes.
  Calculated String Usage: 301.599 MB of 3385108 strings.
  Vector Capacity is 3543306.
  Calculated String Vector Usage: 315.693 MB of 3385108 strings.
  Press any key to continue . . .
  Now releasing the memory...
  Capacity of vector 0
  Mem Usage by Process: 8.2903 MB.
  </code></pre>
  
  <p>But when I only call vector().swap(namesVector), the memory is partially released. By partial I mean it released less than the result above, which around 336 MB.</p>
  
  <pre><code>Mem Usage by Process: **534.77** MB.
  String overhead: 28 Bytes.
  Calculated String Usage: 301.599 MB of 3385108 strings.
  Vector Capacity is 3543306.
  Calculated String Vector Usage: 315.693 MB of 3385108 strings.
  Press any key to continue . . .
  Now releasing the memory...
  Capacity of vector 0
  Mem Usage by Process: **440.459** MB.
  </code></pre>
  
  <p>Or map().swap(namesMap), the memory is close to not released at all.</p>
  
  <pre><code>Mem Usage by Process: **534.774** MB.
  String overhead: 28 Bytes.
  Calculated String Usage: 301.599 MB of 3385108 strings.
  Vector Capacity is 3543306.
  Calculated String Vector Usage: 315.693 MB of 3385108 strings.
  Press any key to continue . . .
  Now releasing the memory...
  Capacity of vector 3543306
  Mem Usage by Process: **535.441** MB.
  </code></pre>
  
  <p>I can't explain what happened. Anyone has any idea what happened here?</p>
  
  <p>Thanks for the help.</p>
  
  <p>Best.</p>