title: Fragmentation in C++ with big chunks of data (Windows)
body: <p>I have been developing my program using <code>malloc()</code> to allocate memory. However, my investigations made me think that I am facing a memory fragmentation problem.</p>
  
  <p>My program needs 5 memory allocations of ~70 MB each. When I run my program using 4 threads, I need 5x4 memory allocations of ~70 MB each (and I cannot use less memory). At the end, I want to be able to use the 8 cores of my i7, this is, 5x8 memory allocations.</p>
  
  <p>If I do 5x2 malloc()s, the program works. Not for 5x3 malloc()s.</p>
  
  <p>I have been reading about <code>std::vector</code> and <code>std::deque</code>. I believe that <code>std::deque</code> is my solution for this problem, as <code>std::vector</code> allocates a big chunk of consecutive memory as <code>malloc()</code> does.</p>
  
  <p>There any other solutions to explore or <code>std::deque</code> is my only solution?</p>
  
  <hr>
  
  <h2>EDIT</h2>
  
  <p>OS: Windows 8.1 (x64)</p>
  
  <p>RAM: 8 GB (5 GB of free space)</p>
  
  <p>I detect <code>malloc()</code> errors by checking <code>errno == ENOMEM</code></p>
  
  <hr>
  
  <p>NOTE: <code>ERROR_MEM_ALLOC_FAILED</code> is one of the errors I generate when memory allocation fails.</p>
  
  <p>A debug trace for the program with 4 threads (i.e. 5x4 <code>malloc()</code>s):</p>
  
  <pre><code>Start
  
  Thread 01
  (+53.40576 MB) Total allocated  53.4/4095 total MB
  (+53.40576 MB) Total allocated 106.8/4095 total MB
  (+0.00008 MB)  Total allocated 106.8/4095 total MB
  (+0.00008 MB)  Total allocated 106.8/4095 total MB
  Tried to allocate 267 MB
  ERROR_MEM_ALLOC_FAILED
  
  Thread 02
  (+53.40576 MB) Total allocated 160.2/4095 total MB
  (+53.40576 MB) Total allocated 213.6/4095 total MB
  (+0.00008 MB)  Total allocated 213.6/4095 total MB
  (+0.00008 MB)  Total allocated 213.6/4095 total MB
  Tried to allocate 267 MB
  ERROR_MEM_ALLOC_FAILED
  
  Thread 03
  (+53.40576 MB) Total allocated 267.0/4095 total MB
  Tried to allocate 53 MB
  ERROR_MEM_ALLOC_FAILED
  
  Thread 04
  Tried to allocate 53 MB
  ERROR_MEM_ALLOC_FAILED
  
  End of program
  </code></pre>
  
  <p>I tried to run the same thing but changing the order of the memory allocations, but no memory was allocated.</p>
  
  <pre><code>Start
  
  Thread 01
  Tried to allocate 267 MB
  ERROR_MEM_ALLOC_FAILED
  
  Thread 02
  Tried to allocate 267 MB
  ERROR_MEM_ALLOC_FAILED
  
  Thread 03
  Tried to allocate 267 MB
  ERROR_MEM_ALLOC_FAILED
  
  Thread 04
  Tried to allocate 267 MB
  ERROR_MEM_ALLOC_FAILED
  
  End of program
  </code></pre>
  
  <hr>
  
  <h2>SOLUTION</h2>
  
  <p>The solution was to compile the application as a 64-bit application. Hence, probably it was not a fragmentation problem.</p>