title: Abort in glibc while trying to use sbrk to reduce the size of the data segment
body: <p>While working with glibc I tried to reduce the data segment using <a href="https://www.gnu.org/software/libc/manual/html_mono/libc.html#Resizing-the-Data-Segment" rel="nofollow noreferrer">sbrk</a> using a negative parameter, and found a most strange behaviour.</p>
  <p>I first <code>malloc</code>, then <code>free</code> it, then reduce data segment with <code>sbrk</code>, and then <code>malloc</code> again with same size as the first one.</p>
  <p>The issue is, if the <code>malloc</code> size (both <code>malloc</code>s with same size) is small enough (32k, or eight 4k pages) then everything works fine. But when I increase a little the <code>malloc</code>-<code>free</code>-<code>malloc</code> size (to nine 4k pages) then I get the core dump. What is even more strange is that when I raise the <code>malloc</code> size to cross the <code>mmap</code> threshold (128k) then I get the adjust-abort behaviour.</p>
  <p>The C code:</p>
  <pre><code>#define _GNU_SOURCE 1
  #include &lt;stdio.h&gt;
  #include &lt;malloc.h&gt;
  #include &lt;unistd.h&gt;
  
  // set MMAP_ALLOC_SIZE to 8 4k-pages it will work, 
  // set it to 9 4k-pages it raises a 'segmentation fault (core dumped)'
  // set it to 33 4k-pages it raises a 'break adjusted to free malloc space' and 'abort (core dumped)'
  #define MMAP_ALLOC_SIZE   (33 * 4096)
  #define PRINT_MEM { \
      struct mallinfo mi; \
      mi = mallinfo(); \
      printf(&quot;ptr    %p\n&quot;, ptr); \
      printf(&quot;brk(0) %p\n&quot;, sbrk(0)); \
      printf(&quot;heap   %d bytes\n&quot;, mi.arena); \
      printf(&quot;mmap   %d bytes\n\n&quot;, mi.hblkhd); \
  }
  
  int main(int argc, char *argv[])
  {
      void *ptr;
      ptr = NULL;                      PRINT_MEM
      printf(&quot;1) will malloc &gt; MMAP_THRESHOLD (128 KiB) ...\n&quot;);
      ptr = malloc(MMAP_ALLOC_SIZE);   PRINT_MEM
      printf(&quot;2) will free malloc ...\n&quot;);
      free(ptr);                       PRINT_MEM
      printf(&quot;3) will reduce brk  ...\n&quot;);
      ptr = sbrk(-100000);             PRINT_MEM
      printf(&quot;4) will malloc &gt; MMAP_THRESHOLD (128 KiB) ... \n&quot;);
      ptr = malloc(MMAP_ALLOC_SIZE);   PRINT_MEM
      printf(&quot;5) completion.\n&quot;); // never happens if MMAP_ALLOC_SIZE is &gt; 8 4k-pages
      return 0;
  }
  </code></pre>
  <p>Compiled with:</p>
  <pre><code>gcc -Wall testbrk.c -o testbrk
  </code></pre>
  <p>Which gives the successful output for <code>MMAP_ALLOC_SIZE (8 * 4096)</code>:</p>
  <pre><code>ptr    (nil)
  brk(0) 0xf46000
  heap   0 bytes
  mmap   0 bytes
  
  1) will malloc &gt; MMAP_THRESHOLD (128 KiB) ...
  ptr    0xf25670
  brk(0) 0xf46000
  heap   135168 bytes
  mmap   0 bytes
  
  2) will free malloc ...
  ptr    0xf25670
  brk(0) 0xf46000
  heap   135168 bytes
  mmap   0 bytes
  
  3) will reduce brk  ...
  ptr    0xf46000
  brk(0) 0xf2d960
  heap   135168 bytes
  mmap   0 bytes
  
  4) will malloc &gt; MMAP_THRESHOLD (128 KiB) ... 
  ptr    0xf25670
  brk(0) 0xf2d960
  heap   135168 bytes
  mmap   0 bytes
  
  5) completion.
  </code></pre>
  <p>The following abort output for <code>MMAP_ALLOC_SIZE (9 * 4096)</code>:</p>
  <pre><code>ptr    (nil)
  brk(0) 0x1b7f000
  heap   0 bytes
  mmap   0 bytes
  
  1) will malloc &gt; MMAP_THRESHOLD (128 KiB) ...
  ptr    0x1b5e670
  brk(0) 0x1b7f000
  heap   135168 bytes
  mmap   0 bytes
  
  2) will free malloc ...
  ptr    0x1b5e670
  brk(0) 0x1b7f000
  heap   135168 bytes
  mmap   0 bytes
  
  3) will reduce brk  ...
  ptr    0x1b7f000
  brk(0) 0x1b66960
  heap   135168 bytes
  mmap   0 bytes
  
  4) will malloc &gt; MMAP_THRESHOLD (128 KiB) ... 
  Segmentation fault (core dumped)
  </code></pre>
  <p>And the following adjust-abort output for <code>MMAP_ALLOC_SIZE (33 * 4096)</code>:</p>
  <pre><code>ptr    (nil)
  brk(0) 0x1093000
  heap   0 bytes
  mmap   0 bytes
  
  1) will malloc &gt; MMAP_THRESHOLD (128 KiB) ...
  ptr    0x7fdd1c7f6010
  brk(0) 0x1093000
  heap   135168 bytes
  mmap   139264 bytes
  
  2) will free malloc ...
  ptr    0x7fdd1c7f6010
  brk(0) 0x1093000
  heap   135168 bytes
  mmap   0 bytes
  
  3) will reduce brk  ...
  ptr    0x1093000
  brk(0) 0x107a960
  heap   135168 bytes
  mmap   0 bytes
  
  4) will malloc &gt; MMAP_THRESHOLD (128 KiB) ... 
  break adjusted to free malloc space
  Aborted (core dumped)
  </code></pre>
  <p>So the <code>sbrk</code> reduction call works without error, but the subsequent <code>malloc</code> raises a core dump even if enough memory was still available.</p>
  <p>I'm I doing something wrong or is this a limitation in the data segment resize?</p>
  <p><strong>EDIT</strong>: Besides the code solution using <code>malloc_trim()</code> I've posted below, very welcome from the accepted answer, there are some important things to know about this issue, as recovered from <a href="https://chat.stackoverflow.com/rooms/240905/discussion-on-question-by-arc-abort-in-glibc-while-trying-to-use-sbrk-to-reduce">chat</a>:</p>
  <p>First, the <code>man</code> page says avoid using <code>sbrk</code>, but the glibc manual does not.</p>
  <p><code>malloc.c</code> from glibc does contains comments on <code>sbrk</code> possibly being called with signed integer - and thus negative integer - values of memory parameter reduction, and does have provisions on the callings of <code>brk</code> and <code>sbrk</code> to be coherent with <code>malloc</code>. The workings of <code>malloc</code> are not unsensitive to <code>sbrk</code>, they are not from &quot;different levels&quot; of coding, they are supposed to work together in harmony, at least by the code comments. Also, my first test case does work well, which means that <code>sbrk</code> is not an issue itself to work with <code>malloc</code>, but is only not treated in some specific cases.</p>
  <p>And, finally, it matters that someone can break glibc allocation, it might be a security loophole. For instance, a hacker could use some instance of the glibc being accessed by another layer of inderection in order to summon <code>sbrk</code> in order to cause library crash. I'm not a security expert, but given that glibc has so many different uses around the planet, it could be in principle that some malicious programmer use this <code>sbrk</code> crash to gain access to unprotected systems. Not sure, but sure it should be investigated by glibc developers.</p>
  <p>And I'm sure this is not a frivolous question.</p>