title: Docker daemon memory leak due to logs from long running process
body: <p>I have the following setup:</p>
  
  <ol>
  <li>Perl service running in a container and writing logs out to <code>STDERR</code></li>
  <li><a href="https://github.com/progrium/logspout" rel="nofollow noreferrer">logspout</a> to ship those logs out to a remote server for archiving</li>
  </ol>
  
  <p>in a 600 MB RAM machine.</p>
  
  <p>I also truncate the logs periodically at:</p>
  
  <pre><code>/var/lib/docker/containers/CID/CID-json.log
  </code></pre>
  
  <p>as suggested <a href="https://serverfault.com/questions/637996/clearing-deleting-docker-logs">here</a> to avoid 100% disk scenarios.</p>
  
  <p><strong>Problem</strong></p>
  
  <p>Docker daemon starts of with low memory usage, 1% initially and slowly increases to 40% after 2 days of running the container. </p>
  
  <p><strong>Reference</strong></p>
  
  <p>Docker daemon memory leak has been talked about in <a href="https://github.com/docker/docker/issues/5923" rel="nofollow noreferrer">this issue</a> and <a href="https://github.com/docker/docker/issues/1931" rel="nofollow noreferrer">this issue</a>. But both of them are closed now saying merged at a commit. Am running the <em>latest major version</em> of docker (Docker version 1.4.0, build 4595d4f), but still face a monotonically increasing memory usage issue.</p>
  
  <p>EDIT: I did this experiment: Just run a bash process in the container, print out a lot of lines to STDERR, docker daemon process's memory usage accelerates very quickly</p>
  
  <p>Does docker do some <a href="https://github.com/docker/docker/issues/1385" rel="nofollow noreferrer">log buffering</a> and doesn't release memory even if underlying log file (<code>/var/lib/docker/containers/CID/CID-json.log</code>) is cleared?</p>
  
  <p>There's apparently <a href="https://github.com/docker/docker/issues/6630" rel="nofollow noreferrer">no way</a> to clear the logs. Will <a href="https://github.com/docker/docker/pull/9753/commits" rel="nofollow noreferrer">this commit</a> solve this issue for long running tasks? </p>
  
  <p>I don't know why docker daemon's memory usage keeps increasing. How do I debug this issue?</p>