title: Diagnosing high CPU usage on Docker for Mac
body: <p>How do I diagnose the cause of Docker on MacOS, specifically <code>com.docker.hyperkit</code> using 100% of CPU?</p>
  
  <p><a href="https://i.stack.imgur.com/N0ftg.png" rel="noreferrer"><img src="https://i.stack.imgur.com/N0ftg.png" alt="docker CPU usage"></a></p>
  
  <h3>Docker stats</h3>
  
  <p>Docker stats shows all the running containers have low CPU, memory, net IO and block IO.</p>
  
  <p><a href="https://i.stack.imgur.com/YhAui.png" rel="noreferrer"><img src="https://i.stack.imgur.com/YhAui.png" alt="docker stats output"></a></p>
  
  <h3>iosnoop</h3>
  
  <p>iosnoop shows that <code>com.docker.hyperkit</code> performs about 50 writes per second totaling 500KB per second to the file <code>Docker.qcow2</code>.  According to <a href="https://stackoverflow.com/questions/49887747/what-is-docker-qcow2">What is Docker.qcow2?</a>, <code>Docker.qcow2</code> is a sparse file that's the persistent storage for all Docker containers.</p>
  
  <p>In my case the file isn't that sparse. The physical size matches the logical size.</p>
  
  <p><a href="https://i.stack.imgur.com/aixXT.png" rel="noreferrer"><img src="https://i.stack.imgur.com/aixXT.png" alt="docker.qcow actual size"></a></p>
  
  <h3>dtrace (dtruss)</h3>
  
  <p>dtruss <code>sudo dtruss -p $DOCKER_PID</code> shows a large number of <code>psynch_cvsignal</code> and <code>psynch_cvwait</code> calls.</p>
  
  <pre><code>psynch_cvsignal(0x7F9946002408, 0x4EA701004EA70200, 0x4EA70100)          = 257 0
  psynch_mutexdrop(0x7F9946002318, 0x5554700, 0x5554700)           = 0 0
  psynch_mutexwait(0x7F9946002318, 0x5554702, 0x5554600)           = 89474819 0
  psynch_cvsignal(0x10BF7B470, 0x4C8095004C809600, 0x4C809300)             = 257 0
  psynch_cvwait(0x10BF7B470, 0x4C8095014C809600, 0x4C809300)               = 0 0
  psynch_cvwait(0x10BF7B470, 0x4C8096014C809700, 0x4C809600)               = -1 Err#316
  psynch_cvsignal(0x7F9946002408, 0x4EA702004EA70300, 0x4EA70200)          = 257 0
  psynch_cvwait(0x7F9946002408, 0x4EA702014EA70300, 0x4EA70200)            = 0 0
  psynch_cvsignal(0x10BF7B470, 0x4C8097004C809800, 0x4C809600)             = 257 0
  psynch_cvwait(0x10BF7B470, 0x4C8097014C809800, 0x4C809600)               = 0 0
  psynch_cvwait(0x10BF7B470, 0x4C8098014C809900, 0x4C809800)               = -1 Err#316
  </code></pre>
  
  <h3>Update: <code>top</code> on Docker host</h3>
  
  <p>From <a href="https://stackoverflow.com/a/58293240/30900">https://stackoverflow.com/a/58293240/30900</a>:</p>
  
  <pre><code>docker run -it --rm --pid host busybox top
  </code></pre>
  
  <p>The CPU usage on docker embedded host is ~3%.  CPU usage on my MacBook was ~100%.  So, the docker embedded host isn't causing the CPU usage spike.</p>
  
  <p><a href="https://i.stack.imgur.com/vkbdV.jpg" rel="noreferrer"><img src="https://i.stack.imgur.com/vkbdV.jpg" alt="docker host top"></a></p>
  
  <h3>Update: running dtrace scripts of most common stack traces</h3>
  
  <p>Stack traces from the dtrace scripts in the answer below: <a href="https://stackoverflow.com/a/58293035/30900">https://stackoverflow.com/a/58293035/30900</a>.</p>
  
  <p>These kernel stack traces look innocuous.</p>
  
  <pre><code>              AppleIntelLpssGspi`AppleIntelLpssGspi::regRead(unsigned int)+0x1f
                AppleIntelLpssGspi`AppleIntelLpssGspi::transferMmioDuplexMulti(void*, void*, unsigned long long, unsigned int)+0x91
                AppleIntelLpssSpiController`AppleIntelLpssSpiController::transferDataMmioDuplexMulti(void*, void*, unsigned int, unsigned int)+0xb2
                AppleIntelLpssSpiController`AppleIntelLpssSpiController::_transferDataSubr(AppleInfoLpssSpiControllerTransferDataRequest*)+0x5bc
                AppleIntelLpssSpiController`AppleIntelLpssSpiController::_transferData(AppleInfoLpssSpiControllerTransferDataRequest*)+0x24f
                kernel`IOCommandGate::runAction(int (*)(OSObject*, void*, void*, void*, void*), void*, void*, void*, void*)+0x138
                AppleIntelLpssSpiController`AppleIntelLpssSpiDevice::transferData(IOMemoryDescriptor*, void*, unsigned long long, unsigned long long, IOMemoryDescriptor*, void*, unsigned long long, unsigned long long, unsigned int, AppleIntelSPICompletion*)+0x151
                AppleHSSPISupport`AppleHSSPIController::transferData(IOMemoryDescriptor*, void*, unsigned long long, unsigned long long, IOMemoryDescriptor*, void*, unsigned long long, unsigned long long, unsigned int, AppleIntelSPICompletion*)+0xcc
                AppleHSSPISupport`AppleHSSPIController::doSPITransfer(bool, AppleHSSPITransferRetryReason*)+0x97
                AppleHSSPISupport`AppleHSSPIController::InterruptOccurred(IOInterruptEventSource*, int)+0xf8
                kernel`IOInterruptEventSource::checkForWork()+0x13c
                kernel`IOWorkLoop::runEventSources()+0x1e2
                kernel`IOWorkLoop::threadMain()+0x2c
                kernel`call_continuation+0x2e
                 53
  
                kernel`waitq_wakeup64_thread+0xa7
                pthread`__psynch_cvsignal+0x495
                pthread`_psynch_cvsignal+0x28
                kernel`psynch_cvsignal+0x38
                kernel`unix_syscall64+0x27d
                kernel`hndl_unix_scall64+0x16
                 60
  
                kernel`hndl_mdep_scall64+0x4
                113
  
                kernel`ml_set_interrupts_enabled+0x19
                524
  
                kernel`ml_set_interrupts_enabled+0x19
                kernel`hndl_mdep_scall64+0x10
               5890
  
                kernel`machine_idle+0x2f8
                kernel`call_continuation+0x2e
              43395
  </code></pre>
  
  <p>The most common stack traces in user space over 17 seconds clearly implicate com.docker.hyperkit.  There 1365 stack traces in 17 seconds in which <code>com.docker.hyperkit</code> created threads which averages to 80 threads per second.</p>
  
  <pre><code>              com.docker.hyperkit`0x000000010cbd20db+0x19f9
                com.docker.hyperkit`0x000000010cbdb98c+0x157
                com.docker.hyperkit`0x000000010cbf6c2d+0x4bd
                libsystem_pthread.dylib`_pthread_body+0x7e
                libsystem_pthread.dylib`_pthread_start+0x42
                libsystem_pthread.dylib`thread_start+0xd
                 19
  
                Hypervisor`hv_vmx_vcpu_read_vmcs+0x1
                com.docker.hyperkit`0x000000010cbd4c4f+0x2a
                com.docker.hyperkit`0x000000010cbd20db+0x174a
                com.docker.hyperkit`0x000000010cbdb98c+0x157
                com.docker.hyperkit`0x000000010cbf6c2d+0x4bd
                libsystem_pthread.dylib`_pthread_body+0x7e
                libsystem_pthread.dylib`_pthread_start+0x42
                libsystem_pthread.dylib`thread_start+0xd
                 22
  
                Hypervisor`hv_vmx_vcpu_read_vmcs
                com.docker.hyperkit`0x000000010cbdb98c+0x157
                com.docker.hyperkit`0x000000010cbf6c2d+0x4bd
                libsystem_pthread.dylib`_pthread_body+0x7e
                libsystem_pthread.dylib`_pthread_start+0x42
                libsystem_pthread.dylib`thread_start+0xd
                 34
  
                com.docker.hyperkit`0x000000010cbd878d+0x36
                com.docker.hyperkit`0x000000010cbd20db+0x42f
                com.docker.hyperkit`0x000000010cbdb98c+0x157
                com.docker.hyperkit`0x000000010cbf6c2d+0x4bd
                libsystem_pthread.dylib`_pthread_body+0x7e
                libsystem_pthread.dylib`_pthread_start+0x42
                libsystem_pthread.dylib`thread_start+0xd
                 47
  
                Hypervisor`hv_vcpu_run+0xd
                com.docker.hyperkit`0x000000010cbd20db+0x6b6
                com.docker.hyperkit`0x000000010cbdb98c+0x157
                com.docker.hyperkit`0x000000010cbf6c2d+0x4bd
                libsystem_pthread.dylib`_pthread_body+0x7e
                libsystem_pthread.dylib`_pthread_start+0x42
                libsystem_pthread.dylib`thread_start+0xd
                135
  </code></pre>
  
  <h3>Related issues</h3>
  
  <p>Github - docker/for-mac: <a href="https://github.com/docker/for-mac/issues/3499" rel="noreferrer">com.docker.hyperkit 100% cpu usage is back again #3499
  </a>.  One comment suggests adding volume caching described here: <a href="https://www.docker.com/blog/user-guided-caching-in-docker-for-mac/" rel="noreferrer">https://www.docker.com/blog/user-guided-caching-in-docker-for-mac/</a>.  I tried this and got a small ~10% reduction in CPU usage.</p>