{"action":"created","issue":{"url":"https://api.github.com/repos/docker/docker/issues/25259","repository_url":"https://api.github.com/repos/docker/docker","labels_url":"https://api.github.com/repos/docker/docker/issues/25259/labels{/name}","comments_url":"https://api.github.com/repos/docker/docker/issues/25259/comments","events_url":"https://api.github.com/repos/docker/docker/issues/25259/events","html_url":"https://github.com/docker/docker/issues/25259","id":168482074,"number":25259,"title":"Inexplicable Performance Characteristics (Macbook Air faster than AWS c4.8xlarge at container creation)","user":{"login":"tolmasky","id":23753,"avatar_url":"https://avatars.githubusercontent.com/u/23753?v=3","gravatar_id":"","url":"https://api.github.com/users/tolmasky","html_url":"https://github.com/tolmasky","followers_url":"https://api.github.com/users/tolmasky/followers","following_url":"https://api.github.com/users/tolmasky/following{/other_user}","gists_url":"https://api.github.com/users/tolmasky/gists{/gist_id}","starred_url":"https://api.github.com/users/tolmasky/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tolmasky/subscriptions","organizations_url":"https://api.github.com/users/tolmasky/orgs","repos_url":"https://api.github.com/users/tolmasky/repos","events_url":"https://api.github.com/users/tolmasky/events{/privacy}","received_events_url":"https://api.github.com/users/tolmasky/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/docker/docker/labels/kind/performance","name":"kind/performance","color":"006b75"},{"url":"https://api.github.com/repos/docker/docker/labels/version/1.12","name":"version/1.12","color":"ededed"}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":11,"created_at":"2016-07-30T19:04:45Z","updated_at":"2016-08-02T09:24:48Z","closed_at":null,"body":"We are trying to optimize for container creation and I am hoping that I am simply doing something silly. The app we're building spins up many containers in response to user actions, and Docker seems to very quickly degrade in performance when it is asked to do this. We've tried just about everything: switching to overlayfs, mounting on a ram disk, running separate docker instances to try to parallalize the work -- nothing really has an affect on our container throughput (at least without impacting performance of individual creation time). Frustrated, I put together what I thought was the simplest test I could think of in this script:\r\n\r\n\r\n  I put together a very simple test script:\r\n\r\n``` bash\r\n#!/bin/bash\r\n        for i in `seq 1 10`;\r\n        do\r\n                echo `time docker run mhart/alpine-node:latest node`;\r\n        done\r\n```\r\n\r\nI then run it twice concurrently like this:\r\n\r\n`$ time ./test.sh & time ./test.sh & wait`\r\n\r\nAs this runs you can watch container creation performance degrade. Now, on my 1.7 GHz Intel Core i7 Macbook Air running Docker 1.12 through the standard installation (which means its using that VirtualBox system and is limited to ONE CPU and 1.58GB of memory), this test takes about real 0m5.512s to complete. The median container creation time is 0.538s.\r\n\r\nNow, we have a c4.8xlarge machine on AWS where I run the same test. Docker there has full access to all 36 CPUs clocked at 2.7Ghz each I believe, 60GB of memory, the test takes about real 9.340s. The median container creation time is 0.9415s, so twice as slow basically.\r\n\r\nSome observations: c4.8xlarge creates more threads (perhaps because it notices it has more CPUs to play with?) Perhaps the threads are all contending with each other or something? Interestingly enough, when we limit Docker to one CPU it appears to perform better (but not 100% sure on this, just started testing - we spun up 1 cpu Digital Ocean machine and it did better than the 36 core AWS machine). Anyways, as an initial step I'd like to at the very least make my server Docker match my Macbook Air's performance. Below are the Docker infos:\r\n\r\nMACBOOK AIR\r\n\r\nContainers: 210\r\n Running: 0\r\n Paused: 0\r\n Stopped: 210\r\nImages: 7\r\nServer Version: 1.12.0\r\nStorage Driver: aufs\r\n Root Dir: /mnt/sda1/var/lib/docker/aufs\r\n Backing Filesystem: extfs\r\n Dirs: 428\r\n Dirperm1 Supported: true\r\nLogging Driver: json-file\r\nCgroup Driver: cgroupfs\r\nPlugins:\r\n Volume: local\r\n Network: null bridge overlay host\r\nSwarm: inactive\r\nRuntimes: runc\r\nDefault Runtime: runc\r\nSecurity Options: seccomp\r\nKernel Version: 4.4.16-boot2docker\r\nOperating System: Boot2Docker 1.12.0 (TCL 7.2); HEAD : e030bab - Fri Jul 29 00:29:14 UTC 2016\r\nOSType: linux\r\nArchitecture: x86_64\r\nCPUs: 1\r\nTotal Memory: 1.955 GiB\r\nName: default\r\nDocker Root Dir: /mnt/sda1/var/lib/docker\r\nDebug Mode (client): false\r\nDebug Mode (server): true\r\n File Descriptors: 13\r\n Goroutines: 24\r\n System Time: 2016-07-30T19:02:29.045775555Z\r\n EventsListeners: 0\r\nRegistry: https://index.docker.io/v1/\r\nLabels:\r\n provider=virtualbox\r\n\r\nAWS\r\n\r\nContainers: 103\r\n Running: 0\r\n Paused: 0\r\n Stopped: 103\r\nImages: 1\r\nServer Version: 1.12.0\r\nStorage Driver: aufs\r\n Root Dir: /var/lib/docker/aufs\r\n Backing Filesystem: extfs\r\n Dirs: 208\r\n Dirperm1 Supported: true\r\nLogging Driver: json-file\r\nCgroup Driver: cgroupfs\r\nPlugins:\r\n Volume: local\r\n Network: null host bridge overlay\r\nSwarm: inactive\r\nRuntimes: runc\r\nDefault Runtime: runc\r\nSecurity Options: apparmor\r\nKernel Version: 4.4.0-31-generic\r\nOperating System: Ubuntu 16.04.1 LTS\r\nOSType: linux\r\nArchitecture: x86_64\r\nCPUs: 36\r\nTotal Memory: 58.97 GiB\r\nDocker Root Dir: /var/lib/docker\r\nDebug Mode (client): false\r\nDebug Mode (server): false\r\nRegistry: https://index.docker.io/v1/\r\nWARNING: No swap limit support"},"comment":{"url":"https://api.github.com/repos/docker/docker/issues/comments/236851492","html_url":"https://github.com/docker/docker/issues/25259#issuecomment-236851492","issue_url":"https://api.github.com/repos/docker/docker/issues/25259","id":236851492,"user":{"login":"unclejack","id":1409668,"avatar_url":"https://avatars.githubusercontent.com/u/1409668?v=3","gravatar_id":"","url":"https://api.github.com/users/unclejack","html_url":"https://github.com/unclejack","followers_url":"https://api.github.com/users/unclejack/followers","following_url":"https://api.github.com/users/unclejack/following{/other_user}","gists_url":"https://api.github.com/users/unclejack/gists{/gist_id}","starred_url":"https://api.github.com/users/unclejack/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/unclejack/subscriptions","organizations_url":"https://api.github.com/users/unclejack/orgs","repos_url":"https://api.github.com/users/unclejack/repos","events_url":"https://api.github.com/users/unclejack/events{/privacy}","received_events_url":"https://api.github.com/users/unclejack/received_events","type":"User","site_admin":false},"created_at":"2016-08-02T09:24:48Z","updated_at":"2016-08-02T09:24:48Z","body":"There seems to be an issue. alpine-node and the `node` command aren't the best choices for benchmarking container startup time. You probably want to stick to the `alpine` image and something such as `echo foo`. You're benchmarking the startup time of the node binary as well, but we're interested only in how long it takes to bring up the container."}}
{"actor":{"display_login":"unclejack"}}