{"action":"created","issue":{"url":"https://api.github.com/repos/docker/docker/issues/22383","repository_url":"https://api.github.com/repos/docker/docker","labels_url":"https://api.github.com/repos/docker/docker/issues/22383/labels{/name}","comments_url":"https://api.github.com/repos/docker/docker/issues/22383/comments","events_url":"https://api.github.com/repos/docker/docker/issues/22383/events","html_url":"https://github.com/docker/docker/issues/22383","id":151536567,"number":22383,"title":"container launch time increases as the number of pre-existing containers increase","user":{"login":"vvb","id":4341190,"avatar_url":"https://avatars.githubusercontent.com/u/4341190?v=3","gravatar_id":"","url":"https://api.github.com/users/vvb","html_url":"https://github.com/vvb","followers_url":"https://api.github.com/users/vvb/followers","following_url":"https://api.github.com/users/vvb/following{/other_user}","gists_url":"https://api.github.com/users/vvb/gists{/gist_id}","starred_url":"https://api.github.com/users/vvb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vvb/subscriptions","organizations_url":"https://api.github.com/users/vvb/orgs","repos_url":"https://api.github.com/users/vvb/repos","events_url":"https://api.github.com/users/vvb/events{/privacy}","received_events_url":"https://api.github.com/users/vvb/received_events","type":"User","site_admin":false},"labels":[{"id":77260064,"url":"https://api.github.com/repos/docker/docker/labels/area/storage/devicemapper","name":"area/storage/devicemapper","color":"FBCA04","default":false},{"id":335124477,"url":"https://api.github.com/repos/docker/docker/labels/kind/performance","name":"kind/performance","color":"006b75","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":9,"created_at":"2016-04-28T02:42:57Z","updated_at":"2016-11-30T22:29:47Z","closed_at":null,"body":"The time taken to launch the container seems to keep growing incrementally as the number of pre-existing containers go up. `docker run` starts to take a much long time after about 1.5k containers. \nThe analysis is available at: https://gist.github.com/vvb/f1cfff29b0738eef9fca202019bae024\n\n```\n$ cat /etc/redhat-release\nCentOS Linux release 7.2.1511 (Core)\n\n$ uname -a\nLinux ucsb-blade2 3.10.0-327.13.1.el7.x86_64 #1 SMP Thu Mar 31 16:04:38 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux\n```\n\n```\n$ docker version\nClient:\n Version:      1.10.3\n API version:  1.22\n Go version:   go1.5.3\n Git commit:   20f81dd\n Built:        Thu Mar 10 15:39:25 2016\n OS/Arch:      linux/amd64\n\nServer:\n Version:      swarm/1.1.2\n API version:  1.22\n Go version:   go1.5.3\n Git commit:   f947993\n Built:        Thu Feb 18 08:28:32 UTC 2016\n OS/Arch:      linux/amd64\n```\n\n```\n$ docker info\nContainers: 217\n Running: 216\n Paused: 0\n Stopped: 1\nImages: 4\nServer Version: 1.10.3\nStorage Driver: devicemapper\n Pool Name: contiv-dockerthin\n Pool Blocksize: 65.54 kB\n Base Device Size: 10.74 GB\n Backing Filesystem: xfs\n Data file:\n Metadata file:\n Data Space Used: 2.867 GB\n Data Space Total: 41.94 GB\n Data Space Available: 39.08 GB\n Metadata Space Used: 39.65 MB\n Metadata Space Total: 8.389 GB\n Metadata Space Available: 8.349 GB\n Udev Sync Supported: true\n Deferred Removal Enabled: false\n Deferred Deletion Enabled: false\n Deferred Deleted Device Count: 0\n Library Version: 1.02.107-RHEL7 (2015-12-01)\nExecution Driver: native-0.2\nLogging Driver: json-file\nPlugins:\n Volume: local\n Network: bridge null host netplugin\nKernel Version: 3.10.0-327.13.1.el7.x86_64\nOperating System: CentOS Linux 7 (Core)\nOSType: linux\nArchitecture: x86_64\nCPUs: 56\nTotal Memory: 220 GiB\nName: ucsb-blade2\nID: KHYO:HIQR:S7H5:FMIO:F3EK:3TEJ:HGDH:32JR:2WDM:2MFT:A25L:IBKJ\nWARNING: bridge-nf-call-ip6tables is disabled\nCluster store: etcd://localhost:2379\n```\n\nswarm cluster against which all tests are done\n\n```\n[admin@blade2 compose]$ docker info\nContainers: 1517\n Running: 1516\n Paused: 0\n Stopped: 1\nImages: 28\nServer Version: swarm/1.1.2\nRole: primary\nStrategy: spread\nFilters: health, port, dependency, affinity, constraint\nNodes: 7\n blade2: 10.193.231.88:2385\n  b Status: Healthy\n  b Containers: 217\n  b Reserved CPUs: 0 / 58\n  b Reserved Memory: 0 B / 231 GiB\n  b Labels: executiondriver=native-0.2, kernelversion=3.10.0-327.13.1.el7.x86_64, operatingsystem=CentOS Linux 7 (Core), storagedriver=devicemapper\n  b Error: (none)\n  b UpdatedAt: 2016-04-28T02:29:09Z\n blade3: 10.193.231.89:2385\n  b Status: Healthy\n  b Containers: 217\n  b Reserved CPUs: 0 / 58\n  b Reserved Memory: 0 B / 231 GiB\n  b Labels: executiondriver=native-0.2, kernelversion=3.10.0-327.13.1.el7.x86_64, operatingsystem=CentOS Linux 7 (Core), storagedriver=devicemapper\n  b Error: (none)\n  b UpdatedAt: 2016-04-28T02:29:05Z\n blade4: 10.193.231.90:2385\n  b Status: Healthy\n  b Containers: 217\n  b Reserved CPUs: 0 / 58\n  b Reserved Memory: 0 B / 231 GiB\n  b Labels: executiondriver=native-0.2, kernelversion=3.10.0-327.13.1.el7.x86_64, operatingsystem=CentOS Linux 7 (Core), storagedriver=devicemapper\n  b Error: (none)\n  b UpdatedAt: 2016-04-28T02:28:44Z\n blade5: 10.193.231.91:2385\n  b Status: Healthy\n  b Containers: 217\n  b Reserved CPUs: 0 / 33\n  b Reserved Memory: 0 B / 231 GiB\n  b Labels: executiondriver=native-0.2, kernelversion=3.10.0-327.13.1.el7.x86_64, operatingsystem=CentOS Linux 7 (Core), storagedriver=devicemapper\n  b Error: (none)\n  b UpdatedAt: 2016-04-28T02:28:50Z\n osd1: 10.193.231.94:2385\n  b Status: Healthy\n  b Containers: 217\n  b Reserved CPUs: 0 / 50\n  b Reserved Memory: 0 B / 198 GiB\n  b Labels: executiondriver=native-0.2, kernelversion=3.10.0-327.13.1.el7.x86_64, operatingsystem=CentOS Linux 7 (Core), storagedriver=devicemapper\n  b Error: (none)\n  b UpdatedAt: 2016-04-28T02:29:05Z\n osd2: 10.193.231.96:2385\n  b Status: Healthy\n  b Containers: 216\n  b Reserved CPUs: 0 / 50\n  b Reserved Memory: 0 B / 198 GiB\n  b Labels: executiondriver=native-0.2, kernelversion=3.10.0-327.13.1.el7.x86_64, operatingsystem=CentOS Linux 7 (Core), storagedriver=devicemapper\n  b Error: (none)\n  b UpdatedAt: 2016-04-28T02:28:41Z\n osd3: 10.193.231.98:2385\n  b Status: Healthy\n  b Containers: 216\n  b Reserved CPUs: 0 / 50\n  b Reserved Memory: 0 B / 198 GiB\n  b Labels: executiondriver=native-0.2, kernelversion=3.10.0-327.13.1.el7.x86_64, operatingsystem=CentOS Linux 7 (Core), storagedriver=devicemapper\n  b Error: (none)\n  b UpdatedAt: 2016-04-28T02:29:15Z\nPlugins:\n Volume:\n Network:\nKernel Version: 3.10.0-327.13.1.el7.x86_64\nOperating System: linux\nArchitecture: amd64\nCPUs: 357\nTotal Memory: 1.482 TiB\nName: 727c9c03ac1d\n```\n"},"comment":{"url":"https://api.github.com/repos/docker/docker/issues/comments/264017207","html_url":"https://github.com/docker/docker/issues/22383#issuecomment-264017207","issue_url":"https://api.github.com/repos/docker/docker/issues/22383","id":264017207,"user":{"login":"rhvgoyal","id":9487534,"avatar_url":"https://avatars.githubusercontent.com/u/9487534?v=3","gravatar_id":"","url":"https://api.github.com/users/rhvgoyal","html_url":"https://github.com/rhvgoyal","followers_url":"https://api.github.com/users/rhvgoyal/followers","following_url":"https://api.github.com/users/rhvgoyal/following{/other_user}","gists_url":"https://api.github.com/users/rhvgoyal/gists{/gist_id}","starred_url":"https://api.github.com/users/rhvgoyal/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rhvgoyal/subscriptions","organizations_url":"https://api.github.com/users/rhvgoyal/orgs","repos_url":"https://api.github.com/users/rhvgoyal/repos","events_url":"https://api.github.com/users/rhvgoyal/events{/privacy}","received_events_url":"https://api.github.com/users/rhvgoyal/received_events","type":"User","site_admin":false},"created_at":"2016-11-30T22:29:47Z","updated_at":"2016-11-30T22:29:47Z","body":"Is this a regression? It did not happen in 1.9? What about 1.12?\r\n\r\nSo how many containers are running when these increased launch time are noticed. All the containers launched so far are still running?\r\n\r\nCould it be related to page cache not being shared with devicemapper? More containers running consume more memory and when one hits system limit, may be more memory reclaim logic kicks in which introduces delay? I am just guessing. May be it is something else.\r\n\r\nIt might also help to run the test with only container creation (and do not run container) and figure out if delay is noticed there or not."}}
{"actor":{"display_login":"rhvgoyal"}}