{"action":"created","issue":{"url":"https://api.github.com/repos/docker/docker/issues/28237","repository_url":"https://api.github.com/repos/docker/docker","labels_url":"https://api.github.com/repos/docker/docker/issues/28237/labels{/name}","comments_url":"https://api.github.com/repos/docker/docker/issues/28237/comments","events_url":"https://api.github.com/repos/docker/docker/issues/28237/events","html_url":"https://github.com/docker/docker/issues/28237","id":188483293,"number":28237,"title":"Docker daemon crash with panic due to concurrent map read and map write","user":{"login":"hubo1016","id":14069404,"avatar_url":"https://avatars.githubusercontent.com/u/14069404?v=3","gravatar_id":"","url":"https://api.github.com/users/hubo1016","html_url":"https://github.com/hubo1016","followers_url":"https://api.github.com/users/hubo1016/followers","following_url":"https://api.github.com/users/hubo1016/following{/other_user}","gists_url":"https://api.github.com/users/hubo1016/gists{/gist_id}","starred_url":"https://api.github.com/users/hubo1016/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hubo1016/subscriptions","organizations_url":"https://api.github.com/users/hubo1016/orgs","repos_url":"https://api.github.com/users/hubo1016/repos","events_url":"https://api.github.com/users/hubo1016/events{/privacy}","received_events_url":"https://api.github.com/users/hubo1016/received_events","type":"User","site_admin":false},"labels":[{"id":375714123,"url":"https://api.github.com/repos/docker/docker/labels/version/1.12","name":"version/1.12","color":"ededed","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2016-11-10T11:23:54Z","updated_at":"2016-11-12T00:09:01Z","closed_at":null,"body":"---------------------------------------------------\r\nBUG REPORT INFORMATION\r\n---------------------------------------------------\r\nDocker daemon occasionally crashes with panic, and report a \"fatal error: concurrent map read and map write\" in the log.\r\n\r\n**Steps to reproduce the issue:**\r\n\r\nI believe there is a race condition so there is not a determined way to reproduce this, but it may be reproduced by:\r\n1.Create many (more than 100) containers in parallel, each with a volume (-v option) (driver=local)\r\n2.Remove the containers in parallel together with the volumes (-v)\r\n\r\n**Describe the results you received:**\r\n\r\nDocker daemon crashes with the following logs:\r\n\r\n```\r\nNov 10 15:17:41 kvm-yz-dev-073-003-009 dockerd: time=\"2016-11-10T15:17:41.017079547+08:00\" level=debug msg=\"devmapper: Unmount done\"\r\nNov 10 15:17:41 kvm-yz-dev-073-003-009 dockerd: time=\"2016-11-10T15:17:41.017117451+08:00\" level=debug msg=\"devmapper: deactivateDevice(7fed5bc81bd59\r\n27bbd8fc3b52a79df8ee617c6bbd90d21b1b0fb2ca11b5a4251-init)\"\r\nNov 10 15:17:41 kvm-yz-dev-073-003-009 dockerd: time=\"2016-11-10T15:17:41.017205184+08:00\" level=debug msg=\"devicemapper: RemoveDeviceDeferred START(\r\ndocker-253:0-166724644-7fed5bc81bd5927bbd8fc3b52a79df8ee617c6bbd90d21b1b0fb2ca11b5a4251-init)\"\r\nNov 10 15:17:41 kvm-yz-dev-073-003-009 dockerd: time=\"2016-11-10T15:17:41.026795020+08:00\" level=debug msg=\"Calling GET /v1.24/containers/json?all=1&\r\nfilters=%7B%22id%22%3A%7B%22f06afd11c6812e580eef549002a883762ff5e22e80dbc06bba51d8b50fe7c41a%22%3Atrue%7D%7D&limit=0\"\r\nNov 10 15:17:41 kvm-yz-dev-073-003-009 dockerd: time=\"2016-11-10T15:17:41.027450951+08:00\" level=debug msg=\"Calling GET /v1.24/containers/json?all=1&\r\nfilters=%7B%22id%22%3A%7B%22f06afd11c6812e580eef549002a883762ff5e22e80dbc06bba51d8b50fe7c41a%22%3Atrue%7D%7D&limit=0\"\r\nNov 10 15:17:41 kvm-yz-dev-073-003-009 dockerd: time=\"2016-11-10T15:17:41.027467289+08:00\" level=debug msg=\"Calling GET /v1.24/containers/json?all=1&\r\nfilters=%7B%22id%22%3A%7B%22f06afd11c6812e580eef549002a883762ff5e22e80dbc06bba51d8b50fe7c41a%22%3Atrue%7D%7D&limit=0\"\r\nNov 10 15:17:41 kvm-yz-dev-073-003-009 dockerd: time=\"2016-11-10T15:17:41.027477995+08:00\" level=debug msg=\"Calling GET /v1.24/containers/json?all=1&\r\nfilters=%7B%22id%22%3A%7B%22f06afd11c6812e580eef549002a883762ff5e22e80dbc06bba51d8b50fe7c41a%22%3Atrue%7D%7D&limit=0\"\r\nNov 10 15:17:41 kvm-yz-dev-073-003-009 dockerd: time=\"2016-11-10T15:17:41.027554115+08:00\" level=debug msg=\"Calling GET /v1.24/containers/json?all=1&\r\nfilters=%7B%22id%22%3A%7B%22f06afd11c6812e580eef549002a883762ff5e22e80dbc06bba51d8b50fe7c41a%22%3Atrue%7D%7D&limit=0\"\r\nNov 10 15:17:41 kvm-yz-dev-073-003-009 dockerd: time=\"2016-11-10T15:17:41.027938561+08:00\" level=debug msg=\"Calling GET /v1.24/containers/f06afd11c68\r\n12e580eef549002a883762ff5e22e80dbc06bba51d8b50fe7c41a/json\"\r\nNov 10 15:17:41 kvm-yz-dev-073-003-009 dockerd: time=\"2016-11-10T15:17:41.028539837+08:00\" level=debug msg=\"Calling GET /v1.24/containers/f06afd11c68\r\n12e580eef549002a883762ff5e22e80dbc06bba51d8b50fe7c41a/json\"\r\nNov 10 15:17:41 kvm-yz-dev-073-003-009 dockerd: time=\"2016-11-10T15:17:41.028787949+08:00\" level=debug msg=\"Calling GET /v1.24/containers/f06afd11c68\r\n12e580eef549002a883762ff5e22e80dbc06bba51d8b50fe7c41a/json\"\r\nNov 10 15:17:41 kvm-yz-dev-073-003-009 dockerd: time=\"2016-11-10T15:17:41.060133516+08:00\" level=debug msg=\"devicemapper: RemoveDeviceDeferred END(do\r\ncker-253:0-166724644-7fed5bc81bd5927bbd8fc3b52a79df8ee617c6bbd90d21b1b0fb2ca11b5a4251-init)\"\r\nNov 10 15:17:41 kvm-yz-dev-073-003-009 dockerd: time=\"2016-11-10T15:17:41.060171548+08:00\" level=debug msg=\"devmapper: deactivateDevice END(7fed5bc81bd5927bbd8fc3b52a79df8ee617c6bbd90d21b1b0fb2ca11b5a4251-init)\"\r\nNov 10 15:17:41 kvm-yz-dev-073-003-009 dockerd: time=\"2016-11-10T15:17:41.060196836+08:00\" level=debug msg=\"devmapper: UnmountDevice(hash=7fed5bc81bd5927bbd8fc3b52a79df8ee617c6bbd90d21b1b0fb2ca11b5a4251-init) END\"\r\nNov 10 15:17:41 kvm-yz-dev-073-003-009 dockerd: time=\"2016-11-10T15:17:41.060214172+08:00\" level=debug msg=\"devmapper: AddDevice(hash=7fed5bc81bd5927bbd8fc3b52a79df8ee617c6bbd90d21b1b0fb2ca11b5a4251 basehash=7fed5bc81bd5927bbd8fc3b52a79df8ee617c6bbd90d21b1b0fb2ca11b5a4251-init)\"\r\nNov 10 15:17:41 kvm-yz-dev-073-003-009 dockerd: time=\"2016-11-10T15:17:41.061304458+08:00\" level=debug msg=\"devmapper: registerDevice(1198, 7fed5bc81bd5927bbd8fc3b52a79df8ee617c6bbd90d21b1b0fb2ca11b5a4251)\"\r\nNov 10 15:17:41 kvm-yz-dev-073-003-009 dockerd: time=\"2016-11-10T15:17:41.061935064+08:00\" level=debug msg=\"devmapper: AddDevice(hash=7fed5bc81bd5927bbd8fc3b52a79df8ee617c6bbd90d21b1b0fb2ca11b5a4251 basehash=7fed5bc81bd5927bbd8fc3b52a79df8ee617c6bbd90d21b1b0fb2ca11b5a4251-init) END\"\r\nNov 10 15:17:41 kvm-yz-dev-073-003-009 dockerd: time=\"2016-11-10T15:17:41.063258176+08:00\" level=debug msg=\"Calling GET /v1.24/containers/f06afd11c6812e580eef549002a883762ff5e22e80dbc06bba51d8b50fe7c41a/json\"\r\nNov 10 15:17:41 kvm-yz-dev-073-003-009 dockerd: time=\"2016-11-10T15:17:41.063545764+08:00\" level=debug msg=\"Calling GET /v1.24/containers/f06afd11c6812e580eef549002a883762ff5e22e80dbc06bba51d8b50fe7c41a/json\"\r\nNov 10 15:17:41 kvm-yz-dev-073-003-009 dockerd: time=\"2016-11-10T15:17:41.066736590+08:00\" level=debug msg=\"Calling GET /containers/f06afd11c6812e580eef549002a883762ff5e22e80dbc06bba51d8b50fe7c41a/json\"\r\nNov 10 15:17:41 kvm-yz-dev-073-003-009 dockerd: time=\"2016-11-10T15:17:41.069140753+08:00\" level=debug msg=\"devmapper: activateDeviceIfNeeded(7fed5bc81bd5927bbd8fc3b52a79df8ee617c6bbd90d21b1b0fb2ca11b5a4251)\"\r\nNov 10 15:17:41 kvm-yz-dev-073-003-009 dockerd: time=\"2016-11-10T15:17:41.069161478+08:00\" level=debug msg=\"devmapper: cancelDeferredRemoval START(docker-253:0-166724644-7fed5bc81bd5927bbd8fc3b52a79df8ee617c6bbd90d21b1b0fb2ca11b5a4251)\"\r\nNov 10 15:17:41 kvm-yz-dev-073-003-009 dockerd: time=\"2016-11-10T15:17:41.069204513+08:00\" level=debug msg=\"devmapper: cancelDeferredRemoval END(docker-253:0-166724644-7fed5bc81bd5927bbd8fc3b52a79df8ee617c6bbd90d21b1b0fb2ca11b5a4251)\"\r\nNov 10 15:17:41 kvm-yz-dev-073-003-009 dockerd: time=\"2016-11-10T15:17:41.071506926+08:00\" level=debug msg=\"Calling POST /v1.22/containers/f06afd11c6812e580eef549002a883762ff5e22e80dbc06bba51d8b50fe7c41a/attach?stderr=1&logs=0&stream=1&stdout=1\"\r\nNov 10 15:17:41 kvm-yz-dev-073-003-009 dockerd: time=\"2016-11-10T15:17:41.071778638+08:00\" level=debug msg=\"attach: stderr: begin\"\r\nNov 10 15:17:41 kvm-yz-dev-073-003-009 dockerd: time=\"2016-11-10T15:17:41.071841077+08:00\" level=debug msg=\"attach: stdout: begin\"\r\nNov 10 15:17:41 kvm-yz-dev-073-003-009 dockerd: time=\"2016-11-10T15:17:41.072211565+08:00\" level=debug msg=\"Calling GET /v1.24/containers/json?all=1&filters=%7B%22id%22%3A%7B%22f06afd11c6812e580eef549002a883762ff5e22e80dbc06bba51d8b50fe7c41a%22%3Atrue%7D%7D&limit=0\"\r\nNov 10 15:17:41 kvm-yz-dev-073-003-009 dockerd: time=\"2016-11-10T15:17:41.072571301+08:00\" level=debug msg=\"Calling GET /v1.24/containers/json?all=1&filters=%7B%22id%22%3A%7B%22f06afd11c6812e580eef549002a883762ff5e22e80dbc06bba51d8b50fe7c41a%22%3Atrue%7D%7D&limit=0\"\r\nNov 10 15:17:41 kvm-yz-dev-073-003-009 dockerd: time=\"2016-11-10T15:17:41.072581765+08:00\" level=debug msg=\"Calling GET /v1.24/containers/json?all=1&filters=%7B%22id%22%3A%7B%22f06afd11c6812e580eef549002a883762ff5e22e80dbc06bba51d8b50fe7c41a%22%3Atrue%7D%7D&limit=0\"\r\nNov 10 15:17:41 kvm-yz-dev-073-003-009 dockerd: time=\"2016-11-10T15:17:41.072772582+08:00\" level=debug msg=\"Calling GET /v1.24/containers/json?all=1&filters=%7B%22id%22%3A%7B%22f06afd11c6812e580eef549002a883762ff5e22e80dbc06bba51d8b50fe7c41a%22%3Atrue%7D%7D&limit=0\"\r\nNov 10 15:17:41 kvm-yz-dev-073-003-009 dockerd: time=\"2016-11-10T15:17:41.074905610+08:00\" level=debug msg=\"Calling GET /containers/f06afd11c6812e580eef549002a883762ff5e22e80dbc06bba51d8b50fe7c41a/json\"\r\nNov 10 15:17:41 kvm-yz-dev-073-003-009 systemd: Device dev-disk-by\\x2duuid-1c31d7f1\\x2dc62f\\x2d48e6\\x2dafa6\\x2dedcee3caec5d.device appeared twice with different sysfs paths /sys/devices/virtual/block/dm-4 and /sys/devices/virtual/block/dm-82\r\nNov 10 15:17:41 kvm-yz-dev-073-003-009 kernel: XFS (dm-82): Mounting V4 Filesystem\r\nNov 10 15:17:41 kvm-yz-dev-073-003-009 dockerd: time=\"2016-11-10T15:17:41.131329748+08:00\" level=debug msg=\"Calling GET /v1.24/containers/json?all=1&limit=0\"\r\nNov 10 15:17:41 kvm-yz-dev-073-003-009 dockerd: time=\"2016-11-10T15:17:41.161736507+08:00\" level=debug msg=\"container mounted via layerStore: /data/docker/devicemapper/mnt/7fed5bc81bd5927bbd8fc3b52a79df8ee617c6bbd90d21b1b0fb2ca11b5a4251/rootfs\"\r\nNov 10 15:17:41 kvm-yz-dev-073-003-009 kernel: XFS (dm-82): Ending clean mount\r\nNov 10 15:17:41 kvm-yz-dev-073-003-009 dockerd: time=\"2016-11-10T15:17:41.167228015+08:00\" level=debug msg=\"Calling POST /v1.22/networks/3f7dd36d3c9bf9af95701e47dd7ec4bbf8b4f9b13fac264400618db76d59af8a/disconnect\"\r\nNov 10 15:17:41 kvm-yz-dev-073-003-009 dockerd: time=\"2016-11-10T15:17:41.167271043+08:00\" level=debug msg=\"form data: {\\\"container\\\":\\\"f06afd11c6812e580eef549002a883762ff5e22e80dbc06bba51d8b50fe7c41a\\\"}\"\r\nNov 10 15:17:41 kvm-yz-dev-073-003-009 dockerd: time=\"2016-11-10T15:17:41.170062945+08:00\" level=debug msg=\"Getting volume reference for name: e48a63c812bed06e34284637d0240b36f6d9b52479eed60383d8c94d8cb5bb33\"\r\nNov 10 15:17:41 kvm-yz-dev-073-003-009 dockerd: time=\"2016-11-10T15:17:41.170093701+08:00\" level=debug msg=\"Probing all drivers for volume with name: e48a63c812bed06e34284637d0240b36f6d9b52479eed60383d8c94d8cb5bb33\"\r\nNov 10 15:17:41 kvm-yz-dev-073-003-009 dockerd: time=\"2016-11-10T15:17:41.170171665+08:00\" level=debug msg=\"Registering new volume reference: driver \\\"local\\\", name \\\"e48a63c812bed06e34284637d0240b36f6d9b52479eed60383d8c94d8cb5bb33\\\"\"\r\nNov 10 15:17:41 kvm-yz-dev-073-003-009 dockerd: time=\"2016-11-10T15:17:41.181402344+08:00\" level=debug msg=\"Calling GET /v1.24/volumes\"\r\nNov 10 15:17:41 kvm-yz-dev-073-003-009 dockerd: time=\"2016-11-10T15:17:41.181918149+08:00\" level=debug msg=\"Getting volume reference for name: 04f62f17845f5e21e5d279434388b55a4d0ea19921499c7ae4c0fda284000c15\"\r\nNov 10 15:17:41 kvm-yz-dev-073-003-009 dockerd: time=\"2016-11-10T15:17:41.181943934+08:00\" level=debug msg=\"Probing all drivers for volume with name: 04f62f17845f5e21e5d279434388b55a4d0ea19921499c7ae4c0fda284000c15\"\r\nNov 10 15:17:41 kvm-yz-dev-073-003-009 dockerd: time=\"2016-11-10T15:17:41.182046537+08:00\" level=debug msg=\"Registering new volume reference: driver \\\"local\\\", name \\\"04f62f17845f5e21e5d279434388b55a4d0ea19921499c7ae4c0fda284000c15\\\"\"\r\nNov 10 15:17:41 kvm-yz-dev-073-003-009 dockerd: fatal error: concurrent map read and map write\r\nNov 10 15:17:41 kvm-yz-dev-073-003-009 dockerd: goroutine 500420 [running]:\r\nNov 10 15:17:41 kvm-yz-dev-073-003-009 dockerd: runtime.throw(0x1f42fa0, 0x21)\r\nNov 10 15:17:41 kvm-yz-dev-073-003-009 dockerd: /usr/local/go/src/runtime/panic.go:547 +0x90 fp=0xc8250fb5e0 sp=0xc8250fb5c8\r\nNov 10 15:17:41 kvm-yz-dev-073-003-009 dockerd: runtime.mapaccess1_faststr(0x16e6f00, 0xc8202d5740, 0xc8207d22b5, 0x40, 0xc821542bd0)\r\nNov 10 15:17:41 kvm-yz-dev-073-003-009 dockerd: /usr/local/go/src/runtime/hashmap_fast.go:202 +0x5b fp=0xc8250fb640 sp=0xc8250fb5e0\r\nNov 10 15:17:41 kvm-yz-dev-073-003-009 dockerd: github.com/docker/docker/volume/store.(*VolumeStore).list.func1(0xc823ff7da0, 0xc8202d5770, 0x7f6f5b9380c0, 0xc8201308c0)\r\nNov 10 15:17:41 kvm-yz-dev-073-003-009 dockerd: /root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/docker/volume/store/store.go:185 +0x381 fp=0xc8250fb7a0 sp=0xc8250fb640\r\nNov 10 15:17:41 kvm-yz-dev-073-003-009 dockerd: runtime.goexit()\r\nNov 10 15:17:41 kvm-yz-dev-073-003-009 dockerd: /usr/local/go/src/runtime/asm_amd64.s:1998 +0x1 fp=0xc8250fb7a8 sp=0xc8250fb7a0\r\nNov 10 15:17:41 kvm-yz-dev-073-003-009 dockerd: created by github.com/docker/docker/volume/store.(*VolumeStore).list\r\nNov 10 15:17:41 kvm-yz-dev-073-003-009 dockerd: /root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/docker/volume/store/store.go:189 +0x233\r\nNov 10 15:17:41 kvm-yz-dev-073-003-009 dockerd: goroutine 1 [chan receive, 4580 minutes]:\r\nNov 10 15:17:41 kvm-yz-dev-073-003-009 dockerd: main.(*DaemonCli).start(0xc820411560, 0x0, 0x0)\r\nNov 10 15:17:41 kvm-yz-dev-073-003-009 dockerd: /root/rpmbuild/BUILD/docker-engine/cmd/dockerd/daemon.go:308 +0x2274\r\nNov 10 15:17:41 kvm-yz-dev-073-003-009 dockerd: main.main()\r\nNov 10 15:17:41 kvm-yz-dev-073-003-009 dockerd: /root/rpmbuild/BUILD/docker-engine/cmd/dockerd/docker.go:68 +0x491\r\nNov 10 15:17:41 kvm-yz-dev-073-003-009 dockerd: goroutine 17 [syscall, 4581 minutes, locked to thread]:\r\nNov 10 15:17:41 kvm-yz-dev-073-003-009 dockerd: runtime.goexit()\r\nNov 10 15:17:41 kvm-yz-dev-073-003-009 dockerd: /usr/local/go/src/runtime/asm_amd64.s:1998 +0x1\r\nNov 10 15:17:41 kvm-yz-dev-073-003-009 dockerd: goroutine 6 [syscall, 4561 minutes]:\r\nNov 10 15:17:41 kvm-yz-dev-073-003-009 dockerd: os/signal.signal_recv(0x7f6f5b91c190)\r\nNov 10 15:17:41 kvm-yz-dev-073-003-009 dockerd: /usr/local/go/src/runtime/sigqueue.go:116 +0x132\r\nNov 10 15:17:41 kvm-yz-dev-073-003-009 dockerd: os/signal.loop()\r\nNov 10 15:17:41 kvm-yz-dev-073-003-009 dockerd: /usr/local/go/src/os/signal/signal_unix.go:22 +0x18\r\nNov 10 15:17:41 kvm-yz-dev-073-003-009 dockerd: created by os/signal.init.1\r\nNov 10 15:17:41 kvm-yz-dev-073-003-009 dockerd: /usr/local/go/src/os/signal/signal_unix.go:28 +0x37\r\nNov 10 15:17:41 kvm-yz-dev-073-003-009 dockerd: goroutine 14 [syscall, 4581 minutes]:\r\nNov 10 15:17:41 kvm-yz-dev-073-003-009 dockerd: syscall.Syscall6(0x3d, 0x104ea, 0xc82005addc, 0x0, 0xc8200c50e0, 0x0, 0x0, 0x4465c1, 0x4465c1, 0xc820001bb8)\r\nNov 10 15:17:41 kvm-yz-dev-073-003-009 dockerd: /usr/local/go/src/syscall/asm_linux_amd64.s:44 +0x5\r\nNov 10 15:17:41 kvm-yz-dev-073-003-009 dockerd: syscall.wait4(0x104ea, 0xc82005addc, 0x0, 0xc8200c50e0, 0x90, 0x0, 0x0)\r\nNov 10 15:17:41 kvm-yz-dev-073-003-009 dockerd: /usr/local/go/src/syscall/zsyscall_linux_amd64.go:172 +0x7f\r\nNov 10 15:17:41 kvm-yz-dev-073-003-009 dockerd: syscall.Wait4(0x104ea, 0xc82005ae24, 0x0, 0xc8200c50e0, 0x41a243, 0x0, 0x0)\r\nNov 10 15:17:41 kvm-yz-dev-073-003-009 dockerd: /usr/local/go/src/syscall/syscall_linux.go:256 +0x55\r\nNov 10 15:17:41 kvm-yz-dev-073-003-009 dockerd: os.(*Process).wait(0xc820428c20, 0xc82000c0c0, 0x0, 0x0)\r\nNov 10 15:17:41 kvm-yz-dev-073-003-009 dockerd: /usr/local/go/src/os/exec_unix.go:22 +0x105\r\n...\r\n```\r\n\r\n**Describe the results you expected:**\r\n\r\nAll process done without crash\r\n\r\n**Additional information you deem important (e.g. issue happens only occasionally):**\r\n\r\nissue happens only occasionally\r\n\r\n**Output of `docker version`:**\r\n```\r\nClient:\r\n Version:      1.12.1\r\n API version:  1.24\r\n Go version:   go1.6.3\r\n Git commit:   23cf638\r\n Built:\r\n OS/Arch:      linux/amd64\r\nServer:\r\n Version:      1.12.1\r\n API version:  1.24\r\n Go version:   go1.6.3\r\n Git commit:   23cf638\r\n Built:\r\n OS/Arch:      linux/amd64\r\n```\r\n**Output of `docker info`:**\r\n```\r\nContainers: 110\r\n Running: 103\r\n Paused: 0\r\n Stopped: 7\r\nImages: 99\r\nServer Version: 1.12.1\r\nStorage Driver: devicemapper\r\n Pool Name: docker-thinpool\r\n Pool Blocksize: 524.3 kB\r\n Base Device Size: 10.74 GB\r\n Backing Filesystem: xfs\r\n Data file:\r\n Metadata file:\r\n Data Space Used: 148.6 GB\r\n Data Space Total: 1.1 TB\r\n Data Space Available: 950.9 GB\r\n Metadata Space Used: 30.46 MB\r\n Metadata Space Total: 10.74 GB\r\n Metadata Space Available: 10.71 GB\r\n Thin Pool Minimum Free Space: 110 GB\r\n Udev Sync Supported: true\r\n Deferred Removal Enabled: true\r\n Deferred Deletion Enabled: false\r\n Deferred Deleted Device Count: 0\r\n Library Version: 1.02.107-RHEL7 (2016-06-09)\r\nLogging Driver: json-file\r\nCgroup Driver: cgroupfs\r\nPlugins:\r\n Volume: local\r\n Network: null host overlay vlcp bridge\r\nSwarm: inactive\r\nRuntimes: runc\r\nDefault Runtime: runc\r\nSecurity Options: seccomp\r\nKernel Version: 3.10.0-327.el7.x86_64\r\nOperating System: CentOS Linux 7 (Core)\r\nOSType: linux\r\nArchitecture: x86_64\r\nCPUs: 48\r\nTotal Memory: 251.6 GiB\r\nName: kvm-yz-dev-073-003-009\r\nID: WOZR:LE5T:PPII:LGRJ:VOEB:UBGT:OFGG:ZMEQ:TEYW:3A3H:HHI2:UNU4\r\nDocker Root Dir: /data/docker\r\nDebug Mode (client): false\r\nDebug Mode (server): true\r\n File Descriptors: 967\r\n Goroutines: 1512\r\n System Time: 2016-11-10T19:07:41.132487639+08:00\r\n EventsListeners: 30\r\nRegistry: https://index.docker.io/v1/\r\nWARNING: bridge-nf-call-ip6tables is disabled\r\nCluster Store: zk://100.73.3.8:2181,100.73.3.9:2181,100.73.3.7:2181\r\nCluster Advertise: 100.73.3.9:2375\r\nInsecure Registries:\r\n jdocker.me\r\n 127.0.0.0/8\r\n```\r\n\r\n**Additional environment details (AWS, VirtualBox, physical, etc.):**\r\nphysical. We are using docker swarm (swarm standalone, not swarmkit) to manage the cluster.\r\n\r\nIt seems to be the same type of issue of #27486 The map used in (*VolumeStore).list.func1\r\nin volume/store/store.go:\r\n```go\r\n\tfor _, vd := range drivers {\r\n\t\tgo func(d volume.Driver) {\r\n\t\t\tvs, err := d.List()\r\n\t\t\tif err != nil {\r\n\t\t\t\tchVols <- vols{driverName: d.Name(), err: &OpErr{Err: err, Name: d.Name(), Op: \"list\"}}\r\n\t\t\t\treturn\r\n\t\t\t}\r\n\t\t\tfor i, v := range vs {\r\n\t\t\t\tvs[i] = volumeWrapper{v, s.labels[v.Name()], d.Scope(), s.options[v.Name()]}\r\n\t\t\t}\r\n\r\n\t\t\tchVols <- vols{vols: vs}\r\n\t\t}(vd)\r\n\t}\r\n```\r\n\r\nseems the *s.labels* and *s.options* does not have locks surrounded which leads to this issue."},"comment":{"url":"https://api.github.com/repos/docker/docker/issues/comments/260084528","html_url":"https://github.com/docker/docker/issues/28237#issuecomment-260084528","issue_url":"https://api.github.com/repos/docker/docker/issues/28237","id":260084528,"user":{"login":"aaronlehmann","id":10601430,"avatar_url":"https://avatars.githubusercontent.com/u/10601430?v=3","gravatar_id":"","url":"https://api.github.com/users/aaronlehmann","html_url":"https://github.com/aaronlehmann","followers_url":"https://api.github.com/users/aaronlehmann/followers","following_url":"https://api.github.com/users/aaronlehmann/following{/other_user}","gists_url":"https://api.github.com/users/aaronlehmann/gists{/gist_id}","starred_url":"https://api.github.com/users/aaronlehmann/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aaronlehmann/subscriptions","organizations_url":"https://api.github.com/users/aaronlehmann/orgs","repos_url":"https://api.github.com/users/aaronlehmann/repos","events_url":"https://api.github.com/users/aaronlehmann/events{/privacy}","received_events_url":"https://api.github.com/users/aaronlehmann/received_events","type":"User","site_admin":false},"created_at":"2016-11-12T00:09:01Z","updated_at":"2016-11-12T00:09:01Z","body":"It looks to me like `s.globalLock` or `s.locks.Lock(v.Name())` needs to be held here (it's not clear to me which one, since they seem to be used inconsistently). Also, it looks like `FilterByDriver` might need locking."}}
{"actor":{"display_login":"aaronlehmann"}}