{"action":"created","issue":{"url":"https://api.github.com/repos/docker/docker/issues/22185","repository_url":"https://api.github.com/repos/docker/docker","labels_url":"https://api.github.com/repos/docker/docker/issues/22185/labels{/name}","comments_url":"https://api.github.com/repos/docker/docker/issues/22185/comments","events_url":"https://api.github.com/repos/docker/docker/issues/22185/events","html_url":"https://github.com/docker/docker/issues/22185","id":149688733,"number":22185,"title":"DNS queries sometimes get stuck since upgrading to 1.11.0","user":{"login":"cupracer","id":7422872,"avatar_url":"https://avatars.githubusercontent.com/u/7422872?v=3","gravatar_id":"","url":"https://api.github.com/users/cupracer","html_url":"https://github.com/cupracer","followers_url":"https://api.github.com/users/cupracer/followers","following_url":"https://api.github.com/users/cupracer/following{/other_user}","gists_url":"https://api.github.com/users/cupracer/gists{/gist_id}","starred_url":"https://api.github.com/users/cupracer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cupracer/subscriptions","organizations_url":"https://api.github.com/users/cupracer/orgs","repos_url":"https://api.github.com/users/cupracer/repos","events_url":"https://api.github.com/users/cupracer/events{/privacy}","received_events_url":"https://api.github.com/users/cupracer/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/docker/docker/labels/group/networking","name":"group/networking","color":"F6FBFF"},{"url":"https://api.github.com/repos/docker/docker/labels/priority/P2","name":"priority/P2","color":"fad8c7"}],"state":"open","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/docker/docker/milestones/54","html_url":"https://github.com/docker/docker/milestones/1.11.1","labels_url":"https://api.github.com/repos/docker/docker/milestones/54/labels","id":1705023,"number":54,"title":"1.11.1","description":"","creator":{"login":"tiborvass","id":827131,"avatar_url":"https://avatars.githubusercontent.com/u/827131?v=3","gravatar_id":"","url":"https://api.github.com/users/tiborvass","html_url":"https://github.com/tiborvass","followers_url":"https://api.github.com/users/tiborvass/followers","following_url":"https://api.github.com/users/tiborvass/following{/other_user}","gists_url":"https://api.github.com/users/tiborvass/gists{/gist_id}","starred_url":"https://api.github.com/users/tiborvass/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tiborvass/subscriptions","organizations_url":"https://api.github.com/users/tiborvass/orgs","repos_url":"https://api.github.com/users/tiborvass/repos","events_url":"https://api.github.com/users/tiborvass/events{/privacy}","received_events_url":"https://api.github.com/users/tiborvass/received_events","type":"User","site_admin":false},"open_issues":21,"closed_issues":21,"state":"open","created_at":"2016-04-14T01:56:12Z","updated_at":"2016-04-22T00:48:39Z","due_on":"2016-04-26T07:00:00Z","closed_at":null},"comments":22,"created_at":"2016-04-20T07:51:46Z","updated_at":"2016-04-22T11:34:48Z","closed_at":null,"body":"**Output of `docker version`:**\r\n\r\n```\r\nClient:\r\n Version:      1.11.0\r\n API version:  1.23\r\n Go version:   go1.6.1\r\n Git commit:   9e83765\r\n Built:\r\n OS/Arch:      linux/amd64\r\n\r\nServer:\r\n Version:      1.11.0\r\n API version:  1.23\r\n Go version:   go1.6.1\r\n Git commit:   9e83765\r\n Built:\r\n OS/Arch:      linux/amd64\r\n```\r\n\r\n\r\n**Output of `docker info`:**\r\n\r\n```\r\nContainers: 27\r\n Running: 26\r\n Paused: 0\r\n Stopped: 1\r\nImages: 8\r\nServer Version: 1.11.0\r\nStorage Driver: devicemapper\r\n Pool Name: docker-254:4-537178368-pool\r\n Pool Blocksize: 65.54 kB\r\n Base Device Size: 10.74 GB\r\n Backing Filesystem: xfs\r\n Data file: /dev/data/dockerdata\r\n Metadata file: /dev/data/dockermetadata\r\n Data Space Used: 17.63 GB\r\n Data Space Total: 107.4 GB\r\n Data Space Available: 89.74 GB\r\n Metadata Space Used: 33.01 MB\r\n Metadata Space Total: 5.369 GB\r\n Metadata Space Available: 5.336 GB\r\n Udev Sync Supported: true\r\n Deferred Removal Enabled: false\r\n Deferred Deletion Enabled: false\r\n Deferred Deleted Device Count: 0\r\n Library Version: 1.03.01 (2011-10-15)\r\nLogging Driver: json-file\r\nCgroup Driver: cgroupfs\r\nPlugins:\r\n Volume: local\r\n Network: null host bridge\r\nKernel Version: 3.16.7-35-default\r\nOperating System: openSUSE 13.2 (Harlequin) (x86_64)\r\nOSType: linux\r\nArchitecture: x86_64\r\nCPUs: 8\r\nTotal Memory: 31.35 GiB\r\nName: shorty\r\nID: NOTV:7M7T:6HMW:I6DG:4MPD:M7XM:TJKR:R6F4:EXVS:4UV5:BIJZ:KWI5\r\nDocker Root Dir: /var/lib/docker\r\nDebug mode (client): false\r\nDebug mode (server): true\r\n File Descriptors: 158\r\n Goroutines: 273\r\n System Time: 2016-04-20T08:28:04.398341864+02:00\r\n EventsListeners: 0\r\nUsername: cupracer\r\nRegistry: https://index.docker.io/v1/\r\nWARNING: No kernel memory limit support\r\n```\r\n\r\n**Additional environment details (AWS, VirtualBox, physical, etc.):**\r\nThis is a single Docker instance on physical hardware.\r\nContainers are usually run by using Docker Compose.\r\nDocker server-side debugging is enabled!\r\n\r\n```\r\nDOCKER_OPTS=\"-D --dns 8.8.8.8 --storage-opt dm.datadev=/dev/data/dockerdata --storage-opt dm.metadatadev=/dev/data/dockermetadata --storage-opt dm.fs=xfs\"\r\nDOCKER_NETWORK_OPTIONS=\"\"\r\n```\r\n\r\n**The issue:**\r\nOne of my containers needs to do a lot of DNS queries (~ 100 per second, if not even more).\r\nInternal (lookup other containers by name): 50%\r\nExternal (lookup various different domain names): 50%\r\n\r\nThis normally works fine and the performance is great, but after a while (~ 1-4 hrs) this container's DNS queries slow down massively, although the query times in the following output stay fast.\r\nSince the container provides an Apache HTTP server with PHP enabled, it appeares unresponsive to the \"outside world\" because of those hanging DNS queries.\r\n\r\n```\r\n# cat /etc/resolv.conf\r\nnameserver 127.0.0.11\r\noptions ndots:0\r\n```\r\n\r\n```\r\nroot@d04348b20152:/# time dig google.de\r\n\r\n;; ANSWER SECTION:\r\ngoogle.de.              231     IN      A       216.58.201.227\r\n\r\n;; Query time: 16 msec\r\n;; SERVER: 127.0.0.11#53(127.0.0.11)\r\n;; WHEN: Wed Apr 20 06:52:35 UTC 2016\r\n;; MSG SIZE  rcvd: 54\r\n\r\nreal    0m0.021s\r\nuser    0m0.004s\r\nsys     0m0.000s\r\n\r\nroot@d04348b20152:/# time dig google.de\r\n\r\n;; ANSWER SECTION:\r\ngoogle.de.              224     IN      A       216.58.201.227\r\n\r\n;; Query time: 15 msec\r\n;; SERVER: 127.0.0.11#53(127.0.0.11)\r\n;; WHEN: Wed Apr 20 06:52:42 UTC 2016\r\n;; MSG SIZE  rcvd: 54\r\n\r\n\r\nreal    0m5.021s\r\nuser    0m0.004s\r\nsys     0m0.000s\r\n\r\nroot@d04348b20152:/# time dig google.de\r\n\r\n; <<>> DiG 9.9.5-9+deb8u6-Debian <<>> google.de\r\n;; global options: +cmd\r\n;; connection timed out; no servers could be reached\r\n\r\nreal    0m15.005s\r\nuser    0m0.004s\r\nsys     0m0.000s\r\n```\r\n\r\nInternal DNS lookups (resolve MongoDB and Redis container IP's) work fine all the time, but queries to resolve external DNS names seem to get stuck most of the time. \r\nRestarting the affected container clears the situation again (for a while).\r\n\r\nFurthermore, my system log contains a lot of the following. I found 104 of them today between 08:20:05 and 08:52:00. On yesterday, 478 of them were logged.\r\n\r\n```\r\n2016-04-20T08:20:05.669869+02:00 shorty docker[11372]: time=\"2016-04-20T08:20:05.669802699+02:00\" level=error msg=\"More than 50 concurrent queries from 127.0.0.11:34244\"\r\n...\r\n2016-04-20T08:52:00.377952+02:00 shorty docker[11372]: time=\"2016-04-20T08:52:00.377857662+02:00\" level=error msg=\"More than 50 concurrent queries from 127.0.0.11:34244\"\r\n```\r\n\r\nPings from within this container to the DNS server 8.8.8.8 and other external addresses work fine. Dig's that use 8.8.8.8 directly instead of 127.0.0.11 work fine, too.\r\nThe Docker host does not seem to have any problems in DNS resolving or reachability.\r\nThe container is reachable, too. It's just the DNS issue which breaks my application.\r\n\r\nNothin special gets logged (even in Debug mode) regarding a failing DNS query. Just:\r\n\r\n```\r\n2016-04-20T09:32:08.257300+02:00 shorty docker[11372]: time=\"2016-04-20T09:32:08.257281875+02:00\" level=debug msg=\"To resolve: www.google.de in \"\r\n2016-04-20T09:32:08.257334+02:00 shorty docker[11372]: time=\"2016-04-20T09:32:08.257295204+02:00\" level=debug msg=\"To resolve: www.google in de\"\r\n2016-04-20T09:32:08.257347+02:00 shorty docker[11372]: time=\"2016-04-20T09:32:08.257301203+02:00\" level=debug msg=\"To resolve: www in google.de\"\r\n2016-04-20T09:32:08.257368+02:00 shorty docker[11372]: time=\"2016-04-20T09:32:08.257309488+02:00\" level=debug msg=\"Query www.google.de.[28] from 172.18.0.4:40514, forwarding to udp:8.8.8.8\"\r\n```\r\n\r\nI believe that #22144 might be related to my described issue.\r\n\r\nBefore upgrading to Docker 1.11.0 I didn't limit Docker to \"--dns 8.8.8.8\", but I also had problems with issue #22081, so I'm using a single Google DNS server as a workaround."},"comment":{"url":"https://api.github.com/repos/docker/docker/issues/comments/213388479","html_url":"https://github.com/docker/docker/issues/22185#issuecomment-213388479","issue_url":"https://api.github.com/repos/docker/docker/issues/22185","id":213388479,"user":{"login":"cupracer","id":7422872,"avatar_url":"https://avatars.githubusercontent.com/u/7422872?v=3","gravatar_id":"","url":"https://api.github.com/users/cupracer","html_url":"https://github.com/cupracer","followers_url":"https://api.github.com/users/cupracer/followers","following_url":"https://api.github.com/users/cupracer/following{/other_user}","gists_url":"https://api.github.com/users/cupracer/gists{/gist_id}","starred_url":"https://api.github.com/users/cupracer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cupracer/subscriptions","organizations_url":"https://api.github.com/users/cupracer/orgs","repos_url":"https://api.github.com/users/cupracer/repos","events_url":"https://api.github.com/users/cupracer/events{/privacy}","received_events_url":"https://api.github.com/users/cupracer/received_events","type":"User","site_admin":false},"created_at":"2016-04-22T11:34:48Z","updated_at":"2016-04-22T11:34:48Z","body":"This is the corresponding Docker debug output (please note that the timestamps are in UTC+2):\r\n\r\n```\r\n2016-04-22T13:28:11.916769+02:00 shorty docker[21496]: time=\"2016-04-22T13:28:11.916728458+02:00\" level=debug msg=\"To resolve: opensuse.org in \"\r\n2016-04-22T13:28:11.916811+02:00 shorty docker[21496]: time=\"2016-04-22T13:28:11.916749553+02:00\" level=debug msg=\"To resolve: opensuse in org\"\r\n2016-04-22T13:28:11.916843+02:00 shorty docker[21496]: time=\"2016-04-22T13:28:11.916762145+02:00\" level=debug msg=\"Query opensuse.org.[1] from 172.19.0.10:43086, forwarding to udp:213.133.98.98\"\r\n2016-04-22T13:28:11.916858+02:00 shorty docker[21496]: time=\"2016-04-22T13:28:11.916771107+02:00\" level=debug msg=\"concurrent queries counter: 0\"\r\n2016-04-22T13:28:11.916869+02:00 shorty docker[21496]: time=\"2016-04-22T13:28:11.916779885+02:00\" level=debug msg=\"client dns id 666, changed id 49032\"\r\n\r\n2016-04-22T13:28:16.916767+02:00 shorty docker[21496]: time=\"2016-04-22T13:28:16.916734642+02:00\" level=debug msg=\"To resolve: opensuse.org in \"\r\n2016-04-22T13:28:16.916808+02:00 shorty docker[21496]: time=\"2016-04-22T13:28:16.916752266+02:00\" level=debug msg=\"To resolve: opensuse in org\"\r\n2016-04-22T13:28:16.916837+02:00 shorty docker[21496]: time=\"2016-04-22T13:28:16.916764026+02:00\" level=debug msg=\"Query opensuse.org.[1] from 172.19.0.10:43086, forwarding to udp:213.133.98.98\"\r\n2016-04-22T13:28:16.916854+02:00 shorty docker[21496]: time=\"2016-04-22T13:28:16.916773097+02:00\" level=debug msg=\"concurrent queries counter: 0\"\r\n2016-04-22T13:28:16.916871+02:00 shorty docker[21496]: time=\"2016-04-22T13:28:16.916779236+02:00\" level=debug msg=\"client dns id 666, changed id 15581\"\r\n\r\n2016-04-22T13:28:21.917068+02:00 shorty docker[21496]: time=\"2016-04-22T13:28:21.916975831+02:00\" level=debug msg=\"To resolve: opensuse.org in \"\r\n2016-04-22T13:28:21.917156+02:00 shorty docker[21496]: time=\"2016-04-22T13:28:21.917017353+02:00\" level=debug msg=\"To resolve: opensuse in org\"\r\n2016-04-22T13:28:21.917181+02:00 shorty docker[21496]: time=\"2016-04-22T13:28:21.917033612+02:00\" level=debug msg=\"Query opensuse.org.[1] from 172.19.0.10:43086, forwarding to udp:213.133.98.98\"\r\n2016-04-22T13:28:21.917204+02:00 shorty docker[21496]: time=\"2016-04-22T13:28:21.917043920+02:00\" level=debug msg=\"concurrent queries counter: 0\"\r\n2016-04-22T13:28:21.917214+02:00 shorty docker[21496]: time=\"2016-04-22T13:28:21.917050960+02:00\" level=debug msg=\"client dns id 666, changed id 49068\"\r\n```"}}