title: Docker Traefik can't resolve DNS (Fails reaching server and obtaining certificates)
body: <p>Googling the following issue shows that this hasn't been posted the first time, however, none of them really give an answer.</p>
  
  <p>When starting Traefik (v2.2.1 aka. latest) as a container in Docker, no matter what I try, I keep getting following error, for ALL the domains configured:</p>
  
  <pre><code>time="2020-05-24T15:48:57Z" level=error msg="Unable to obtain ACME certificate for domains \"&lt;my domain&gt;\": cannot get ACME client get directory at 'https://acme-staging-v02.api.letsencrypt.org/directory': Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 127.0.0.11:53: read udp 127.0.0.1:44687-&gt;127.0.0.11:53: i/o timeout" routerName=traefik@docker rule="Host(`&lt;my domain&gt;`)" providerName=le.acme
  </code></pre>
  
  <hr>
  
  <p>Checking <a href="https://letsencrypt.status.io/" rel="noreferrer">https://letsencrypt.status.io/</a> it doesn't seem to be a problem of Let's Encrypt's servers</p>
  
  <p><a href="https://i.stack.imgur.com/ykiVK.png" rel="noreferrer"><img src="https://i.stack.imgur.com/ykiVK.png" alt="enter image description here"></a>
  <a href="https://i.stack.imgur.com/Ntki2.png" rel="noreferrer"><img src="https://i.stack.imgur.com/Ntki2.png" alt="enter image description here"></a></p>
  
  <hr>
  
  <p>I have tried with two different OSs on the server Debian 10, Ubuntu Server 18.04 and 20.04. While installing the OS, I always follow my guide I created for myself here: <a href="https://gist.github.com/D3strukt0r/5aaba1a021d16b31fa19adf6eb26a102" rel="noreferrer">https://gist.github.com/D3strukt0r/5aaba1a021d16b31fa19adf6eb26a102</a></p>
  
  <p>Yes, I do as little as possible in the system and as much as possible with the containers.</p>
  
  <hr>
  
  <p>Following is my <code>docker-compose.yml</code> for Traefik</p>
  
  <pre class="lang-yaml prettyprint-override"><code>version: "2"
  
  # Manage domain access to services
  services:
    traefik:
      container_name: traefik
      image: traefik
      command:
        - --log.level=DEBUG
        - --api.dashboard=true
        - --providers.docker=true
        - --providers.docker.exposedbydefault=false
        - --providers.docker.network=traefik_proxy
        - --entrypoints.http.address=:80
        - --entrypoints.https.address=:443
        - --certificatesresolvers.le.acme.email=${ACME_EMAIL}
        - --certificatesresolvers.le.acme.storage=acme.json
        - --certificatesresolvers.le.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory
        - --certificatesresolvers.le.acme.dnschallenge=true
        - --certificatesresolvers.le.acme.dnschallenge.provider=cloudflare
        # - --certificatesresolvers.le.acme.dnschallenge.resolvers=1.1.1.1:53,8.8.8.8:53
      restart: always
      networks:
        - traefik_proxy
      ports:
        - 80:80
        - 443:443
      volumes:
        - /var/run/docker.sock:/var/run/docker.sock:ro
        #- ./acme.json:/acme.json
        - ./acme_testing.json:/acme.json
      environment:
        CF_API_EMAIL: ${CF_API_EMAIL}
        CF_API_KEY: ${CF_API_KEY}
      labels:
        - traefik.enable=true
  
        - traefik.http.routers.traefik0.entrypoints=http
        - traefik.http.routers.traefik0.rule=Host(`&lt;my domain&gt;`)
        - traefik.http.routers.traefik0.middlewares=to_https
  
        - traefik.http.routers.traefik.entrypoints=https
        - traefik.http.routers.traefik.rule=Host(`&lt;my domain&gt;`)
        - traefik.http.routers.traefik.middlewares=traefik_auth
        - traefik.http.routers.traefik.tls=true
        - traefik.http.routers.traefik.tls.certresolver=le
        - traefik.http.routers.traefik.service=api@internal
  
        # Declaring the user list
        #
        # Note: all dollar signs in the hash need to be doubled for escaping.
        # To create user:password pair, it's possible to use this command:
        # echo $(htpasswd -nb user password) | sed -e s/\\$/\\$\\$/g
        - traefik.http.middlewares.traefik_auth.basicauth.users=${TRAEFIK_USERS}
  
        # Standard middleware for other containers to use
        - traefik.http.middlewares.to_https.redirectscheme.scheme=https
        - traefik.http.middlewares.to_https_perm.redirectscheme.scheme=https
        - traefik.http.middlewares.to_https_perm.redirectscheme.permanent=true
  
  networks:
    traefik_proxy:
      external: true
  </code></pre>
  
  <p>The folder structure in there:</p>
  
  <pre><code>root@server:/opt/traefik# ls -Al
  total 8
  -rw------- 1 root root      0 May 24 00:37 acme.json
  -rw------- 1 root root      0 May 24 00:37 acme_testing.json
  -rw-rw-r-- 1 root docker 2406 May 24 18:04 docker-compose.yml
  -rw-rw-r-- 1 root docker  185 May 23 23:49 .env
  </code></pre>
  
  <p>That's all there is to the configuration.</p>
  
  <p>An <code>nslookup</code> outside will give the following:</p>
  
  <pre><code>root@server:/opt/traefik# nslookup acme-staging-v02.api.letsencrypt.org
  Server:         192.168.1.1
  Address:        192.168.1.1#53
  
  Non-authoritative answer:
  acme-staging-v02.api.letsencrypt.org    canonical name = staging.api.letsencrypt.org.
  staging.api.letsencrypt.org     canonical name = 56a5f4b0bc8146689ec3e272c43525f9.pacloudflare.com.
  Name:   56a5f4b0bc8146689ec3e272c43525f9.pacloudflare.com
  Address: 172.65.46.172
  Name:   56a5f4b0bc8146689ec3e272c43525f9.pacloudflare.com
  Address: 2606:4700:60:0:f41b:d4fe:4325:6026
  </code></pre>
  
  <p>An <code>nslookup</code> INSIDE the container will give the following:</p>
  
  <pre><code>manuele@server:/opt$ docker exec -it traefik /bin/sh
  / # nslookup acme-staging-v02.api.letsencrypt.org
  ;; connection timed out; no servers could be reached
  </code></pre>
  
  <p>Maybe for further information, here is also the log</p>
  
  <pre><code>root@server:/opt/traefik# docker-compose up
  Recreating traefik ... done
  Attaching to traefik
  traefik    | time="2020-05-24T16:05:34Z" level=info msg="Configuration loaded from flags."
  traefik    | time="2020-05-24T16:05:34Z" level=info msg="Traefik version 2.2.1 built on 2020-04-29T18:02:09Z"
  traefik    | time="2020-05-24T16:05:34Z" level=debug msg="Static configuration loaded {\"global\":{\"checkNewVersion\":true},\"serversTransport\":{\"maxIdleConnsPerHost\":200},\"entryPoints\":{\"http\":{\"address\":\":80\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":10000000000},\"respondingTimeouts\":{\"idleTimeout\":180000000000}},\"forwardedHeaders\":{},\"http\":{}},\"https\":{\"address\":\":443\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":10000000000},\"respondingTimeouts\":{\"idleTimeout\":180000000000}},\"forwardedHeaders\":{},\"http\":{}}},\"providers\":{\"providersThrottleDuration\":2000000000,\"docker\":{\"watch\":true,\"endpoint\":\"unix:///var/run/docker.sock\",\"defaultRule\":\"Host(`{{ normalize .Name }}`)\",\"network\":\"traefik_proxy\",\"swarmModeRefreshSeconds\":15000000000}},\"api\":{\"dashboard\":true},\"log\":{\"level\":\"DEBUG\",\"format\":\"common\"},\"certificatesResolvers\":{\"le\":{\"acme\":{\"email\":\"&lt;ACME Email&gt;\",\"caServer\":\"https://acme-staging-v02.api.letsencrypt.org/directory\",\"storage\":\"acme.json\",\"keyType\":\"RSA4096\",\"dnsChallenge\":{\"provider\":\"cloudflare\"}}}}}"
  traefik    | time="2020-05-24T16:05:34Z" level=info msg="\nStats collection is disabled.\nHelp us improve Traefik by turning this feature on :)\nMore details on: https://docs.traefik.io/contributing/data-collection/\n"
  traefik    | time="2020-05-24T16:05:34Z" level=info msg="Starting provider aggregator.ProviderAggregator {}"
  traefik    | time="2020-05-24T16:05:34Z" level=debug msg="Start TCP Server" entryPointName=https
  traefik    | time="2020-05-24T16:05:34Z" level=info msg="Starting provider *acme.Provider {\"email\":\"&lt;ACME Email&gt;\",\"caServer\":\"https://acme-staging-v02.api.letsencrypt.org/directory\",\"storage\":\"acme.json\",\"keyType\":\"RSA4096\",\"dnsChallenge\":{\"provider\":\"cloudflare\"},\"ResolverName\":\"le\",\"store\":{},\"ChallengeStore\":{}}"
  traefik    | time="2020-05-24T16:05:34Z" level=debug msg="Start TCP Server" entryPointName=http
  traefik    | time="2020-05-24T16:05:34Z" level=info msg="Testing certificate renew..." providerName=le.acme
  traefik    | time="2020-05-24T16:05:34Z" level=info msg="Starting provider *docker.Provider {\"watch\":true,\"endpoint\":\"unix:///var/run/docker.sock\",\"defaultRule\":\"Host(`{{ normalize .Name }}`)\",\"network\":\"traefik_proxy\",\"swarmModeRefreshSeconds\":15000000000}"
  traefik    | time="2020-05-24T16:05:34Z" level=info msg="Starting provider *traefik.Provider {}"
  traefik    | time="2020-05-24T16:05:34Z" level=debug msg="Configuration received from provider le.acme: {\"http\":{},\"tls\":{}}" providerName=le.acme
  traefik    | time="2020-05-24T16:05:34Z" level=debug msg="Configuration received from provider internal: {\"http\":{\"services\":{\"api\":{},\"dashboard\":{},\"noop\":{}}},\"tcp\":{},\"tls\":{}}" providerName=internal
  traefik    | time="2020-05-24T16:05:34Z" level=debug msg="No default certificate, generating one"
  traefik    | time="2020-05-24T16:05:34Z" level=debug msg="Provider connection established with docker 19.03.9 (API 1.40)" providerName=docker
  traefik    | time="2020-05-24T16:05:34Z" level=debug msg="Configuration received from provider docker: {\"http\":{\"routers\":{\"traefik\":{\"entryPoints\":[\"https\"],\"middlewares\":[\"traefik_auth\"],\"service\":\"api@internal\",\"rule\":\"Host(`&lt;my domain&gt;`)\",\"tls\":{\"certResolver\":\"le\"}},\"traefik0\":{\"entryPoints\":[\"http\"],\"middlewares\":[\"to_https\"],\"service\":\"traefik-traefik\",\"rule\":\"Host(`&lt;my domain&gt;`)\"}},\"services\":{\"traefik-traefik\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://172.18.0.2:80\"}],\"passHostHeader\":true}}},\"middlewares\":{\"to_https\":{\"redirectScheme\":{\"scheme\":\"https\"}},\"to_https_perm\":{\"redirectScheme\":{\"scheme\":\"https\",\"permanent\":true}},\"traefik_auth\":{\"basicAuth\":{\"users\":[\"&lt;traefik users&gt;\"]}}}},\"tcp\":{},\"udp\":{}}" providerName=docker
  traefik    | time="2020-05-24T16:05:34Z" level=debug msg="No default certificate, generating one"
  traefik    | time="2020-05-24T16:05:34Z" level=debug msg="Creating middleware" serviceName=traefik-traefik entryPointName=http routerName=traefik0@docker middlewareName=pipelining middlewareType=Pipelining
  traefik    | time="2020-05-24T16:05:34Z" level=debug msg="Creating load-balancer" entryPointName=http routerName=traefik0@docker serviceName=traefik-traefik
  traefik    | time="2020-05-24T16:05:34Z" level=debug msg="Creating server 0 http://172.18.0.2:80" routerName=traefik0@docker serviceName=traefik-traefik serverName=0 entryPointName=http
  traefik    | time="2020-05-24T16:05:34Z" level=debug msg="Added outgoing tracing middleware traefik-traefik" middlewareType=TracingForwarder routerName=traefik0@docker entryPointName=http middlewareName=tracing
  traefik    | time="2020-05-24T16:05:34Z" level=debug msg="Creating middleware" entryPointName=http routerName=traefik0@docker middlewareName=to_https@docker middlewareType=RedirectScheme
  traefik    | time="2020-05-24T16:05:34Z" level=debug msg="Setting up redirection to https " entryPointName=http routerName=traefik0@docker middlewareName=to_https@docker middlewareType=RedirectScheme
  traefik    | time="2020-05-24T16:05:34Z" level=debug msg="Adding tracing to middleware" entryPointName=http routerName=traefik0@docker middlewareName=to_https@docker
  traefik    | time="2020-05-24T16:05:34Z" level=debug msg="Creating middleware" middlewareType=Recovery entryPointName=http middlewareName=traefik-internal-recovery
  traefik    | time="2020-05-24T16:05:34Z" level=debug msg="Added outgoing tracing middleware api@internal" middlewareName=tracing middlewareType=TracingForwarder entryPointName=https routerName=traefik@docker
  traefik    | time="2020-05-24T16:05:34Z" level=debug msg="Creating middleware" middlewareType=BasicAuth routerName=traefik@docker entryPointName=https middlewareName=traefik_auth@docker
  traefik    | time="2020-05-24T16:05:34Z" level=debug msg="Adding tracing to middleware" routerName=traefik@docker middlewareName=traefik_auth@docker entryPointName=https
  traefik    | time="2020-05-24T16:05:34Z" level=debug msg="Creating middleware" entryPointName=https middlewareName=traefik-internal-recovery middlewareType=Recovery
  traefik    | time="2020-05-24T16:05:34Z" level=debug msg="No default certificate, generating one"
  traefik    | time="2020-05-24T16:05:34Z" level=debug msg="Try to challenge certificate for domain [&lt;my domain&gt;] found in HostSNI rule" providerName=le.acme rule="Host(`&lt;my domain&gt;`)" routerName=traefik@docker
  traefik    | time="2020-05-24T16:05:34Z" level=debug msg="Looking for provided certificate(s) to validate [\"&lt;my domain&gt;\"]..." providerName=le.acme rule="Host(`&lt;my domain&gt;`)" routerName=traefik@docker
  traefik    | time="2020-05-24T16:05:34Z" level=debug msg="Domains [\"&lt;my domain&gt;\"] need ACME certificates generation for domains \"&lt;my domain&gt;\"." providerName=le.acme rule="Host(`&lt;my domain&gt;`)" routerName=traefik@docker
  traefik    | time="2020-05-24T16:05:34Z" level=debug msg="Loading ACME certificates [&lt;my domain&gt;]..." providerName=le.acme rule="Host(`&lt;my domain&gt;`)" routerName=traefik@docker
  traefik    | time="2020-05-24T16:05:35Z" level=debug msg="Building ACME client..." providerName=le.acme
  traefik    | time="2020-05-24T16:05:35Z" level=debug msg="https://acme-staging-v02.api.letsencrypt.org/directory" providerName=le.acme
  traefik    | time="2020-05-24T16:05:55Z" level=error msg="Unable to obtain ACME certificate for domains \"&lt;my domain&gt;\": cannot get ACME client get directory at 'https://acme-staging-v02.api.letsencrypt.org/directory': Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 127.0.0.11:53: read udp 127.0.0.1:49272-&gt;127.0.0.11:53: i/o timeout" routerName=traefik@docker providerName=le.acme rule="Host(`&lt;my domain&gt;`)"
  </code></pre>
  
  <hr>
  
  <p>Another option now is to use <code>docker run ...</code> instead, so let's try with:</p>
  
  <pre class="lang-sh prettyprint-override"><code>docker run -it \
      -v /var/run/docker.sock:/var/run/docker.sock:ro \
      -v /opt/traefik/acme_testing.json:/acme.json \
      -e CF_API_EMAIL="&lt;Cloudflare Email&gt;" \
      -e CF_API_KEY="&lt;Cloudflare API&gt;" \
      -p 80:80 \
      -p 443:443 \
      --network traefik_proxy \
      --name traefik \
      traefik \
      --log.level=DEBUG \
      --api.dashboard=true \
      --providers.docker=true \
      --providers.docker.exposedbydefault=false \
      --providers.docker.network=traefik_proxy \
      --entrypoints.http.address=:80 \
      --entrypoints.https.address=:443 \
      --certificatesresolvers.le.acme.email="&lt;ACME Email&gt;" \
      --certificatesresolvers.le.acme.storage=acme.json \
      --certificatesresolvers.le.acme.caserver="https://acme-staging-v02.api.letsencrypt.org/directory" \
      --certificatesresolvers.le.acme.dnschallenge=true \
      --certificatesresolvers.le.acme.dnschallenge.provider=cloudflare
  </code></pre>
  
  <p>Which gives:</p>
  
  <pre><code>root@server:/opt/traefik# docker exec -it traefik /bin/sh
  / # nslookup acme-staging-v02.api.letsencrypt.org
  ;; connection timed out; no servers could be reached
  </code></pre>
  
  <p>Alright try again without networks:</p>
  
  <pre class="lang-sh prettyprint-override"><code>docker run -it \
      -v /var/run/docker.sock:/var/run/docker.sock:ro \
      -v /opt/traefik/acme_testing.json:/acme.json \
      -e CF_API_EMAIL="&lt;Cloudflare Email&gt;" \
      -e CF_API_KEY="&lt;Cloudflare API&gt;" \
      -p 80:80 \
      -p 443:443 \
      --name traefik \
      traefik \
      --log.level=DEBUG \
      --api.dashboard=true \
      --providers.docker=true \
      --providers.docker.exposedbydefault=false \
      --providers.docker.network=traefik_proxy \
      --entrypoints.http.address=:80 \
      --entrypoints.https.address=:443 \
      --certificatesresolvers.le.acme.email="&lt;ACME Email&gt;" \
      --certificatesresolvers.le.acme.storage=acme.json \
      --certificatesresolvers.le.acme.caserver="https://acme-staging-v02.api.letsencrypt.org/directory" \
      --certificatesresolvers.le.acme.dnschallenge=true \
      --certificatesresolvers.le.acme.dnschallenge.provider=cloudflare
  </code></pre>
  
  <p>Which leads to:</p>
  
  <pre><code>root@server:/opt/traefik# docker exec -it traefik /bin/sh
  / # nslookup acme-staging-v02.api.letsencrypt.org
  nslookup: write to '192.168.1.233': Connection refused
  Server:         192.168.1.1
  Address:        192.168.1.1:53
  
  Non-authoritative answer:
  acme-staging-v02.api.letsencrypt.org    canonical name = staging.api.letsencrypt.org
  staging.api.letsencrypt.org     canonical name = 56a5f4b0bc8146689ec3e272c43525f9.pacloudflare.com
  Name:   56a5f4b0bc8146689ec3e272c43525f9.pacloudflare.com
  Address: 172.65.46.172
  
  Non-authoritative answer:
  acme-staging-v02.api.letsencrypt.org    canonical name = staging.api.letsencrypt.org
  staging.api.letsencrypt.org     canonical name = 56a5f4b0bc8146689ec3e272c43525f9.pacloudflare.com
  Name:   56a5f4b0bc8146689ec3e272c43525f9.pacloudflare.com
  Address: 2606:4700:60:0:f41b:d4fe:4325:6026
  </code></pre>
  
  <hr>
  
  <p>Through all of this, the acme files stayed empty, so the problem still persists.</p>
  
  <pre><code>root@server:/opt/traefik# ls -Al
  total 12
  -rw------- 1 root    root       0 May 24 00:37 acme.json
  -rw------- 1 root    root       0 May 24 00:37 acme_testing.json
  -rw-rw-r-- 1 root    docker  2406 May 24 18:04 docker-compose.yml
  -rw-rw-r-- 1 root    docker   185 May 23 23:49 .env
  </code></pre>
  
  <hr>
  
  <p>If someone can help to fix this, thank you very much in advance.</p>
  
  <p>If you need more information than even all of the stuff I added, feel free to tell me, so I'll provide it.</p>