title: Docker HAProxy SSL termination with Letsencrypt
body: <p>I currently have a docker setup working with haproxy as a load balancer directing traffic to containers running my web app. I'm trying to add SSL termination to HAProxy and have run into some trouble. When I add <code>DEFAULT_SSL_CERT</code> as an environment variable to my haproxy container I get these errors:</p>
  
  <pre><code>Mar 20 20:15:03 escapes-artist kernel:  [3804709.167813] aufs au_opts_verify:1597:dockerd[1595]: dirperm1 breaks the protection by the permission bits on the lower branch
  Mar 20 20:15:03 escapes-artist kernel:  [3804709.213993] aufs au_opts_verify:1597:dockerd[1595]: dirperm1 breaks the protection by the permission bits on the lower branch
  Mar 20 20:15:04 escapes-artist kernel:  [3804709.674840] aufs au_opts_verify:1597:dockerd[1595]: dirperm1 breaks the protection by the permission bits on the lower branch
  Mar 20 20:15:04 escapes-artist kernel:  [3804709.688631] device vethebd7d1d entered promiscuous mode
  Mar 20 20:15:04 escapes-artist kernel:  [3804709.688767] IPv6: ADDRCONF(NETDEV_UP): vethebd7d1d: link is not ready
  Mar 20 20:15:04 escapes-artist systemd-udevd:  Could not generate persistent MAC address for veth5c0585c: No such file or directory
  Mar 20 20:15:04 escapes-artist systemd-udevd:  Could not generate persistent MAC address for vethebd7d1d: No such file or directory
  Mar 20 20:15:04 escapes-artist dockerd:  time="2017-03-21T02:15:04.671620998Z" level=warning msg="Your kernel does not support swap memory limit."
  Mar 20 20:15:04 escapes-artist dockerd:  time="2017-03-21T02:15:04.672345380Z" level=warning msg="Your kernel does not support cgroup rt period"
  Mar 20 20:15:04 escapes-artist dockerd:  time="2017-03-21T02:15:04.672732724Z" level=warning msg="Your kernel does not support cgroup rt runtime"
  Mar 20 20:15:04 escapes-artist dockerd:  time="2017-03-21T02:15:04Z" level=info msg="Firewalld running: false"
  Mar 20 20:15:05 escapes-artist kernel:  [3804710.392546] eth0: renamed from veth5c0585c
  Mar 20 20:15:05 escapes-artist kernel:  [3804710.395273] IPv6: ADDRCONF(NETDEV_CHANGE): vethebd7d1d: link becomes ready
  Mar 20 20:15:05 escapes-artist kernel:  [3804710.395303] br-5c6735a37ece: port 3(vethebd7d1d) entered forwarding state
  Mar 20 20:15:05 escapes-artist kernel:  [3804710.395313] br-5c6735a37ece: port 3(vethebd7d1d) entered forwarding state
  Mar 20 20:15:05 escapes-artist kernel:  [3804711.072047] br-5c6735a37ece: port 2(vethbaf33bd) entered forwarding state
  Mar 20 20:15:08 escapes-artist kernel:  [3804713.819317] haproxy[29684]: segfault at 7f560000003b ip 00007f56f6ac74bb sp 00007ffe45011290 error 4 in libcrypto.so.1.0.0[7f56f69ce000+3f3000]
  Mar 20 20:15:11 escapes-artist sshd:  Received disconnect from 122.194.229.7 port 21903:11:  [preauth]
  Mar 20 20:15:11 escapes-artist sshd:  Disconnected from 122.194.229.7 port 21903 [preauth]
  Mar 20 20:15:13 escapes-artist kernel:  [3804718.789238] haproxy[29686]: segfault at 7fbb0000003b ip 00007fbb747b74bb sp 00007ffc944fcc10 error 4 in libcrypto.so.1.0.0[7fbb746be000+3f3000]
  Mar 20 20:15:17 escapes-artist kernel:  [3804722.944073] br-5c6735a37ece: port 1(veth610d1f4) entered forwarding state
  Mar 20 20:15:18 escapes-artist kernel:  [3804723.790663] haproxy[29688]: segfault at 7ff10000003b ip 00007ff1ad6004bb sp 00007fffa6f03cb0 error 4 in libcrypto.so.1.0.0[7ff1ad507000+3f3000]
  Mar 20 20:15:20 escapes-artist kernel:  [3804725.408060] br-5c6735a37ece: port 3(vethebd7d1d) entered forwarding state
  Mar 20 20:15:23 escapes-artist kernel:  [3804728.792134] haproxy[29690]: segfault at 7f130000003b ip 00007f13210c54bb sp 00007ffcbe3f7670 error 4 in libcrypto.so.1.0.0[7f1320fcc000+3f3000]
  Mar 20 20:15:28 escapes-artist kernel:  [3804733.823940] haproxy[29692]: segfault at 7f500000003b ip 00007f500b9d94bb sp 00007ffe6d044f10 error 4 in libcrypto.so.1.0.0[7f500b8e0000+3f3000]
  Mar 20 20:15:33 escapes-artist kernel:  [3804738.780797] haproxy[29694]: segfault at 7f000000003b ip 00007f00310124bb sp 00007fffd6e979b0 error 4 in libcrypto.so.1.0.0[7f0030f19000+3f3000]
  </code></pre>
  
  <p>Does anyone know how to fix this? I've experimented for hours trying different formats for the cert file, environment variables, etc. and can't seem to figure anything out. Here is the <code>docker-compose.yml</code> file I'm using:</p>
  
  <pre><code>version: '2'
  services:
    db:
      image: mysql
      restart: always
      environment:
        MYSQL_ROOT_PASSWORD: password
        MYSQL_DATABASE: docker
        MYSQL_USER: admin
        MYSQL_PASSWORD: password
      volumes:
        - /storage/docker/mysql-datadir:/var/lib/mysql
      ports:
        - 3306:3306
    web:
      image: myimage
      restart: always
      depends_on:
        - db
      volumes:
        - /home/docker/persistent/media/:/home/docker/code/media/
    lb:
      image: dockercloud/haproxy
      links:
        - web
      volumes:
        - /var/run/docker.sock:/var/run/docker.sock
        - /etc/haproxy/certs:/certs
      environment:
        STATS_AUTH: admin:password
        RSYSLOG_DESTINATION: logs5.papertrailapp.com:41747
        DEFAULT_SSL_CERT: (I've tried both pasting cert here directly and a path to cert)
      ports:
        - 80:80
        - 443:443
        - 1936:1936
  </code></pre>
  
  <p>I have Letsencrypt setup on the host machine to autorenew. The cert that I've been trying to use is a combination of the privkey.pem and fullchian.pem.  I've tried concatenating them, using <code>awk 1 ORS='\\n'</code> like the dockercloud/haproxy <a href="https://github.com/docker/dockercloud-haproxy#ssl-termination" rel="nofollow noreferrer">docs</a> suggest, and just about every other configuration I can think of. Any help would be greatly appreciated.</p>
  
  <p>Also, if I use <code>CERT_FOLDER: /certs/</code> instead of <code>DEFAULT_SSL_CERT</code> and have my certificate stored in <code>/certs/cert0.pem</code> I get this error instead...</p>
  
  <pre><code>Mar 20 21:19:38 escapes-artist dockerd:  time="2017-03-21T03:19:38.840340234Z" level=error msg="containerd: deleting container" error="exit status 1: \"container ce6c0b6df31419691b6593be6744d01c8ccecf5f38851106aa4bb8fac915a63a does not exist\\none or more of the container deletions failed\\n\""
  Mar 20 21:19:38 escapes-artist kernel:  [3808584.302038] br-5c6735a37ece: port 3(veth8b1ea8e) entered disabled state
  Mar 20 21:19:38 escapes-artist kernel:  [3808584.302192] veth0bcd06c: renamed from eth0
  Mar 20 21:19:38 escapes-artist kernel:  [3808584.320863] br-5c6735a37ece: port 3(veth8b1ea8e) entered disabled state
  Mar 20 21:19:38 escapes-artist kernel:  [3808584.321869] device veth8b1ea8e left promiscuous mode
  Mar 20 21:19:38 escapes-artist kernel:  [3808584.321874] br-5c6735a37ece: port 3(veth8b1ea8e) entered disabled state
  Mar 20 21:19:39 escapes-artist dockerd:  time="2017-03-21T03:19:39.055316431Z" level=error msg="Handler for GET /v1.25/exec/c79e3c9b77f0c84d849cc641a425950d55fcbb22bf566922d3fd12e6a0e12e07/json returned error: Container ce6c0b6df31419691b6593be6744d01c8ccecf5f38851106aa4bb8fac915a63a is not running: Exited (0) Less than a second ago"
  Mar 20 21:19:39 escapes-artist kernel:  [3808584.964578] aufs au_opts_verify:1597:dockerd[23058]: dirperm1 breaks the protection by the permission bits on the lower branch
  Mar 20 21:19:39 escapes-artist kernel:  [3808585.005699] aufs au_opts_verify:1597:dockerd[23058]: dirperm1 breaks the protection by the permission bits on the lower branch
  Mar 20 21:19:40 escapes-artist kernel:  [3808585.489799] aufs au_opts_verify:1597:dockerd[1595]: dirperm1 breaks the protection by the permission bits on the lower branch
  Mar 20 21:19:40 escapes-artist kernel:  [3808585.500609] device veth24d6316 entered promiscuous mode
  Mar 20 21:19:40 escapes-artist systemd-udevd:  Could not generate persistent MAC address for veth24d6316: No such file or directory
  Mar 20 21:19:40 escapes-artist kernel:  [3808585.505055] IPv6: ADDRCONF(NETDEV_UP): veth24d6316: link is not ready
  Mar 20 21:19:40 escapes-artist systemd-udevd:  Could not generate persistent MAC address for vethedaad7c: No such file or directory
  Mar 20 21:19:40 escapes-artist dockerd:  time="2017-03-21T03:19:40.259076690Z" level=warning msg="Your kernel does not support swap memory limit."
  Mar 20 21:19:40 escapes-artist dockerd:  time="2017-03-21T03:19:40.260183880Z" level=warning msg="Your kernel does not support cgroup rt period"
  Mar 20 21:19:40 escapes-artist dockerd:  time="2017-03-21T03:19:40.260663645Z" level=warning msg="Your kernel does not support cgroup rt runtime"
  Mar 20 21:19:40 escapes-artist dockerd:  time="2017-03-21T03:19:40Z" level=info msg="Firewalld running: false"
  Mar 20 21:19:40 escapes-artist kernel:  [3808585.904671] eth0: renamed from vethedaad7c
  Mar 20 21:19:40 escapes-artist kernel:  [3808585.918744] IPv6: ADDRCONF(NETDEV_CHANGE): veth24d6316: link becomes ready
  Mar 20 21:19:40 escapes-artist kernel:  [3808585.919040] br-5c6735a37ece: port 3(veth24d6316) entered forwarding state
  Mar 20 21:19:40 escapes-artist kernel:  [3808585.919058] br-5c6735a37ece: port 3(veth24d6316) entered forwarding state
  Mar 20 21:19:44 escapes-artist kernel:  [3808589.585674] haproxy[32235]: segfault at 341 ip 0000000000000341 sp 00007ffe732fe5b8 error 14 in haproxy[55f6998b1000+d1000]
  Mar 20 21:19:49 escapes-artist kernel:  [3808594.704226] haproxy[32237]: segfault at 341 ip 0000000000000341 sp 00007ffcb4d1aa08 error 14 in haproxy[563827d10000+d1000]
  Mar 20 21:19:54 escapes-artist kernel:  [3808599.669540] haproxy[32239]: segfault at 341 ip 0000000000000341 sp 00007ffd1e8bb1b8 error 14 in haproxy[562d926fa000+d1000]
  Mar 20 21:19:55 escapes-artist kernel:  [3808600.928110] br-5c6735a37ece: port 3(veth24d6316) entered forwarding state
  Mar 20 21:19:59 escapes-artist kernel:  [3808604.602704] haproxy[32241]: segfault at 341 ip 0000000000000341 sp 00007fff142d0898 error 14 in haproxy[5592e3a63000+d1000]
  </code></pre>