title: Docker swarm is not correctly copying volumes across other worker nodes
body: <p>I'm trying to build a Laravel application (LEMP Stack) in Docker Swarm. I'm using <a href="https://docs.docker.com/engine/swarm/ingress/" rel="nofollow noreferrer">HaProxy</a>, as described in Dockers networking documentation on ingress networks, to load balance from the manager nodes public IPV4 between all worker nodes.</p>
  <p>My current configuration looks like so:</p>
  <pre><code>frontend http_front
     bind *:80
     stats uri /haproxy?stats
     default_backend http_back
  
  frontend http_front_ssl
      bind *:443
      stats uri /haproxy?stats
      default_backend http_back_ssl
  
  backend http_back
     balance roundrobin
     server vps1 xx.xx.xxx.xxx:8080 check
     server vps2 xx.xx.xxx.xxx:8080 check
     server vps3 xx.xx.xxx.xxx:8080 check
  
  backend http_back_ssl
     balance roundrobin
     server vps1 xx.xx.xxx.xxx:4043 check
     server vps2 xx.xx.xxx.xxx:4043 check
     server vps3 xx.xx.xxx.xxx:4043 check
  </code></pre>
  <p>I have then a custom image that provides and exposes a PHP-FPM on port 9000:</p>
  <pre><code>docker service create \
  --name php-fpm \
  --replicas=3 \
  --network app-net \
  --secret source=proxy.conf,target=/etc/nginx/conf.d/site.conf \
  --secret site.key \
  --secret site.crt \
  --mount type=volume,src=web,dst=/var/www \
  jaquarh/php-fpm:alpha
  </code></pre>
  <p>The keys are for self-signed SSL (testing Https). The Dockerfile to build this is:</p>
  <pre><code>FROM php:7.4-fpm
  COPY laravel/composer.lock laravel/composer.json /var/www/
  WORKDIR /var/www
  
  RUN apt-get update &amp;&amp; apt-get install -y \
      build-essential \
      libpng-dev \
      libjpeg62-turbo-dev \
      libfreetype6-dev \
      locales \
      zip \
      jpegoptim optipng pngquant gifsicle \
      vim \
      unzip \
      git \
      curl
  
  RUN apt-get clean &amp;&amp; rm -rf /var/lib/apt/lists/*
  
  RUN docker-php-ext-install pdo_mysql
  RUN docker-php-ext-configure gd --with-freetype --with-jpeg
  RUN docker-php-ext-install gd
  
  RUN pecl install memcached; \
      docker-php-ext-install intl; \
      docker-php-ext-install exif; \
      echo &quot;extension=memcached.so&quot; &gt;&gt; /usr/local/etc/php/conf.d/memcached.ini; \
      docker-php-ext-enable exif;
  
  RUN curl -sS https://getcomposer.org/installer | php -- --install-dir=/usr/local/bin --filename=composer
  
  RUN groupadd -g 1000 www
  RUN useradd -u 1000 -ms /bin/bash -g www www
  
  COPY . /var/www
  COPY --chown=www:www . /var/www
  
  USER www
  
  EXPOSE 9000
  CMD [&quot;php-fpm&quot;]
  </code></pre>
  <p>I am then building an nginx service as the proxy for the php-fpm service.</p>
  <pre><code>docker service create \
  --name nginx \
  --secret site.key \
  --secret site.crt \
  --secret source=site.conf,target=/etc/nginx/conf.d/site.conf \
  -p 8080:80 \
  -p 4043:443 \
  --replicas=3 \
  --network app-net \
  --mount type=volume,src=web,dst=/var/www \
  nginx:latest \
  sh -c &quot;exec nginx -g 'daemon off;'&quot;
  </code></pre>
  <p>Inside of my <code>web</code> volume, I hold all the laravel files. I am only testing Http at the moment so my nginx configuration looks like so:</p>
  <pre><code>server {
      listen 80;
      listen [::]:80;
      server_name example.co.uk; # this is my actual domain for testing
  
      index index.php index.html;
  
      error_log /var/log/nginx/error.log;
      access_log /var/log/nginx/access.log;
  
      root /var/www/public;
  
      add_header X-Frame-Options &quot;SAMEORIGIN&quot;;
      add_header X-XSS-Protection &quot;1; mode=block&quot;;
      add_header X-Content-Type-Options &quot;nosniff&quot;;
  
      charset utf-8;
      
      location / {
          try_files $uri $uri/ /index.php?$query_string;
          gzip_static on;
      }
  
     location = /favicon.ico {
          access_log off;
          log_not_found off;
      }
  
      location = /robots.txt {
         access_log off;
          log_not_found off;
     }
  
      error_page 404 /index.php;
  
     location ~ \.php$ {
          try_files $uri =404;
          fastcgi_split_path_info ^(.+\.php)(/.+)$;
          fastcgi_pass php-fpm:9000;
          fastcgi_index index.php;
         fastcgi_param SCRIPT_FILENAME $realpath_root$fastcgi_script_name;
          fastcgi_param PATH_INFO $fastcgi_path_info;
         include fastcgi_params;
      }
  
      # ready for when letsencrypt is introduced
      location ~ /.well-known/acme-challenge {
         allow all;
         root /var/www/html;
      }
  }
  </code></pre>
  <p>Finally, I have a pre-built SQL database volume and service ready to use which the laravel <code>.env</code> uses.</p>
  <pre><code>docker service create \
  --name mysql \
  --replicas=3 \
  --network app-net \
  --secret site.key \
  --secret site.crt \
  --mount type=volume,src=dbdata,target=/var/lib/mysql/ \
  --env MYSQL_DATABASE=xxxxxx \
  --env MYSQL_ROOT_PASSWORD=xxxxxx \
  --env MYSQL_USER=xxxxx \
  --env MYSQL_PASSWORD=xxxxx \
  -p 3306:3306 \
  mysql:latest
  </code></pre>
  <p>The issue I am facing is that VPS1 (in this case the manager) gets these files perfectly fine because the volumes exist in the manager VPS. Both VPS2 and VPS3 show these volumes have been mounted correctly but are not copying the data over.</p>
  <p><a href="https://i.stack.imgur.com/PKugU.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/PKugU.png" alt="Mount" /></a></p>
  <p><a href="https://i.stack.imgur.com/8mW2e.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/8mW2e.png" alt="Different" /></a></p>
  <p>As seen in the images, on my development pc (which is also a worker for testing) I can see the mount has been successfully created and my docker is assigned the nginx container however, when I inspect the directories in bash, only the directory housing the laravel application exists in the manager node.</p>
  <p>What am I doing wrong? How can I copy this volume across my worker nodes? If I inspect the volume on the manager it looks like this:</p>
  <pre><code>ubuntu@vps-438f3f94:~$ docker volume inspect web
  [
      {
          &quot;CreatedAt&quot;: &quot;2021-04-23T09:29:23Z&quot;,
          &quot;Driver&quot;: &quot;local&quot;,
          &quot;Labels&quot;: {},
          &quot;Mountpoint&quot;: &quot;/var/lib/docker/volumes/web/_data&quot;,
          &quot;Name&quot;: &quot;web&quot;,
          &quot;Options&quot;: {},
          &quot;Scope&quot;: &quot;local&quot;
      }
  ]
  </code></pre>
  <p><strong>Update:</strong> If I look at the HaProxy stats, I can see my development PC is <code>DOWN</code> because I'm not port forwarded (as its just for me to debug containers) but I'm assuming this has no real effect on volumes but just to give you an insight to my environment better.</p>
  <pre><code>http_back   
  Queue   Session rate    Sessions    Bytes   Denied  Errors  Warnings    Server
  Cur Max Limit   Cur Max Limit   Cur Max Limit   Total   LbTot   Last    In  Out Req Resp    Req Conn    Resp    Retr    Redis   Status  LastChk Wght    Act Bck Chk Dwn Dwntme  Thrtle
  vps1    0   0   -   0   3       0
  1   -   38
  38  25m15s  14191   23441       0       0   0   0   0   25m30s UP   L4OK in 0ms 1   Y   -   28  10  0s  -
  vps2    0   0   -   0   1       0
  1   -   30
  30  25m6s   10055   16896       0       0   0   0   0   25m28s UP   L4OK in 0ms 1   Y   -   30  10  0s  -
  vps3    0   0   -   0   0       0
  0   -   0
  0   ?   0   0       0       0   0   0   0   13h5m DOWN  * L4TOUT in 2001ms  1   Y   -   1   1   13h5m   -
  Backend 0   0       0   8       0   1   26212   121
  68  25m6s   43399   52050
  0   0       53  0   0   0   25m30s UP       2   2   0       9   2h20m   
  </code></pre>