title: How to add new user to docker image when running distributed airflow architecture using docker-compose
body: <p>(THE ORIGINAL QUESTION WAS EDITED TO MAKE IT MORE CLEAR)</p>
  <ol>
  <li>SOLUTION AT THE END OF THE QUESTION</li>
  <li>ANOTHER SOLUTION IN THE ANSWER</li>
  </ol>
  <p>The goal and the setup</p>
  <p>The main goal is to run container based processing(using the DockerOperator) when the airflow celery worker is also running inside a docker container. At the moment, I'm testing the setup at one machine, but in the end I'll run the celery worker containers at separate machines operating in the same network sharing some of the airflow specific mount points(dags,logs,plugins) and user ids etc.</p>
  <p>I'm launching the whole setup from a docker-compose.yml where I set AIRFLOW_UID to match my UID at the host machine and AIRFLOW_GID to 0 as suggested in the airflow documentation. At the host, my UID belongs to docker group, but it doesn't belong to group 0. The <code>/var/run/docker.sock</code> is mounted into the containers.</p>
  <p>TEST 1</p>
  <p>I followed the example represented here <a href="https://towardsdatascience.com/using-apache-airflow-dockeroperator-with-docker-compose-57d0217c8219" rel="nofollow noreferrer">https://towardsdatascience.com/using-apache-airflow-dockeroperator-with-docker-compose-57d0217c8219</a> .
  Using the above-mentioned setup with the official airflow image 2.1.4 and DockerOperator. Task run fails, which is related to the fact that the default user doesn't have the needed permissions to <code>/var/run/docker.sock</code>. (I still need to check if adding the user to group 0 at the host would solve the issue as pointed out by @JarekPotiuk in the his comment. The problem is that group 0 is the root group and most likely I'll not get permission to add the user to it)</p>
  <pre><code>[2021-09-27 05:38:30,863] {taskinstance.py:1463} ERROR - Task failed with exception
  Traceback (most recent call last):
    File &quot;/home/airflow/.local/lib/python3.6/site-packages/urllib3/connectionpool.py&quot;, line 706, in urlopen
      chunked=chunked,
    File &quot;/home/airflow/.local/lib/python3.6/site-packages/urllib3/connectionpool.py&quot;, line 394, in _make_request
      conn.request(method, url, **httplib_request_kw)
    File &quot;/usr/local/lib/python3.6/http/client.py&quot;, line 1291, in request
      self._send_request(method, url, body, headers, encode_chunked)
    File &quot;/usr/local/lib/python3.6/http/client.py&quot;, line 1337, in _send_request
      self.endheaders(body, encode_chunked=encode_chunked)
    File &quot;/usr/local/lib/python3.6/http/client.py&quot;, line 1286, in endheaders
      self._send_output(message_body, encode_chunked=encode_chunked)
    File &quot;/usr/local/lib/python3.6/http/client.py&quot;, line 1046, in _send_output
      self.send(msg)
    File &quot;/usr/local/lib/python3.6/http/client.py&quot;, line 984, in send
      self.connect()
    File &quot;/home/airflow/.local/lib/python3.6/site-packages/docker/transport/unixconn.py&quot;, line 30, in connect
      sock.connect(self.unix_socket)
  PermissionError: [Errno 13] Permission denied
  </code></pre>
  <p>TEST 2</p>
  <p>I created custom image from the official image by adding 'newuser' with an UID that matches my UID at the host and 'docker' group that matches the one at the host.</p>
  <p>However, when I launch the setup, the user I created in the image build phase is not there and I can't understand why. There is a 'default' user with uid=1234 and gid=0. This default user is created if I use the official Image and just define AIRFLOW_UID in the docker-compose.yml.</p>
  <p>Dockerfile:</p>
  <pre><code>FROM apache/airflow:2.1.0
  
  USER root
  RUN useradd newuser -u 1234 -g 0
  
  RUN groupadd --gid 986 docker \
      &amp;&amp; usermod -aG docker newuser
  USER newuser
  </code></pre>
  <p>Also, if I don't create the newuser and just add airflow user to docker group then the airflow user is really added to the docker group as it should.</p>
  <p>Does docker-compose overwrite the users created at the image build phase? What would be the best way to solve this issue?</p>
  <p>SOLUTION</p>
  <p>This solution makes it possible to user DockerOperator from airflow container to launch DockerContainers at host.</p>
  <p>You can choose either the default UID=50000 and GID=0 or a custom UID and GID=0. Create a docker group at the host and add the chosen UID to it. Then add the airflow user inside the container into the docker group. You can do this by adding the group in the compose file</p>
  <pre><code>group_add:
    - &lt;docker GID&gt;
  </code></pre>
  <p>In addition, you have to mount the docker.sock file to the container</p>
  <pre><code>volumes:
    - /var/run/docker.sock:/var/run/docker.sock
  </code></pre>
  <p>and add a variable AIRFLOW__CORE__ENABLE_XCOM_PICKLING=True</p>