title: Docker fails to pull the image from within Azure App Service
body: <p>The Container Setting on the App Service it self look solid:
  <a href="https://i.stack.imgur.com/V4Ejm.png" rel="noreferrer"><img src="https://i.stack.imgur.com/V4Ejm.png" alt="enter image description here" /></a></p>
  <p>But the log pane shows errors:</p>
  <p><a href="https://i.stack.imgur.com/1ErOu.png" rel="noreferrer"><img src="https://i.stack.imgur.com/1ErOu.png" alt="enter image description here" /></a></p>
  <pre><code>2020-02-11 06:31:40.621 ERROR - Image pull failed: Verify docker image configuration and credentials (if using private repository)
  2020-02-11 06:31:41.240 INFO  - Stoping site app505-dfpg-qa2-web-eastus2-gateway-apsvc because it failed during startup.
  2020-02-11 06:36:05.546 INFO  - Starting container for site
  2020-02-11 06:36:05.551 INFO  - docker run -d -p 9621:8081 --name app505-dfpg-qa2-web-eastus2-gateway-apsvc_0_a9c8277e_msiProxy -e WEBSITE_SITE_NAME=app505-dfpg-qa2-web-eastus2-gateway-apsvc -e WEBSITE_AUTH_ENABLED=False -e WEBSITE_ROLE_INSTANCE_ID=0 -e WEBSITE_HOSTNAME=app505-dfpg-qa2-web-eastus2-gateway-apsvc.azurewebsites.net -e WEBSITE_INSTANCE_ID=7d18d5957d129d3dc3a25d7a2c85147ef57f1a6b93910c50eb850417ab59dc56 appsvc/msitokenservice:1904260237  
  
  2020-02-11 06:36:05.552 INFO  - Logging is not enabled for this container.
  Please use https://aka.ms/linux-diagnostics to enable logging to see container logs here.
  2020-02-11 06:36:17.766 INFO  - Pulling image: a...cr/gateway:1.0.20042.2
  2020-02-11 06:36:17.922 ERROR - DockerApiException: Docker API responded with status code=NotFound, response={&quot;message&quot;:&quot;pull access denied for a...cr/gateway, repository does not exist or may require 'docker login': denied: requested access to the resource is denied&quot;}
  
  2020-02-11 06:36:17.923 ERROR - Pulling docker image a...cr/gateway:1.0.20042.2 failed:
  2020-02-11 06:36:17.923 INFO  - Pulling image from Docker hub: a...cr/gateway:1.0.20042.2
  2020-02-11 06:36:18.092 ERROR - DockerApiException: Docker API responded with status code=NotFound, response={&quot;message&quot;:&quot;pull access denied for a...cr/gateway, repository does not exist or may require 'docker login': denied: requested access to the resource is denied&quot;}
  
  2020-02-11 06:36:18.094 ERROR - Image pull failed: Verify docker image configuration and credentials (if using private repository)
  2020-02-11 06:36:19.062 INFO  - Stoping site app505-dfpg-qa2-web-eastus2-gateway-apsvc because it failed during startup.
  </code></pre>
  <p>The Service Principal used to deploy the App Service has AcrPush access to the parent resource group of the container registry:
  <a href="https://i.stack.imgur.com/Euvsx.png" rel="noreferrer"><img src="https://i.stack.imgur.com/Euvsx.png" alt="enter image description here" /></a></p>
  <p>The setting are present:</p>
  <p><a href="https://i.stack.imgur.com/PqIcU.png" rel="noreferrer"><img src="https://i.stack.imgur.com/PqIcU.png" alt="enter image description here" /></a></p>
  <p>I did <code>az login</code> with that service principal and then tried <code>az acr login</code> to the registry. It works fine. So what am I missing here?</p>
  <p><strong>EDIT 1</strong></p>
  <p>I know the credentials are correct, because I tested them like this:
  <a href="https://i.stack.imgur.com/Ux9H2.png" rel="noreferrer"><img src="https://i.stack.imgur.com/Ux9H2.png" alt="enter image description here" /></a></p>
  <p>Where I just copied the values from the app service configuration and pasted on the console. docker has no problem logging in.</p>
  <p>It must be something else.</p>
  <p><strong>EDIT 2</strong></p>
  <p>However, I also get this:</p>
  <pre><code>C:\Dayforce\fintech [shelve/terraform ≡]&gt; docker pull a...r/gateway
  Using default tag: latest
  Error response from daemon: pull access denied for a...r/gateway, repository does not exist or may require 'docker login': denied: requested access to the resource is denied
  </code></pre>
  <p>So, I can login, but not pull. Very strange, because the account is configured to have <code>AcrPush</code> access to the container, which includes <code>AcrPull</code>:
  <a href="https://i.stack.imgur.com/BtSJI.png" rel="noreferrer"><img src="https://i.stack.imgur.com/BtSJI.png" alt="enter image description here" /></a></p>
  <p><strong>EDIT 3</strong></p>
  <p>I was able to pull successfully when using the FQDN for the registry:
  <a href="https://i.stack.imgur.com/l2cSj.png" rel="noreferrer"><img src="https://i.stack.imgur.com/l2cSj.png" alt="enter image description here" /></a></p>
  <p>I updated the pipeline, but I still get the same errors:</p>
  <pre><code>2020-02-11 16:03:50.227 ERROR - Pulling docker image a...r.azurecr.io/gateway:1.0.20042.2 failed:
  2020-02-11 16:03:50.228 INFO  - Pulling image from Docker hub: a...r.azurecr.io/gateway:1.0.20042.2
  2020-02-11 16:03:50.266 ERROR - DockerApiException: Docker API responded with status code=InternalServerError, response={&quot;message&quot;:&quot;Get https://a...r.azurecr.io/v2/gateway/manifests/1.0.20042.2: unauthorized: authentication required&quot;}
  
  2020-02-11 16:03:50.269 ERROR - Image pull failed: Verify docker image configuration and credentials (if using private repository)
  2020-02-11 16:03:50.853 INFO  - Stoping site app505-dfpg-qa2-web-eastus2-gateway-apsvc because it failed during startup.
  </code></pre>
  <p><strong>EDIT 4</strong></p>
  <p>The only way that I found working was to enable the Admin User on the ACR and pass its credentials in the DOCKER_... variables instead of credentials of the Service Principal.</p>
  <p>This is frustrating, I know the Service Principal can login and pull when ran locally, it is a mystery why it does not work for docker running on an App Service Host. We have another team here which faced the same issue and they have not found any solution, but enable the Admin User.</p>
  <p><strong>EDIT 5</strong></p>
  <p>The entire process runs as part of the Azure DevOps on-prem release pipeline using a dedicated Service Principal. Let me call it Pod Deploy Service Principal or just <strong>SP</strong> for short.</p>
  <p>Let <code>DOCKER_xyz</code> denote the three app settings controlling the docker running on the App Service host:</p>
  <ul>
  <li><code>DOCKER_REGISTRY_SERVER_URL</code></li>
  <li><code>DOCKER_REGISTRY_SERVER_USERNAME</code></li>
  <li><code>DOCKER_REGISTRY_SERVER_PASSWORD</code></li>
  </ul>
  <p>I think we need to distinguish two parts here:</p>
  <ol>
  <li>App Service needs to talk to the ACR in order to pull from it the details about the image and present them in this GUI - <a href="https://i.stack.imgur.com/V4Ejm.png" rel="noreferrer"><img src="https://i.stack.imgur.com/V4Ejm.png" alt="enter image description here" /></a> For that to work, the SP must have the AcrPull role in the ACR. Failure to do so results in the GUI presenting a spinning icon for the Image and Tag rows. I stumbled on it before - <a href="https://stackoverflow.com/questions/59914397/how-to-configure-an-azure-app-service-to-pull-images-from-an-acr-with-terraform">How to configure an Azure app service to pull images from an ACR with terraform?</a> Now the answer to that question suggests that I have to assign the AcrPull role <strong>and</strong> set the <code>DOCKER_xyz</code> app settings. I think that the <code>DOCKER_xyz</code> app settings are not for that, but for the second part.</li>
  <li>It seems to me that when an App Service is started, the host uses docker to actually pull the right image from the ACR. This part seems to be detached from (1). For it to work, the app settings must have the <code>DOCKER_xyz</code> app settings.</li>
  </ol>
  <p>My problem is that part (1) works great, but part (2) does not even if <code>DOCKER_xyz</code> app settings specify the credentials of the SP from part (1). The only way I could make it work if I point <code>DOCKER_xyz</code> at the Admin User of the ACR.</p>
  <p>But that why on Earth the <code>DOCKER_xyz</code> app settings cannot point to the pipeline SP, which was good enough for the part (1)?</p>
  <p><strong>EDIT 6</strong></p>
  <p>The current state of affairs is this. Azure App Service is unable to communicate with an ACR except using ACR admin user and password. So, even if the docker runtime running on the App Service host machine may know how to login using any service principal, the App Service would not use any identity or Service Principal to read metadata from the ACR - only admin user and password. The relevant references are:</p>
  <ul>
  <li><a href="https://feedback.azure.com/forums/169385-web-apps/suggestions/36145444-web-app-for-containers-acr-access-requires-admin#%7btoggle_previous_statuses%7d" rel="noreferrer">https://feedback.azure.com/forums/169385-web-apps/suggestions/36145444-web-app-for-containers-acr-access-requires-admin#%7btoggle_previous_statuses%7d</a></li>
  <li><a href="https://github.com/MicrosoftDocs/azure-docs/issues/49186" rel="noreferrer">https://github.com/MicrosoftDocs/azure-docs/issues/49186</a></li>
  </ul>
  <p>On a personal note I find it amazing that Microsoft recommends not to use ACR admin user, yet a very core piece of their offering, namely Azure App Service, depends on it being enable. Makes me wonder whether different teams in Microsoft are aware of what others are doing or not doing...</p>