<p>I have gone through all the motions and I have what appears to be a common problem. Unfortunately, all of the solutions I've tried from github and SO have yet to work. Here's the error:  </p>

<pre><code>Warning  Failed     4m (x4 over 5m)    kubelet, aks-agentpool-97052351-0  Failed to pull image "ussmicroserviceregistry.azurecr.io/simpledotnetapi_simpledotnetapi": [rpc error: code = Unknown desc = Error response from daemon: Get https://ussmicroserviceregistry.azurecr.io/v2/simpledotnetapi_simpledotnetapi/manifests/latest: unauthorized: authentication required, rpc error: code = Unknown desc = Error response from daemon: Get https://ussmicroserviceregistry.azurecr.io/v2/simpledotnetapi_simpledotnetapi/manifests/latest: unauthorized: authentication required, rpc error: code = Unknown desc = Error response from daemon: Get https://ussmicroserviceregistry.azurecr.io/v2/simpledotnetapi_simpledotnetapi/manifests/latest: unauthorized: authentication required]
</code></pre>

<p>-- created the service principal</p>

<pre><code>az ad sp create-for-rbac 
  --scopes /subscriptions/11870e73-bdb2-47b0-bf27-25d24c41ae24/resourcegroups/USS-MicroService-Test/providers/Microsoft.ContainerRegistry/registries/UssMicroServiceRegistry
  --role Reader 
  --name kimage-reader
</code></pre>

<p>-- created the secret for Kube</p>

<pre><code>kubectl create secret docker-registry kimagereadersecret --docker-server ussmicroserviceregistry.azurecr.io --docker-email coreyp@united-systems.com --docker-username=kimage-reader --docker-password 4b37b896-a04e-48b4-a950-5f1abdd3e7aa
</code></pre>

<p>-- <code>kubectl.exe describe pod simpledotnetapi-deployment-6fbf97df55-2hg2m</code></p>

<pre><code>Name:               simpledotnetapi-deployment-6fbf97df55-2hg2m
Namespace:          default
Priority:           0
PriorityClassName:  &lt;none&gt;
Node:               aks-agentpool-97052351-0/10.240.0.4
Start Time:         Mon, 17 Jun 2019 15:22:30 -0500
Labels:             app=simpledotnetapi-pod
                    pod-template-hash=6fbf97df55
Annotations:        &lt;none&gt;
Status:             Pending
IP:                 10.240.0.26
Controlled By:      ReplicaSet/simpledotnetapi-deployment-6fbf97df55
Containers:
  simpledotnetapi-simpledotnetapi:
    Container ID:
    Image:          ussmicroserviceregistry.azurecr.io/simpledotnetapi_simpledotnetapi
    Image ID:
    Port:           5000/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       ImagePullBackOff
    Ready:          False
    Restart Count:  0
    Environment:    &lt;none&gt;
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-hj9b5 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  default-token-hj9b5:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-hj9b5
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  &lt;none&gt;
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age                From                               Message
  ----     ------     ----               ----                               -------
  Normal   Scheduled  5m                 default-scheduler                  Successfully assigned default/simpledotnetapi-deployment-6fbf97df55-2hg2m to aks-agentpool-97052351-0
  Normal   BackOff    4m (x6 over 5m)    kubelet, aks-agentpool-97052351-0  Back-off pulling image "ussmicroserviceregistry.azurecr.io/simpledotnetapi_simpledotnetapi"
  Normal   Pulling    4m (x4 over 5m)    kubelet, aks-agentpool-97052351-0  pulling image "ussmicroserviceregistry.azurecr.io/simpledotnetapi_simpledotnetapi"
  Warning  Failed     4m (x4 over 5m)    kubelet, aks-agentpool-97052351-0  Failed to pull image "ussmicroserviceregistry.azurecr.io/simpledotnetapi_simpledotnetapi": [rpc error: code = Unknown desc = Error response from daemon: Get https://ussmicroserviceregistry.azurecr.io/v2/simpledotnetapi_simpledotnetapi/manifests/latest: unauthorized: authentication required, rpc error: code = Unknown desc = Error response from daemon: Get https://ussmicroserviceregistry.azurecr.io/v2/simpledotnetapi_simpledotnetapi/manifests/latest: unauthorized: authentication required, rpc error: code = Unknown desc = Error response from daemon: Get https://ussmicroserviceregistry.azurecr.io/v2/simpledotnetapi_simpledotnetapi/manifests/latest: unauthorized: authentication required]
  Warning  Failed     4m (x4 over 5m)    kubelet, aks-agentpool-97052351-0  Error: ErrImagePull
  Warning  Failed     24s (x22 over 5m)  kubelet, aks-agentpool-97052351-0  Error: ImagePullBackOff
</code></pre>

<p>-- <code>kubectl.exe get pod simpledotnetapi-deployment-6fbf97df55-2hg2m -o yaml</code></p>

<pre><code>apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: 2019-06-17T20:22:30Z
  generateName: simpledotnetapi-deployment-6fbf97df55-
  labels:
    app: simpledotnetapi-pod
    pod-template-hash: 6fbf97df55
  name: simpledotnetapi-deployment-6fbf97df55-2hg2m
  namespace: default
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: ReplicaSet
    name: simpledotnetapi-deployment-6fbf97df55
    uid: a99e4ac8-8ec3-11e9-9bf8-86d46846735e
  resourceVersion: "813190"
  selfLink: /api/v1/namespaces/default/pods/simpledotnetapi-deployment-6fbf97df55-2hg2m
  uid: a1c220a2-913d-11e9-801a-c6aef815c06a
spec:
  containers:
  - image: ussmicroserviceregistry.azurecr.io/simpledotnetapi_simpledotnetapi
    imagePullPolicy: Always
    name: simpledotnetapi-simpledotnetapi
    ports:
    - containerPort: 5000
      protocol: TCP
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: default-token-hj9b5
      readOnly: true
  dnsPolicy: ClusterFirst
  imagePullSecrets:
  - name: kimagereadersecret
  nodeName: aks-agentpool-97052351-0
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: default
  serviceAccountName: default
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - name: default-token-hj9b5
    secret:
      defaultMode: 420
      secretName: default-token-hj9b5
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: 2019-06-17T20:22:30Z
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: 2019-06-17T20:22:30Z
    message: 'containers with unready status: [simpledotnetapi_simpledotnetapi]'
    reason: ContainersNotReady
    status: "False"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: 2019-06-17T20:22:30Z
    message: 'containers with unready status: [simpledotnetapi_simpledotnetapi]'
    reason: ContainersNotReady
    status: "False"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: 2019-06-17T20:22:30Z
    status: "True"
    type: PodScheduled
  containerStatuses:
  - image: ussmicroserviceregistry.azurecr.io/simpledotnetapi_simpledotnetapi
    imageID: ""
    lastState: {}
    name: simpledotnetapi-simpledotnetapi
    ready: false
    restartCount: 0
    state:
      waiting:
        message: Back-off pulling image "ussmicroserviceregistry.azurecr.io/simpledotnetapi-simpledotnetapi"
        reason: ImagePullBackOff
  hostIP: 10.240.0.4
  phase: Pending
  podIP: 10.240.0.26
  qosClass: BestEffort
  startTime: 2019-06-17T20:22:30Z
</code></pre>

<p>-- yaml configuration file</p>

<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
    name: simpledotnetapi-deployment
spec:
    replicas: 3
    selector:
        matchLabels:
            app: simpledotnetapi-pod
    template:
        metadata:
            labels:
                app: simpledotnetapi-pod
        spec:
            imagePullSecrets:
              - name: kimagereadersecret
            containers:
              - name: simpledotnetapi_simpledotnetapi
                image: ussmicroserviceregistry.azurecr.io/simpledotnetapi-simpledotnetapi
                ports:
                  - containerPort: 5000
---
apiVersion: v1
kind: Service
metadata:
    name: simpledotnetapi-service
spec:
    type: LoadBalancer
    ports:
    - port: 80
    selector:
       app: simpledotnetapi
       type: front-end
</code></pre>

<p>-- output of kubectl get secret kimagereadersecret</p>

<pre><code>NAME                 TYPE                             DATA      AGE
kimagereadersecret   kubernetes.io/dockerconfigjson   1         1h
</code></pre>

<p>-- credentials/secret from Kube dashboard</p>

<pre><code>{
  "kind": "Secret",
  "apiVersion": "v1",
  "metadata": {
    "name": "kimagereadersecret",
    "namespace": "default",
    "selfLink": "/api/v1/namespaces/default/secrets/kimagereadersecret",
    "uid": "86006aff-9156-11e9-801a-c6aef815c06a",
    "resourceVersion": "830006",
    "creationTimestamp": "2019-06-17T23:20:41Z"
  },
  "data": {
    ".dockerconfigjson": "eyJhdXRocyI6eyJ1c3NtaWNyb3NlcnZpY2VyZWdpc3RyeS5henVyZWNyLmlvIjp7InVzZXJuYW1lIjoiMzNjYjBjZTQtOTVmMC00NGJkLWJiYmYtNTZkNTA2ZmY0ZWIzIiwicGFzc3dvcmQiOiI0YjM3Yjg5Ni1hMDRlLTQ4YjQtYTk1MC01ZjFhYmRkM2U3YWEiLCJlbWFpbCI6ImNvcmV5cEB1bml0ZWQtc3lzdGVtcy5jb20iLCJhdXRoIjoiTXpOallqQmpaVFF0T1RWbU1DMDBOR0prTFdKaVltWXROVFprTlRBMlptWTBaV0l6T2pSaU16ZGlPRGsyTFdFd05HVXRORGhpTkMxaE9UVXdMVFZtTVdGaVpHUXpaVGRoWVE9PSJ9fX0="
  },
  "type": "kubernetes.io/dockerconfigjson"
}
</code></pre>

<p>-- Full dump from the Kube Dashboard</p>

<pre><code>Failed to pull image "ussmicroserviceregistry.azurecr.io/simpledotnetapi_simpledotnetapi": [rpc error: code = Unknown desc = Error response from daemon: manifest for ussmicroserviceregistry.azurecr.io/simpledotnetapi_simpledotnetapi:latest not found: manifest unknown: manifest unknown, rpc error: code = Unknown desc = Error response from daemon: Get https://ussmicroserviceregistry.azurecr.io/v2/simpledotnetapi_simpledotnetapi/manifests/latest: unauthorized: authentication required, rpc error: code = Unknown desc = Error response from daemon: Get https://ussmicroserviceregistry.azurecr.io/v2/simpledotnetapi_simpledotnetapi/manifests/latest: unauthorized: authentication required]
</code></pre>

<p>The entire project is in GitHub @ <a href="https://github.com/coreyperkins/KubeSimpleDotNetApi" rel="nofollow noreferrer">https://github.com/coreyperkins/KubeSimpleDotNetApi</a> </p>

<p>-- ACR screenshot
<a href="https://i.stack.imgur.com/m2rFv.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/m2rFv.png" alt="enter image description here"></a></p>

<p>-- Pod Failure in Kube
<a href="https://i.stack.imgur.com/07Nvm.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/07Nvm.png" alt="enter image description here"></a></p>

<p>I'm fairly certain you didn't give it enough permissions:</p>

<pre><code>az ad sp create-for-rbac 
  --scopes /subscriptions/11870e73-bdb2-47b0-bf27-25d24c41ae24/resourcegroups/USS-MicroService-Test/providers/Microsoft.ContainerRegistry/registries/UssMicroServiceRegistry
  --role Reader 
  --name kimage-reader
</code></pre>

<p>role should be <code>acrpull</code>, not reader. and just delete this secret: `kimagereadersecret 1 and reference to it in the pod. kubernetes will handle that for you.</p>
