<p>I'm new AKS, ACR, and DevOps Pipelines and I'm trying to setup a CI/CD pipeline.</p>

<p>I have a resource group setup that has both AKS and ACR in it. AKS is using <code>Standard_B2s</code> and only one node at this point since I'm just playing around.</p>

<p>Images are being deployed to ACR automatically on a commit to master--haven't figured out how to setup testing yet--but when it comes to deploying to AKS, I just keep getting a:</p>

<pre><code>##[error]error: deployment "client-deployment" exceeded its progress deadline
</code></pre>

<p>I've changed my <code>client.yaml</code> to include a <code>progressDeadlineSeconds</code> of like an hour as 10, 15, and 20 minutes didn't work:</p>

<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: client-deployment
spec:
  progressDeadlineSeconds: 3600
  replicas: 1
  selector:
    matchLabels:
      component: client
  template:
    metadata:
      labels:
        component: client
    spec:
      containers:
        - name: client
          image: testappcontainers.azurecr.io/testapp-client
          ports:
            - containerPort: 3000
---
apiVersion: v1
kind: Service
metadata:
  name: client-cluster-ip-service
spec:
  type: ClusterIP
  selector:
    component: client
  ports:
    - port: 3000
      targetPort: 3000
</code></pre>

<p>I've just been modifying the <code>azure-pipelines.yml</code> that Pipelines generated for me, which I currently have as the following:</p>

<pre><code># Docker
# Build and push an image to Azure Container Registry
# https://learn.microsoft.com/azure/devops/pipelines/languages/docker

trigger:
- master

resources:
- repo: self

variables:
  # Container registry service connection established during pipeline creation
  dockerRegistryServiceConnection: &lt;dockerRegistryServiceConnection_key&gt;
  imageRepository: 'testapp'
  containerRegistry: 'testappcontainers.azurecr.io'
  dockerfilePath: '$(Build.SourcesDirectory)'
  tag: '$(Build.BuildId)'
  imagePullSecret: &lt;imagePullSecret_key&gt;

  # Agent VM image name
  vmImageName: 'ubuntu-latest'

stages:
- stage: Build
  displayName: Build and push stage
  jobs:  
  - job: Build
    displayName: Build
    pool:
      vmImage: $(vmImageName)
    steps:
    - task: Docker@2
      displayName: Build and push client image to container registry
      inputs:
        command: buildAndPush
        repository: $(imageRepository)-client
        dockerfile: $(dockerfilePath)/client/Dockerfile
        containerRegistry: $(dockerRegistryServiceConnection)
        tags: |
          $(tag)

    - upload: manifests
      artifact: manifests

- stage: Deploy
  displayName: Deploy stage
  dependsOn: Build

  jobs:
  - deployment: Deploy
    displayName: Deploy job
    pool:
      vmImage: $(vmImageName)
    environment: 'testapp.default'
    strategy:
      runOnce:
        deploy:
          steps:
          - task: KubernetesManifest@0
            displayName: Create imagePullSecret
            inputs:
              action: createSecret
              secretName: $(imagePullSecret)
              dockerRegistryEndpoint: $(dockerRegistryServiceConnection)

          - task: KubernetesManifest@0
            displayName: Deploy to Kubernetes cluster
            inputs:
              action: deploy
              manifests: |
                $(Pipeline.Workspace)/manifests/client.yaml
              imagePullSecrets: |
                $(imagePullSecret)
              containers: |
                $(containerRegistry)/$(imageRepository):$(tag)
</code></pre>

<p>Here is the log too for the Task that fails:</p>

<pre><code>##[debug]Evaluating condition for step: 'Deploy to Kubernetes cluster'
##[debug]Evaluating: SucceededNode()
##[debug]Evaluating SucceededNode:
##[debug]=&gt; True
##[debug]Result: True
##[section]Starting: Deploy to Kubernetes cluster
==============================================================================
Task         : Deploy to Kubernetes
Description  : Use Kubernetes manifest files to deploy to clusters or even bake the manifest files to be used for deployments using Helm charts
Version      : 0.162.1
Author       : Microsoft Corporation
Help         : https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/deploy/kubernetes-manifest
==============================================================================
##[debug]agent.TempDirectory=/home/vsts/work/_temp
##[debug]loading inputs and endpoints
##[debug]loading INPUT_ACTION
##[debug]loading INPUT_KUBERNETESSERVICECONNECTION
##[debug]loading INPUT_STRATEGY
##[debug]loading INPUT_TRAFFICSPLITMETHOD
##[debug]loading INPUT_PERCENTAGE
##[debug]loading INPUT_BASELINEANDCANARYREPLICAS
##[debug]loading INPUT_MANIFESTS
##[debug]loading INPUT_CONTAINERS
##[debug]loading INPUT_IMAGEPULLSECRETS
##[debug]loading INPUT_RENDERTYPE
##[debug]loading INPUT_DOCKERCOMPOSEFILE
##[debug]loading INPUT_HELMCHART
##[debug]loading INPUT_KUSTOMIZATIONPATH
##[debug]loading INPUT_RESOURCETOPATCH
##[debug]loading INPUT_RESOURCEFILETOPATCH
##[debug]loading INPUT_MERGESTRATEGY
##[debug]loading INPUT_SECRETTYPE
##[debug]loading ENDPOINT_AUTH_&lt;token&gt;
##[debug]loading ENDPOINT_AUTH_SCHEME_&lt;token&gt;
##[debug]loading ENDPOINT_AUTH_PARAMETER_&lt;token&gt;_AZUREENVIRONMENT
##[debug]loading ENDPOINT_AUTH_PARAMETER_&lt;token&gt;_AZURETENANTID
##[debug]loading ENDPOINT_AUTH_PARAMETER_&lt;token&gt;_SERVICEACCOUNTNAME
##[debug]loading ENDPOINT_AUTH_PARAMETER_&lt;token&gt;_ROLEBINDINGNAME
##[debug]loading ENDPOINT_AUTH_PARAMETER_&lt;token&gt;_SECRETNAME
##[debug]loading ENDPOINT_AUTH_PARAMETER_&lt;token&gt;_APITOKEN
##[debug]loading ENDPOINT_AUTH_PARAMETER_&lt;token&gt;_SERVICEACCOUNTCERTIFICATE
##[debug]loading ENDPOINT_AUTH_SYSTEMVSSCONNECTION
##[debug]loading ENDPOINT_AUTH_SCHEME_SYSTEMVSSCONNECTION
##[debug]loading ENDPOINT_AUTH_PARAMETER_SYSTEMVSSCONNECTION_ACCESSTOKEN
##[debug]loading SECRET_CONTAINER_PASSWORD
##[debug]loading SECRET_CONTAINER_USERNAME
##[debug]loading SECRET_SYSTEM_ACCESSTOKEN
##[debug]loaded 32
##[debug]Agent.ProxyUrl=undefined
##[debug]Agent.CAInfo=undefined
##[debug]Agent.ClientCert=undefined
##[debug]Agent.SkipCertValidation=undefined
##[debug]SYSTEM_HOSTTYPE=build
##[debug]System.TeamFoundationCollectionUri=https://dev.azure.com/thetestcompany/
##[debug]Build.BuildNumber=20191231.5
##[debug]Build.DefinitionName=test-app
##[debug]System.DefinitionId=4
##[debug]Agent.JobName=Deploy job
##[debug]System.TeamProject=test-app
##[debug]Build.BuildId=41
##[debug]System.TeamProject=test-app
##[debug]namespace=null
##[debug]containers=***/testapp:41
##[debug]imagePullSecrets=testappcontainers&lt;key&gt;-auth
##[debug]manifests=/home/vsts/work/1/manifests/client.yaml
##[debug]percentage=0
##[debug]strategy=none
##[debug]trafficSplitMethod=pod
##[debug]baselineAndCanaryReplicas=0
##[debug]arguments=null
##[debug]secretArguments=null
##[debug]secretType=dockerRegistry
##[debug]secretName=null
##[debug]dockerRegistryEndpoint=null
##[debug]kubernetesServiceConnection=&lt;token&gt;
##[debug]&lt;token&gt; data namespace = default
##[debug]System.TeamFoundationCollectionUri=https://dev.azure.com/thetestcompany/
##[debug]System.HostType=build
##[debug]System.DefaultWorkingDirectory=/home/vsts/work/1/s
##[debug]Build.SourceBranchName=master
##[debug]Build.Repository.Provider=TfsGit
##[debug]Build.Repository.Uri=https://thetestcompany@dev.azure.com/thetestcompany/test-app/_git/test-app
##[debug]agent.proxyurl=undefined
##[debug]VSTS_ARM_REST_IGNORE_SSL_ERRORS=undefined
##[debug]AZURE_HTTP_USER_AGENT=VSTS_&lt;hash&gt;_build_4_0
##[debug]Agent.ProxyUrl=undefined
##[debug]Agent.CAInfo=undefined
##[debug]Agent.ClientCert=undefined
##[debug]check path : /home/vsts/work/_tasks/KubernetesManifest_&lt;hash&gt;/0.162.1/node_modules/azure-pipelines-tool-lib/lib.json
##[debug]adding resource file: /home/vsts/work/_tasks/KubernetesManifest_&lt;hash&gt;/0.162.1/node_modules/azure-pipelines-tool-lib/lib.json
##[debug]system.culture=en-US
##[debug]check path : /home/vsts/work/_tasks/KubernetesManifest_&lt;hash&gt;/0.162.1/task.json
##[debug]adding resource file: /home/vsts/work/_tasks/KubernetesManifest_&lt;hash&gt;/0.162.1/task.json
##[debug]system.culture=en-US
##[debug]action=deploy
##[debug]kubernetesServiceConnection=&lt;token&gt;
##[debug]agent.tempDirectory=/home/vsts/work/_temp
##[debug]&lt;token&gt; data authorizationType = AzureSubscription
##[debug]&lt;token&gt;=https://testappk8s-dns-&lt;key&gt;.hcp.westus.azmk8s.io/
##[debug]&lt;token&gt; auth param serviceAccountCertificate = ***
##[debug]&lt;token&gt; auth param apiToken = ***
##[debug]set KUBECONFIG=/home/vsts/work/_temp/kubectlTask/1577816701759/config
##[debug]Processed: ##vso[task.setvariable variable=KUBECONFIG;issecret=false;]/home/vsts/work/_temp/kubectlTask/1577816701759/config
##[debug]&lt;token&gt; data acceptUntrustedCerts = undefined
##[debug]which 'kubectl'
##[debug]found: '/usr/bin/kubectl'
##[debug]which 'kubectl'
##[debug]found: '/usr/bin/kubectl'
##[debug]System.DefaultWorkingDirectory=/home/vsts/work/1/s
##[debug]defaultRoot: '/home/vsts/work/1/s'
##[debug]findOptions.allowBrokenSymbolicLinks: 'false'
##[debug]findOptions.followSpecifiedSymbolicLink: 'true'
##[debug]findOptions.followSymbolicLinks: 'true'
##[debug]matchOptions.debug: 'false'
##[debug]matchOptions.nobrace: 'true'
##[debug]matchOptions.noglobstar: 'false'
##[debug]matchOptions.dot: 'true'
##[debug]matchOptions.noext: 'false'
##[debug]matchOptions.nocase: 'false'
##[debug]matchOptions.nonull: 'false'
##[debug]matchOptions.matchBase: 'false'
##[debug]matchOptions.nocomment: 'false'
##[debug]matchOptions.nonegate: 'false'
##[debug]matchOptions.flipNegate: 'false'
##[debug]pattern: '/home/vsts/work/1/manifests/client.yaml'
##[debug]findPath: '/home/vsts/work/1/manifests/client.yaml'
##[debug]statOnly: 'true'
##[debug]found 1 paths
##[debug]applying include pattern
##[debug]1 matches
##[debug]1 final results
##[debug]agent.tempDirectory=/home/vsts/work/_temp
##[debug]New K8s objects after addin imagePullSecrets are :[{"apiVersion":"apps/v1","kind":"Deployment","metadata":{"name":"client-deployment"},"spec":{"progressDeadlineSeconds":3600,"replicas":1,"selector":{"matchLabels":{"component":"client"}},"template":{"metadata":{"labels":{"component":"client"}},"spec":{"containers":[{"name":"client","image":"***/testapp-client","ports":[{"containerPort":3000}]}],"imagePullSecrets":[{"name":"testappcontainers1741032e-auth"}]}}}},{"apiVersion":"v1","kind":"Service","metadata":{"name":"client-cluster-ip-service"},"spec":{"type":"ClusterIP","selector":{"component":"client"},"ports":[{"port":3000,"targetPort":3000}]}}]
##[debug]agent.tempDirectory=/home/vsts/work/_temp
##[debug]agent.tempDirectory=/home/vsts/work/_temp
##[debug]which '/usr/bin/kubectl'
##[debug]found: '/usr/bin/kubectl'
##[debug]which '/usr/bin/kubectl'
##[debug]found: '/usr/bin/kubectl'
##[debug]/usr/bin/kubectl arg: apply
##[debug]/usr/bin/kubectl arg: ["-f","/home/vsts/work/_temp/Deployment_client-deployment_1577816701782,/home/vsts/work/_temp/Service_client-cluster-ip-service_1577816701782"]
##[debug]/usr/bin/kubectl arg: ["--namespace","default"]
##[debug]exec tool: /usr/bin/kubectl
##[debug]arguments:
##[debug]   apply
##[debug]   -f
##[debug]   /home/vsts/work/_temp/Deployment_client-deployment_1577816701782,/home/vsts/work/_temp/Service_client-cluster-ip-service_1577816701782
##[debug]   --namespace
##[debug]   default
[command]/usr/bin/kubectl apply -f /home/vsts/work/_temp/Deployment_client-deployment_1577816701782,/home/vsts/work/_temp/Service_client-cluster-ip-service_1577816701782 --namespace default
deployment.apps/client-deployment unchanged
service/client-cluster-ip-service unchanged
##[debug]which '/usr/bin/kubectl'
##[debug]found: '/usr/bin/kubectl'
##[debug]which '/usr/bin/kubectl'
##[debug]found: '/usr/bin/kubectl'
##[debug]/usr/bin/kubectl arg: ["rollout","status"]
##[debug]/usr/bin/kubectl arg: Deployment/client-deployment
##[debug]/usr/bin/kubectl arg: ["--namespace","default"]
##[debug]exec tool: /usr/bin/kubectl
##[debug]arguments:
##[debug]   rollout
##[debug]   status
##[debug]   Deployment/client-deployment
##[debug]   --namespace
##[debug]   default
[command]/usr/bin/kubectl rollout status Deployment/client-deployment --namespace default
error: deployment "client-deployment" exceeded its progress deadline
##[debug]which '/usr/bin/kubectl'
##[debug]found: '/usr/bin/kubectl'
##[debug]which '/usr/bin/kubectl'
##[debug]found: '/usr/bin/kubectl'
##[debug]/usr/bin/kubectl arg: get
##[debug]/usr/bin/kubectl arg: service/client-cluster-ip-service
##[debug]/usr/bin/kubectl arg: ["-o","json"]
##[debug]/usr/bin/kubectl arg: ["--namespace","default"]
##[debug]exec tool: /usr/bin/kubectl
##[debug]arguments:
##[debug]   get
##[debug]   service/client-cluster-ip-service
##[debug]   -o
##[debug]   json
##[debug]   --namespace
##[debug]   default
[command]/usr/bin/kubectl get service/client-cluster-ip-service -o json --namespace default
{
    "apiVersion": "v1",
    "kind": "Service",
    "metadata": {
        "annotations": {
            "kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"v1\",\"kind\":\"Service\",\"metadata\":{\"annotations\":{},\"name\":\"client-cluster-ip-service\",\"namespace\":\"default\"},\"spec\":{\"ports\":[{\"port\":3000,\"targetPort\":3000}],\"selector\":{\"component\":\"client\"},\"type\":\"ClusterIP\"}}\n"
        },
        "creationTimestamp":         "name": "client-cluster-ip-service",
        "namespace": "default",
        "resourceVersion": "1234045",
        "selfLink": "/api/v1/namespaces/default/services/client-cluster-ip-service",
        "uid": "5f077159-2bdd-11ea-af20-3eaa105eb2b3"
    },
    "spec": {
        "clusterIP": "10.0.181.220",
        "ports": [
            {
                "port": 3000,
                "protocol": "TCP",
                "targetPort": 3000
            }
        ],
        "selector": {
            "component": "client"
        },
        "sessionAffinity": "None",
        "type": "ClusterIP"
    },
    "status": {
        "loadBalancer": {}
    }
}
##[debug]KUBECONFIG=/home/vsts/work/_temp/kubectlTask/1577816701759/config
##[debug]set KUBECONFIG=
##[debug]Processed: ##vso[task.setvariable variable=KUBECONFIG;issecret=false;]
##[debug]task result: Failed
##[error]error: deployment "client-deployment" exceeded its progress deadline
##[debug]Processed: ##vso[task.issue type=error;]error: deployment "client-deployment" exceeded its progress deadline
##[debug]Processed: ##vso[task.complete result=Failed;]error: deployment "client-deployment" exceeded its progress deadline
##[section]Finishing: Deploy to Kubernetes cluster
</code></pre>

<p>Then in Azure CLI, it shows the deployment is there, but with no available pods:</p>

<pre><code>eox-dev@Azure:~$ kubectl get deployments
NAME                READY   UP-TO-DATE   AVAILABLE   AGE
client-deployment   0/1     1            0           3h47m
eox-dev@Azure:~$ kubectl describe deployment client-deployment
Name:                   client-deployment
Namespace:              default
CreationTimestamp:      Tue, 31 Dec 2019 15:50:30 +0000
Labels:                 &lt;none&gt;
Annotations:            deployment.kubernetes.io/revision: 1
                        kubectl.kubernetes.io/last-applied-configuration:
                          {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"name":"client-deployment","namespace":"default"},"spec":{"progre...
Selector:               component=client
Replicas:               1 desired | 1 updated | 1 total | 0 available | 1 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:  component=client
  Containers:
   client:
    Image:        testappcontainers.azurecr.io/testapp-client
    Port:         3000/TCP
    Host Port:    0/TCP
    Environment:  &lt;none&gt;
    Mounts:       &lt;none&gt;
  Volumes:        &lt;none&gt;
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      False   MinimumReplicasUnavailable
  Progressing    False   ProgressDeadlineExceeded
OldReplicaSets:  &lt;none&gt;
NewReplicaSet:   client-deployment-5688bdc69c (1/1 replicas created)
Events:          &lt;none&gt;
</code></pre>

<p>So what am I doing wrong here?</p>

cjones
<blockquote>
  <p>Error from server (BadRequest): container "client" in pod
  "client-deployment-5688bdc69c-hxlcf" is waiting to start: trying and
  failing to pull image</p>
</blockquote>

<p>Based on my experience, this is more relative with <code>imagePullSecrets</code> and the <code>Kubernetes namespace</code>. </p>

<p>In your <code>Create imagePullSecret</code> and <code>Deploy to Kubernetes cluster</code> task, I saw that you did not provide the value to task parameter: <code>namespace</code>. This will lead to a new namespace which name is <code>default</code>  will be created, since you <strong>unspecified</strong> the namespace.</p>

<p>And, the <a href="https://learn.microsoft.com/en-us/azure/aks/concepts-security#kubernetes-secrets" rel="noreferrer">kubernetes secret</a> which generated by <code>createSecret</code> action is seperated for each namespace. In one word, different namespace has different secret value:</p>

<blockquote>
  <p>Secrets are stored within a given namespace and can only be accessed
  by pods within the same namespace.</p>
</blockquote>

<hr>

<p>Now, let’s back to your build compile process.</p>

<p>In your yml definition, <code>Create imagePullSecret</code> will create a <code>secret</code> for new namespace <code>default</code> which created by task automatically as you did not provide the given namespace value. </p>

<p>Then, in next task <code>Deploy to Kubernetes cluster</code>, because of the same reason, here the task will re-created a another new namespace <code>default</code>(<strong>Note:</strong> this is not same with the previous one). Also, you could see this progress from the log:</p>

<p><a href="https://i.stack.imgur.com/oXgHC.png" rel="noreferrer"><img src="https://i.stack.imgur.com/oXgHC.png" alt="enter image description here"></a></p>

<p>At this time, the <code>secret</code> that generated from the previous task will not available for the current namespace. BUT, as you know, the <code>ACR</code> is a private container registry which our system must verify whether the <code>kubernetes secret</code> is available.</p>

<p>In addition, in your Deploy to Kubernetes cluster task, you were specifying the repository as <code>$(imageRepository)</code> which does not same with the repository you push the image to <strong><code>$(imageRepository)-client</code></strong>. </p>

<p>This can also be checked in your log:</p>

<p><a href="https://i.stack.imgur.com/hbOB4.png" rel="noreferrer"><img src="https://i.stack.imgur.com/hbOB4.png" alt="enter image description here"></a></p>

<p>That's why there's no available node in your kubernetes, and you failed to pull the image also.</p>

<hr>

<p>To avoid the issue, please ensure you provide the <code>namespace</code> value in <code>KubernetesManifest@0</code> task.</p>

<pre><code>      - task: KubernetesManifest@0
        displayName: Create imagePullSecret
        inputs:
          action: createSecret
          secretName: $(imagePullSecret)
          namespace: $(k8sNamespace)
          dockerRegistryEndpoint: $(DRServiceConnection)

      - task: KubernetesManifest@0
        displayName: Deploy to Kubernetes cluster
        inputs:
          action: deploy
          namespace: $(k8sNamespace)
          manifests: |
            $(System.ArtifactsDirectory)/manifests/deployment.yml
          imagePullSecrets: |
            $(imagePullSecret)
          containers: |
            $(containerRegistry)/$(imageRepository)-client:$(tag)
</code></pre>

<p>secret to imagePullSecrets of each namespace</p>
