<p>First, I've created a GKE Autopilot cluster using the GCP GUI Console in my browser, with default settings so I tried applying my deployments with <code>kubectl apply -f thisfile.yaml</code>:</p>
<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  replicas: 3
  selector:
    matchLabels:
      run: my-app
  template:
    metadata:
      labels:
        run: my-app
    spec:
      containers:
      - name: hello-app
        image: us-docker.pkg.dev/google-samples/containers/gke/hello-app:1.0
</code></pre>
<p>After, I've been rewriting this into a Terraform file, resulting in this:</p>
<pre><code>resource &quot;google_container_cluster&quot; &quot;my_gke&quot; {
  name             = &quot;my-gke&quot;
  enable_autopilot = &quot;true&quot;
  location         = &quot;southamerica-east1&quot;
}

data &quot;google_client_config&quot; &quot;default&quot; {}

provider &quot;kubernetes&quot; {
  host                   = &quot;https://${google_container_cluster.my_gke.endpoint}&quot;
  token                  = data.google_client_config.default.access_token
  cluster_ca_certificate = base64decode(google_container_cluster.my_gke.master_auth[0].cluster_ca_certificate)
}

resource &quot;kubernetes_deployment&quot; &quot;my_deployment&quot; {
  metadata {
    name = &quot;my-app&quot;
  }
  spec {
    replicas = 2
    selector {
      match_labels = {
        run = &quot;my-app&quot;
      }
    }
    template {
      metadata {
        labels = {
          run = &quot;my-app&quot;
        }
      }
      spec {
        container {
          image = &quot;us-docker.pkg.dev/google-samples/containers/gke/hello-app:1.0&quot;
          name  = &quot;hello-app&quot;
        }
      }
    }
  }


  lifecycle {
    ignore_changes = [
      metadata[0].annotations,
      metadata[0].resource_version,
      spec[0].template[0].spec[0].container[0].security_context
    ]
  }
}
</code></pre>
<p>The problem is as follows:</p>
<ul>
<li>When I apply it with <code>kubectl apply -f thisfile.yaml</code>, using the YAML notation, everything deploys fine.</li>
<li>When I remove the cluster and apply everything with Terraform, the first revision applies fine after some time, but the next revisions of the deployment keeps resulting on the GCP Console printing &quot;Unschedulable&quot; errors claiming &quot;insufficient cpu&quot; and/or &quot;insufficient memory&quot;</li>
</ul>
<p>PS. I've already tried to set resources' limits and requests inside of the PodSpec before, but nothing changed.</p>
<p>I am new into GKE and everything looks so unreliable for me now. What am I doing wrong?</p>

deniable_encryption
<p>I can confirm this is an ongoing issue. We have a support ticket opened with GCP to investigate the same behaviour.</p>
<p>The first deployment works fine but the deployment of a new revision fails. Pods gets created and deleted quickly (a few per second) and the ReplicaSets get to the hundreds pretty fast.</p>
<p>From the logs we can see the same Unschedulable errors, sometimes we see pods failing to start due to missing Volumes in the nodes (our own volumes most of the time, but also &quot;kube-api-access&quot; ones).</p>
<p>There is also errors like this in the logs:</p>
<p><code>message: &quot;Operation cannot be fulfilled on replicasets.apps &quot;deploytest-747c54b87d&quot;: the object has been modified; please apply your changes to the latest version and try again&quot;</code></p>
<p>It only happens if we try to redeploy via terraform. If we patch the yaml directly with kubectl the new version goes fine through the replacement strategy, no errors.</p>
<p>The exact same terraform code works on GKE regular cluster. Problem seems to be limited to the Autopilot ones. Already tested in a brand new project, everything out of the box default.</p>
<p>Google is currently investigating the issue on an internal ticket. I'll try to report back here what else we can figure out.</p>
<p>-- edit:</p>
<p>It seems the pods need the pod.securityContext.seccompProfile parameter, which is not yet supported by Terraform. For some reason the same terraform code can instantiate a new deployment if it doesn't exist yet. But it cannot update the deployment.</p>
<p>The patch bellow adds the missing config, but it's not a solution. I guess we need to wait for that securityContext to be added to TF. (ps. there is a deprecated annotation method to add this config, but I didn't try it).</p>
<pre><code>kubectl patch deployment deploy-test -p='[{&quot;op&quot;: &quot;replace&quot;, &quot;path&quot;: &quot;/spec/template/spec/securityContext&quot;, &quot;value&quot;:{&quot;seccompProfile&quot;:{&quot;type&quot;:&quot;RuntimeDefault&quot;}}}]' --type='json'
</code></pre>
<p>We just decided to move away from TF for the K8S deployments. Our TF only goes up to the cluster creation. Workloads will be managed somewhere else.</p>
