<p>Trying to create pods in a cluster in GKE. There is a docker container containing some python code with a sidecar container to access the sql database. The deployment worked perfectly previously, however after a few weeks I tried to redeploy with kubectl apply -f file_name.yaml.</p>
<p>The pods got temporarily created with a 'Pending' status and disappeared after 15 seconds. Happens every time. I am unable to access logs. kubectl get pods returns nothing after 15 seconds as well.
Not sure where to go from here... Any help would be appreciated!</p>
<pre><code>apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubernetes.io/limit-ranger: 'LimitRanger plugin set: cpu request for container
      pyxy-web-v1'
  creationTimestamp: &quot;2020-05-14T00:38:09Z&quot;
  labels:
    run: pyxy-web-v1
  name: pyxy-web-v1
  namespace: default
  resourceVersion: &quot;1215073&quot;
  selfLink: /api/v1/namespaces/default/pods/pyxy-web-v1
  uid: *omitted
spec:
  containers:
  - image: gcr.io/my-project-{*omitted}/pyxy-web:latest
    imagePullPolicy: Always
    name: pyxy-web-v1
    ports:
    - containerPort: 8080
      protocol: TCP
    env:
    - name: DB_USER
      valueFrom:
        secretKeyRef:
          name: cloudsql-db-credentials
          key: *omitted
    - name: DB_PASS
      valueFrom:
        secretKeyRef:
          name: cloudsql-db-credentials
          key: *omitted
    resources:
      requests:
        cpu: 100m
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: default-token-94bct
      readOnly: true

  - name: cloudsql-proxy
    image: gcr.io/cloudsql-docker/gce-proxy:1.16
    command: [&quot;/cloud_sql_proxy&quot;,
              &quot;-instances=my-project-{*omitted}:us-central1:routing-app-v1=tcp:3306&quot;,
              # If running on a VPC, the Cloud SQL proxy can connect via Private IP. See:
              # https://cloud.google.com/sql/docs/mysql/private-ip for more info.
              # &quot;-ip_address_types=PRIVATE&quot;,
              &quot;-credential_file=/secrets/cloudsql/credentials.json&quot;]
    # [START cloudsql_security_context]
    securityContext:
      runAsUser: 2  # non-root user
      allowPrivilegeEscalation: false
    volumeMounts:
    - name: cloudsql-instance-credentials
      mountPath: /secrets/cloudsql
      readOnly: true

  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  nodeName: gke-pyxy-cluster-default-pool-{*omitted}
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: default
  serviceAccountName: default
  terminationGracePeriodSeconds: 180
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - name: default-token-94bct
    secret:
      defaultMode: 420
      secretName: default-token-94bct
  - name: cloudsql-instance-credentials
    secret:
      secretName: cloudsql-instance-credentials
</code></pre>
<p>During the 15 seconds-long pending period the kubectl describe pods returns the following.</p>
<pre><code>Name:         pyxy-web-v1
Namespace:    default
Priority:     0
Node:         gke-pyxy-cluster-default-pool-{*omitted}/
Labels:       run=pyxy-web-v1
Annotations:  kubernetes.io/limit-ranger: LimitRanger plugin set: cpu request for container cloudsql-proxy
Status:       Pending
IP:           
IPs:          &lt;none&gt;
Containers:
  pyxy-web-v1:
    Image:      gcr.io/my-project-{*omitted}/pyxy-web:latest
    Port:       8080/TCP
    Host Port:  0/TCP
    Requests:
      cpu:  100m
    Environment:
      DB_USER:  &lt;set to the key '*omitted' in secret 'cloudsql-db-credentials'&gt;  Optional: false
      DB_PASS:  &lt;set to the key '*omitted' in secret 'cloudsql-db-credentials'&gt;  Optional: false
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-94bct (ro)
  cloudsql-proxy:
    Image:      gcr.io/cloudsql-docker/gce-proxy:1.16
    Port:       &lt;none&gt;
    Host Port:  &lt;none&gt;
    Command:
      /cloud_sql_proxy
      -instances=my-project-{*omitted}:us-central1:routing-app-v1=tcp:3306
      -credential_file=/secrets/cloudsql/credentials.json
    Requests:
      cpu:        100m
    Environment:  &lt;none&gt;
    Mounts:
      /secrets/cloudsql from cloudsql-instance-credentials (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-94bct (ro)
Volumes:
  default-token-94bct:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-94bct
    Optional:    false
  cloudsql-instance-credentials:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  cloudsql-instance-credentials
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  &lt;none&gt;
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:          &lt;none&gt;
</code></pre>
<p>However after this time, it returns
'No resources found in default namespace.'</p>

<h2>Answer</h2>
<p>The Pod spec had a Node Name for a Node that was no longer in the cluster (due to an upgrade). That is to say the <code>pod.spec.nodeName</code> was erroneous.</p>
<p>From <code>kubectl explain pod.spec</code>:</p>
<pre><code>   nodeName &lt;string&gt;
     NodeName is a request to schedule this pod onto a specific node. If it is
     non-empty, the scheduler simply schedules this pod onto that node, assuming
     that it fits resource requirements.
</code></pre>
<p>During the ~15 second window the Pod was in Pending state, the following error log pointed to the solution:</p>
<pre><code>Error from server (NotFound): pods &quot;gke-pyxy-cluster-default-pool-94aa0302-pm35&quot; not found
</code></pre>
