<p>I am a beginner in Kubernetes and have been using the kubectl command to create pods for several months. However, I recently encountered a problem where Kubernetes did not create a pod after I executed the <code>kubectl create -f mypod.yaml</code> command. When I run kubectl get pods, the mypod does not appear in the list of pods and I am unable to access it by name as if it does not exist. However, if I try to create it again, I receive a message saying that the pod has already been created.</p>
<p>To illustrate my point, let me give you an example. I frequently generate pods using a YAML file called tpcds-25-query.yaml. The contents of this file are as follows:</p>
<pre><code>apiVersion: &quot;sparkoperator.k8s.io/v1beta2&quot;
kind: SparkApplication
metadata:
name: tpcds-25-query
namespace: default
spec:
type: Scala
mode: cluster
image: registry.cn-beijing.aliyuncs.com/kube-ai/ack-spark-benchmark:1.0.1
imagePullPolicy: Always
sparkVersion: 2.4.5
mainClass: com.aliyun.spark.benchmark.tpcds.BenchmarkSQL
mainApplicationFile: &quot;local:///opt/spark/jars/ack-spark-benchmark-assembly-0.1.jar&quot;
arguments:
    # TPC-DS data localtion
    - &quot;oss://spark/data/tpc-ds-data/150g&quot;
    # results location
    - &quot;oss://spark/result/tpcds-25-query&quot;
    # Path to kit in the docker image
    - &quot;/tmp/tpcds-kit/tools&quot;
    # Data Format
    - &quot;parquet&quot;
    # Scale factor (in GB)
    - &quot;150&quot;
    # Number of iterations
    - &quot;1&quot;
    # Optimize queries
    - &quot;false&quot;
    # Filter queries, will run all if empty - &quot;q70-v2.4,q82-v2.4,q64-v2.4&quot;
    - &quot;q1-v2.4,q11-v2.4,q14a-v2.4,q14b-v2.4,q16-v2.4,q17-v2.4,q22-v2.4,q23a-v2.4,q23b-v2.4,q24a-v2.4,q24b-v2.4,q25-v2.4,q28-v2.4,q29-v2.4,q4-v2.4,q49-v2.4,q5-v2.4,q51-v2.4,q64-v2.4,q74-v2.4,q75-v2.4,q77-v2.4,q78-v2.4,q80-v2.4,q9-v2.4&quot;
    # Logging set to WARN
    - &quot;true&quot;
hostNetwork: true
dnsPolicy: ClusterFirstWithHostNet
restartPolicy:
    type: Never
timeToLiveSeconds: 86400
hadoopConf:
    # OSS 
    &quot;fs.oss.impl&quot;: &quot;OSSFileSystem&quot;
    &quot;fs.oss.endpoint&quot;: &quot;oss.com&quot;
    &quot;fs.oss.accessKeyId&quot;: &quot;DFDSMGDNDFMSNGDFMNGCU&quot;
    &quot;fs.oss.accessKeySecret&quot;: &quot;secret&quot;
sparkConf:
    &quot;spark.kubernetes.allocation.batch.size&quot;: &quot;200&quot;
    &quot;spark.sql.adaptive.join.enabled&quot;: &quot;true&quot;
    &quot;spark.eventLog.enabled&quot;: &quot;true&quot;
    &quot;spark.eventLog.dir&quot;: &quot;oss://spark/spark-events&quot;
driver:
    cores: 4
    memory: &quot;8192m&quot;
    labels:
    version: 2.4.5
    spark-app: spark-tpcds
    role: driver
    serviceAccount: spark
    nodeSelector:
    beta.kubernetes.io/instance-type: ecs.g6.13xlarge
executor:
    cores: 48
    instances: 1
    memory: &quot;160g&quot;
    memoryOverhead: &quot;16g&quot;
    labels:
    version: 2.4.5
    role: executor
    nodeSelector:
    beta.kubernetes.io/instance-type: ecs.g6.13xlarge
</code></pre>
<p>After I executed <code>kubectl create --validate=false -f tpcds-25-query.yaml</code> command, k8s returns this:</p>
<pre><code>sparkapplication.sparkoperator.k8s.io/tpcds-25-query created
</code></pre>
<p>which means the pod has been created. However, when I executed <code>kubectl get pods</code>, it gave me this:</p>
<pre><code>No resources found in default namespace.
</code></pre>
<p>When I created the pod again, it gave me this:</p>
<pre><code>Error from server (AlreadyExists): error when creating &quot;tpcds-25-query.yaml&quot;: sparkapplications.sparkoperator.k8s.io &quot;tpcds-25-query&quot; already exists
</code></pre>
<p>I know the option <code>-v=8</code> can print more detailed logs. So I execute the command <code>kubectl create --validate=false -f tpcds-25-query.yaml -v=8</code>, its output is:</p>
<pre><code>I0219 05:50:17.121661 2148722 loader.go:372] Config loaded from file:  /root/.kube/config
I0219 05:50:17.124735 2148722 round_trippers.go:432] GET https://172.16.0.212:6443/apis/metrics.k8s.io/v1beta1?timeout=32s
I0219 05:50:17.124747 2148722 round_trippers.go:438] Request Headers:
I0219 05:50:17.124753 2148722 round_trippers.go:442]     Accept: application/json, */*
I0219 05:50:17.124759 2148722 round_trippers.go:442]     User-Agent: kubectl/v1.22.3 (linux/amd64) kubernetes/9377577
I0219 05:50:17.132864 2148722 round_trippers.go:457] Response Status: 503 Service Unavailable in 8 milliseconds
I0219 05:50:17.132876 2148722 round_trippers.go:460] Response Headers:
I0219 05:50:17.132881 2148722 round_trippers.go:463]     X-Kubernetes-Pf-Prioritylevel-Uid: e75a0286-dd47-4533-a65c-79d95dac5bb1
I0219 05:50:17.132890 2148722 round_trippers.go:463]     Content-Length: 20
I0219 05:50:17.132894 2148722 round_trippers.go:463]     Date: Sun, 19 Feb 2023 05:50:17 GMT
I0219 05:50:17.132898 2148722 round_trippers.go:463]     Audit-Id: 3ab06f73-0c88-469a-834d-54ec06e910f1
I0219 05:50:17.132902 2148722 round_trippers.go:463]     Cache-Control: no-cache, private
I0219 05:50:17.132906 2148722 round_trippers.go:463]     Content-Type: text/plain; charset=utf-8
I0219 05:50:17.132909 2148722 round_trippers.go:463]     X-Content-Type-Options: nosniff
I0219 05:50:17.132913 2148722 round_trippers.go:463]     X-Kubernetes-Pf-Flowschema-Uid: 7f136704-82ad-4f6c-8c86-b470a972fede
I0219 05:50:17.134365 2148722 request.go:1181] Response Body: service unavailable
I0219 05:50:17.135255 2148722 request.go:1372] body was not decodable (unable to check for Status): couldn't get version/kind; json parse error: json: cannot unmarshal string into Go value of type struct { APIVersion string &quot;json:\&quot;apiVersion,omitempty\&quot;&quot;; Kind string &quot;json:\&quot;kind,omitempty\&quot;&quot; }
I0219 05:50:17.135265 2148722 cached_discovery.go:78] skipped caching discovery info due to the server is currently unable to handle the request
I0219 05:50:17.136050 2148722 request.go:1181] Request Body: {&quot;apiVersion&quot;:&quot;sparkoperator.k8s.io/v1beta2&quot;,&quot;kind&quot;:&quot;SparkApplication&quot;,&quot;metadata&quot;:{&quot;name&quot;:&quot;tpcds-25-query&quot;,&quot;namespace&quot;:&quot;default&quot;},&quot;spec&quot;:{&quot;arguments&quot;:[&quot;oss://lfpapertest/spark/data/tpc-ds-data/150g&quot;,&quot;oss://lfpapertest/spark/result/tpcds-runc-150g-48core-160g-1pod-25-query&quot;,&quot;/tmp/tpcds-kit/tools&quot;,&quot;parquet&quot;,&quot;150&quot;,&quot;1&quot;,&quot;false&quot;,&quot;q1-v2.4,q11-v2.4,q14a-v2.4,q14b-v2.4,q16-v2.4,q17-v2.4,q22-v2.4,q23a-v2.4,q23b-v2.4,q24a-v2.4,q24b-v2.4,q25-v2.4,q28-v2.4,q29-v2.4,q4-v2.4,q49-v2.4,q5-v2.4,q51-v2.4,q64-v2.4,q74-v2.4,q75-v2.4,q77-v2.4,q78-v2.4,q80-v2.4,q9-v2.4&quot;,&quot;true&quot;],&quot;dnsPolicy&quot;:&quot;ClusterFirstWithHostNet&quot;,&quot;driver&quot;:{&quot;cores&quot;:4,&quot;labels&quot;:{&quot;role&quot;:&quot;driver&quot;,&quot;spark-app&quot;:&quot;spark-tpcds&quot;,&quot;version&quot;:&quot;2.4.5&quot;},&quot;memory&quot;:&quot;8192m&quot;,&quot;nodeSelector&quot;:{&quot;beta.kubernetes.io/instance-type&quot;:&quot;ecs.g6.13xlarge&quot;},&quot;serviceAccount&quot;:&quot;spark&quot;},&quot;executor&quot;:{&quot;cores&quot;:48,&quot;instances&quot;:1,&quot;labels&quot;:{&quot;role&quot;:&quot;executor&quot;,&quot;version&quot;:&quot;2.4.5&quot;},&quot;memory&quot;:&quot;160g&quot;,&quot;memoryOverhead&quot;:&quot;16g&quot;,&quot;nodeSelector&quot;:{&quot;beta.kubernetes.io/instance-type&quot;:&quot;ecs.g6.13xlarge&quot;}},&quot;hadoopConf&quot;:{&quot;fs.oss.acce [truncated 802 chars]
I0219 05:50:17.136091 2148722 round_trippers.go:432] POST https://172.16.0.212:6443/apis/sparkoperator.k8s.io/v1beta2/namespaces/default/sparkapplications?fieldManager=kubectl-create
I0219 05:50:17.136098 2148722 round_trippers.go:438] Request Headers:
I0219 05:50:17.136104 2148722 round_trippers.go:442]     Accept: application/json
I0219 05:50:17.136108 2148722 round_trippers.go:442]     Content-Type: application/json
I0219 05:50:17.136113 2148722 round_trippers.go:442]     User-Agent: kubectl/v1.22.3 (linux/amd64) kubernetes/9377577
I0219 05:50:17.144313 2148722 round_trippers.go:457] Response Status: 201 Created in 8 milliseconds
I0219 05:50:17.144327 2148722 round_trippers.go:460] Response Headers:
I0219 05:50:17.144332 2148722 round_trippers.go:463]     X-Kubernetes-Pf-Prioritylevel-Uid: e75a0286-dd47-4533-a65c-79d95dac5bb1
I0219 05:50:17.144337 2148722 round_trippers.go:463]     Content-Length: 2989
I0219 05:50:17.144341 2148722 round_trippers.go:463]     Date: Sun, 19 Feb 2023 05:50:17 GMT
I0219 05:50:17.144345 2148722 round_trippers.go:463]     Audit-Id: 8eef9d08-04c0-44f7-87bf-e820853cd9c6
I0219 05:50:17.144349 2148722 round_trippers.go:463]     Cache-Control: no-cache, private
I0219 05:50:17.144352 2148722 round_trippers.go:463]     Content-Type: application/json
I0219 05:50:17.144356 2148722 round_trippers.go:463]     X-Kubernetes-Pf-Flowschema-Uid: 7f136704-82ad-4f6c-8c86-b470a972fede
I0219 05:50:17.144396 2148722 request.go:1181] Response Body: {&quot;apiVersion&quot;:&quot;sparkoperator.k8s.io/v1beta2&quot;,&quot;kind&quot;:&quot;SparkApplication&quot;,&quot;metadata&quot;:{&quot;creationTimestamp&quot;:&quot;2023-02-19T05:50:17Z&quot;,&quot;generation&quot;:1,&quot;managedFields&quot;:[{&quot;apiVersion&quot;:&quot;sparkoperator.k8s.io/v1beta2&quot;,&quot;fieldsType&quot;:&quot;FieldsV1&quot;,&quot;fieldsV1&quot;:{&quot;f:spec&quot;:{&quot;.&quot;:{},&quot;f:arguments&quot;:{},&quot;f:driver&quot;:{&quot;.&quot;:{},&quot;f:cores&quot;:{},&quot;f:labels&quot;:{&quot;.&quot;:{},&quot;f:role&quot;:{},&quot;f:spark-app&quot;:{},&quot;f:version&quot;:{}},&quot;f:memory&quot;:{},&quot;f:nodeSelector&quot;:{&quot;.&quot;:{},&quot;f:beta.kubernetes.io/instance-type&quot;:{}},&quot;f:serviceAccount&quot;:{}},&quot;f:executor&quot;:{&quot;.&quot;:{},&quot;f:cores&quot;:{},&quot;f:instances&quot;:{},&quot;f:labels&quot;:{&quot;.&quot;:{},&quot;f:role&quot;:{},&quot;f:version&quot;:{}},&quot;f:memory&quot;:{},&quot;f:memoryOverhead&quot;:{},&quot;f:nodeSelector&quot;:{&quot;.&quot;:{},&quot;f:beta.kubernetes.io/instance-type&quot;:{}}},&quot;f:hadoopConf&quot;:{&quot;.&quot;:{},&quot;f:fs.oss.accessKeyId&quot;:{},&quot;f:fs.oss.accessKeySecret&quot;:{},&quot;f:fs.oss.endpoint&quot;:{},&quot;f:fs.oss.impl&quot;:{}},&quot;f:image&quot;:{},&quot;f:imagePullPolicy&quot;:{},&quot;f:mainApplicationFile&quot;:{},&quot;f:mainClass&quot;:{},&quot;f:mode&quot;:{},&quot;f:restartPolicy&quot;:{&quot;.&quot;:{},&quot;f:type&quot;:{}},&quot;f:sparkConf&quot;:{&quot;.&quot;:{},&quot;f:spark.eventLog.dir&quot;:{},&quot;f:spark.eventLog.enabled&quot;:{},&quot;f:spark.kubernetes. [truncated 1965 chars]
sparkapplication.sparkoperator.k8s.io/tpcds-25-query created
</code></pre>
<p>From the logs, we can see the only error &quot;Response Status: 503 Service Unavailable in 8 milliseconds&quot;, I don't know what it means.</p>
<p>So I want to ask what may cause this, and how would I diagnose the problem? Any help is appreciated!</p>

<p>There might be multiple reasons for this, initially let’s check whether the pod is really created or not. Like <code>ehmad11</code> suggested use <code>kubectl get pods --all-namespaces</code> for listing pods in all the namespaces. However in your case it might not work because your application is getting directly deployed in defaulf namespace. Regarding the error “Response Status: 503 Service Unavailable in 8 milliseconds” once you are able to locate the pod use <code>kubectl describe &lt;pod&gt;</code> for finding logs specific to your pod and follow the troubleshooting steps provided in this <a href="https://komodor.com/learn/how-to-fix-kubernetes-service-503-service-unavailable-error/" rel="nofollow noreferrer">document</a> for rectifying it.</p>
<p><strong>Note:</strong> The reference document is provided from <code>komodor</code> site, here they have articulated each troubleshooting step in highly detailed and understandable manner.</p>
