<p>So I am deploying an EMR application using EKS, following the EMR-EKS workshop by AWS, declaring the Fargate profile.</p>
<p>I tried everything to be serverless, so even my EKS Cluster runs on Fargate (kube-sytem, default, etc).</p>
<p>I created a custom namespace with a Fargate profile, and submit a Spark job. The job stayed at Pending status.</p>
<p>When I added some managed nodegroups, the job was submitted successfully.</p>
<p>I tried submitting very light jobs and by the time I removed the managed node groups, spark jobs stay at pending status.</p>
<pre><code>Events:
  Type     Reason            Age                  From               Message
  ----     ------            ----                 ----               -------
  Warning  FailedScheduling  33s (x7 over 5m43s)  default-scheduler  0/2 nodes are available: 2 node(s) had taint {eks.amazonaws.com/compute-type: fargate}, that the pod didn't tolerate.
</code></pre>
<p><strong>eksworkshop-eksctl.yaml</strong></p>
<pre><code>apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: eksworkshop-eksctl
  region: ${AWS_REGION}
  version: &quot;1.21&quot;
    
availabilityZones: [&quot;${AZS[0]}&quot;, &quot;${AZS[1]}&quot;, &quot;${AZS[2]}&quot;, &quot;${AZS[3]}&quot;]

fargateProfiles:
  - name: default
    selectors:
      # All workloads in the &quot;default&quot; Kubernetes namespace will be
      # scheduled onto Fargate:
      - namespace: default
      # All workloads in the &quot;kube-system&quot; Kubernetes namespace will be
      # scheduled onto Fargate:
      - namespace: kube-system
</code></pre>
<p>Create the EKS cluster:</p>
<pre><code>eksctl create cluster -f eksworkshop-eksctl.yaml
</code></pre>
<p>Create namespace and fargate profile:</p>
<pre><code>kubectl create namespace spark
eksctl create fargateprofile --cluster eksworkshop-eksctl --name emr \
--namespace spark --labels type=etl
</code></pre>
<p>Create virtual cluster:</p>
<p>I have already added the labels under sparkSubmitParameters, but it is still stuck in Pending state. :( Is there additional configuration I need to add when creating virtual cluster:</p>
<pre><code>aws emr-containers create-virtual-cluster \
--name eksworkshop-eksctl \
--container-provider '{
    &quot;id&quot;: &quot;eksworkshop-eksctl&quot;,
    &quot;type&quot;: &quot;EKS&quot;,
    &quot;info&quot;: {
        &quot;eksInfo&quot;: {
            &quot;namespace&quot;: &quot;spark&quot;
        }
    }
}'
</code></pre>
<p>Submit the spark job:</p>
<pre><code>aws emr-containers start-job-run --virtual-cluster-id $VIRTUAL_CLUSTER_ID --name spark-pi-logging --execution-role-arn $EMR_ROLE_ARN --release-label emr-6.2.0-latest --job-driver '{
    &quot;sparkSubmitJobDriver&quot;: {
        &quot;entryPoint&quot;: &quot;s3://aws-data-analytics-workshops/emr-eks-workshop/scripts/pi.py&quot;,
        &quot;sparkSubmitParameters&quot;: &quot;--conf spark.kubernetes.driver.label.type=etl --conf spark.kubernetes.executor.label.type=etl --conf spark.executor.instances=2 --conf spark.executor.memory=2G --conf spark.executor.cores=2 --conf spark.driver.cores=1&quot;
        }
    }' --configuration-overrides '{
    &quot;applicationConfiguration&quot;: [
      {
        &quot;classification&quot;: &quot;spark-defaults&quot;,
        &quot;properties&quot;: {
          &quot;spark.driver.memory&quot;:&quot;2G&quot;
         }
      }
    ],
    &quot;monitoringConfiguration&quot;: {
      &quot;cloudWatchMonitoringConfiguration&quot;: {
        &quot;logGroupName&quot;: &quot;/emr-containers/jobs&quot;,
        &quot;logStreamNamePrefix&quot;: &quot;emr-eks-workshop&quot;
      },
      &quot;s3MonitoringConfiguration&quot;: {
        &quot;logUri&quot;: &quot;'&quot;$s3DemoBucket&quot;'/logs/&quot;
      }
    }
}'
</code></pre>
<p>For EKS cluster, is NodeGroup must be declared and is mandatory, and we can't run Spark Job just be using Fargate only?</p>

<p>Your situation:</p>
<p><code>...created fargate-profile with the same namespace: Job completed</code></p>
<p><code>... created fargate-profile with the same namespace but with labels...Job stucked in Pending </code></p>
<p>If a namespace is associated with &gt;1 profile, EKS will randomly choose a profile. When it picks a profile that requires a label and your spark job don't have it - it goes into pending state since there is no other node group in your cluster.</p>
<p><code>Is it required to have 2 profiles, one with label, and one with no label</code></p>
<p>No, in fact you should not do this. To run job on a profile with labels your job <code>sparkSubmitParameters</code> must specify the label:</p>
<p><code>&quot;sparkSubmitParameters&quot;: &quot;...--conf spark.kubernetes.driver.label.&lt;label key&gt;=&lt;value&gt; --conf spark.kubernetes.executor.label.&lt;label key&gt;=&lt;value&gt;...&quot;</code></p>
<p>You don't need &gt;1 Fargate profile unless you need to distinguish spark jobs by Fargate profile selector.</p>
