<p>Am in very early stages of exploring Argo with Spark operator to run Spark samples on the minikube setup on my EC2 instance.</p>
<p>Following are the resources details, not sure why am not able to see the spark app logs.</p>
<p><strong>WORKFLOW.YAML</strong></p>
<pre><code>kind: Workflow
metadata:
  name: spark-argo-groupby
spec:
  entrypoint: sparkling-operator
  templates:
  - name: spark-groupby
    resource:
      action: create
      manifest: |
        apiVersion: &quot;sparkoperator.k8s.io/v1beta2&quot;
        kind: SparkApplication
        metadata:
          generateName: spark-argo-groupby
        spec:
          type: Scala
          mode: cluster
          image: gcr.io/spark-operator/spark:v3.0.3
          imagePullPolicy: Always
          mainClass: org.apache.spark.examples.GroupByTest
          mainApplicationFile:  local:///opt/spark/spark-examples_2.12-3.1.1-hadoop-2.7.jar
          sparkVersion: &quot;3.0.3&quot;
          driver:
            cores: 1
            coreLimit: &quot;1200m&quot;
            memory: &quot;512m&quot;
            labels:
              version: 3.0.0
          executor:
            cores: 1
            instances: 1
            memory: &quot;512m&quot;
            labels:
              version: 3.0.0
  - name: sparkling-operator
    dag:
      tasks:
      - name: SparkGroupBY
        template: spark-groupby

</code></pre>
<p><strong>ROLES</strong></p>
<pre><code># Role for spark-on-k8s-operator to create resources on cluster
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: spark-cluster-cr
  labels:
    rbac.authorization.kubeflow.org/aggregate-to-kubeflow-edit: &quot;true&quot;
rules:
  - apiGroups:
      - sparkoperator.k8s.io
    resources:
      - sparkapplications
    verbs:
      - '*'
---
# Allow airflow-worker service account access for spark-on-k8s
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: argo-spark-crb
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: spark-cluster-cr
subjects:
  - kind: ServiceAccount
    name: default
    namespace: argo

</code></pre>
<p><strong>ARGO UI</strong></p>
<p><a href="https://i.stack.imgur.com/6KqvX.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/6KqvX.png" alt="Workflow status" /></a></p>
<p><a href="https://i.stack.imgur.com/pzuAZ.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/pzuAZ.png" alt="Workflow logs" /></a></p>
<p>To dig deep I tried all the steps that's listed on <a href="https://dev.to/crenshaw_dev/how-to-debug-an-argo-workflow-31ng" rel="nofollow noreferrer">https://dev.to/crenshaw_dev/how-to-debug-an-argo-workflow-31ng</a> yet could not get app logs.</p>
<p>Basically when I run these examples am expecting spark app logs to be printed - in this case output of following Scala example</p>
<p><a href="https://github.com/apache/spark/blob/master/examples/src/main/scala/org/apache/spark/examples/GroupByTest.scala" rel="nofollow noreferrer">https://github.com/apache/spark/blob/master/examples/src/main/scala/org/apache/spark/examples/GroupByTest.scala</a></p>
<p>Interesting when I list PODS, I was expecting to see driver pods and executor pods but always see only one POD and it has above logs as in the image attached. <strong>Please help me to understand why logs are not generated and how can I get it?</strong></p>
<pre><code>RAW LOGS
$ kubectl logs spark-pi-dag-739246604 -n argo

time=&quot;2021-12-10T13:28:09.560Z&quot; level=info msg=&quot;Starting Workflow Executor&quot; version=&quot;{v3.0.3 2021-05-11T21:14:20Z 02071057c082cf295ab8da68f1b2027ff8762b5a v3.0.3 clean go1.15.7 gc linux/amd64}&quot;
time=&quot;2021-12-10T13:28:09.581Z&quot; level=info msg=&quot;Creating a docker executor&quot;
time=&quot;2021-12-10T13:28:09.581Z&quot; level=info msg=&quot;Executor (version: v3.0.3, build_date: 2021-05-11T21:14:20Z) initialized (pod: argo/spark-pi-dag-739246604) with template:\n{\&quot;name\&quot;:\&quot;sparkpi\&quot;,\&quot;inputs\&quot;:{},\&quot;outputs\&quot;:{},\&quot;metadata\&quot;:{},\&quot;resource\&quot;:{\&quot;action\&quot;:\&quot;create\&quot;,\&quot;manifest\&quot;:\&quot;apiVersion: \\\&quot;sparkoperator.k8s.io/v1beta2\\\&quot;\\nkind: SparkApplication\\nmetadata:\\n  generateName: spark-pi-dag\\nspec:\\n  type: Scala\\n  mode: cluster\\n  image: gjeevanm/spark:v3.1.1\\n  imagePullPolicy: Always\\n  mainClass: org.apache.spark.examples.SparkPi\\n  mainApplicationFile: local:///opt/spark/spark-examples_2.12-3.1.1-hadoop-2.7.jar\\n  sparkVersion: 3.1.1\\n  driver:\\n    cores: 1\\n    coreLimit: \\\&quot;1200m\\\&quot;\\n    memory: \\\&quot;512m\\\&quot;\\n    labels:\\n      version: 3.0.0\\n  executor:\\n    cores: 1\\n    instances: 1\\n    memory: \\\&quot;512m\\\&quot;\\n    labels:\\n      version: 3.0.0\\n\&quot;},\&quot;archiveLocation\&quot;:{\&quot;archiveLogs\&quot;:true,\&quot;s3\&quot;:{\&quot;endpoint\&quot;:\&quot;minio:9000\&quot;,\&quot;bucket\&quot;:\&quot;my-bucket\&quot;,\&quot;insecure\&quot;:true,\&quot;accessKeySecret\&quot;:{\&quot;name\&quot;:\&quot;my-minio-cred\&quot;,\&quot;key\&quot;:\&quot;accesskey\&quot;},\&quot;secretKeySecret\&quot;:{\&quot;name\&quot;:\&quot;my-minio-cred\&quot;,\&quot;key\&quot;:\&quot;secretkey\&quot;},\&quot;key\&quot;:\&quot;spark-pi-dag/spark-pi-dag-739246604\&quot;}}}&quot;
time=&quot;2021-12-10T13:28:09.581Z&quot; level=info msg=&quot;Loading manifest to /tmp/manifest.yaml&quot;
time=&quot;2021-12-10T13:28:09.581Z&quot; level=info msg=&quot;kubectl create -f /tmp/manifest.yaml -o json&quot;
time=&quot;2021-12-10T13:28:10.348Z&quot; level=info msg=argo/SparkApplication.sparkoperator.k8s.io/spark-pi-daghhl6s
time=&quot;2021-12-10T13:28:10.348Z&quot; level=info msg=&quot;Starting SIGUSR2 signal monitor&quot;
time=&quot;2021-12-10T13:28:10.348Z&quot; level=info msg=&quot;No output parameters&quot;

</code></pre>

<p>As Michael mentioned in his answer, Argo Workflows does not know how other CRDs (such as <code>SparkApplication</code> that you used) work and thus could not pull the logs from the pods created by that particular CRD.</p>
<p>However, you can add the label <code>workflows.argoproj.io/workflow: {{workflow.name}}</code> to the pods generated by <code>SparkApplication</code> to let Argo Workflows know and then use <code>argo logs -c &lt;container-name&gt;</code> to pull the logs from those pods.</p>
<p>You can find an example here but Kubeflow CRD but in your case you'll want to add labels to the <code>executor</code> and <code>driver</code> to your <code>SparkApplication</code> CRD in the resource template: <a href="https://github.com/argoproj/argo-workflows/blob/master/examples/k8s-resource-log-selector.yaml" rel="nofollow noreferrer">https://github.com/argoproj/argo-workflows/blob/master/examples/k8s-resource-log-selector.yaml</a></p>
