<p>Use case:
Get stream from Kafka store in parquet file using spark.
Open these parquet files and generate graph using graphframes.</p>
<p>Infra:
i have a bitnami spark infra on kubernetes connected to Kafka.</p>
<p>The goal is to call the spark-submit into a kubernetes pods.
With that all the code run into kubernetes and i doesn't install spark outside kubernetes.</p>
<p>Without kubernetes, i have do the job into spark master container:</p>
<pre><code>docker cp ./Spark/Python_code/edge_stream.py spark_spark_1:/opt/bitnami/spark/edge_stream.py
docker cp ./Spark/Python_code/config.json spark_spark_1:/opt/bitnami/spark/config.json
docker exec spark_spark_1 \
    spark-submit \
    --master spark://0.0.0.0:7077 \
    --deploy-mode client \
    --conf spark.cores.max=1 \
    --conf spark.executor.memory=1g \
    --conf spark.eventLog.enabled=true \
    --conf spark.eventLog.dir=/tmp/spark-events \
    --conf spark.eventLog.rolling.maxFileSize=256m\
    /opt/bitnami/spark/edge_stream.py
</code></pre>
<p>Is it possible to do the same job in kubernetes ?</p>
<p>Best regards</p>

<p>Using exec command of kubernetes</p>
<pre><code>minikube kubectl -- exec my-spark-master-0 -- spark-submit \
    --master spark://0.0.0.0:7077 \
    --deploy-mode client \
    --conf spark.cores.max=1 \
    --conf spark.executor.memory=1g \
    --conf spark.eventLog.enabled=true \
    --conf spark.eventLog.dir=/tmp/spark-events \
    --conf spark.eventLog.rolling.maxFileSize=256m\
    ../Python/edge_stream.py
</code></pre>
