<p>I have created an HTTP server from the directory using this command:</p>

<pre><code>python -c 'import BaseHTTPServer as bhs, SimpleHTTPServer as shs; bhs.HTTPServer(("0.0.0.0", 8089), shs.SimpleHTTPRequestHandler).serve_forever()'
</code></pre>

<p>I use Spark-K8s operator for the Spark application execution. As main file I want to use jar file stored in the directory (from which I create http server). However, I do not know to which host I should map my directory so it could be used by the Spark application running through the deployment.
When I run it with a current host I get this connection error:</p>

<pre><code>java.net.ConnectException: Connection refused
</code></pre>

<p>Basically, I have HTTP server which refers to specified host and port and I want to run this jar file using Spark on top of K8s. How can I define this host?</p>

<p>For now application definition looks like this:</p>

<pre><code>apiVersion: sparkoperator.k8s.io/v1alpha1
kind: SparkApplication
metadata:
  name: spark-example
  namespace: default
spec:
  type: Scala
  image: gcr.io/spark-operator/spark:v2.4.0
  mainClass: org.apache.spark.examples.SparkExample
  mainApplicationFile: https://0.0.0.0:8089/spark_k8s_airflow.jar
  mode: cluster
  deps: {}
  driver:
    coreLimit: 1000m
    cores: 0.1
    labels:
      version: 2.4.0
    memory: 1024m
    serviceAccount: intended-mink-spark
  executor:
    cores: 1
    instances: 1
    labels:
      version: 2.4.0
    memory: 1024m
  imagePullPolicy: Never
</code></pre>

Cassie
<blockquote>
  <p>Basically, I have HTTP server which refers to specified host and port and I want to run this jar file using Spark on top of K8s. How can I define this host?</p>
</blockquote>

<p>The kubernetes way of doing that is via a <code>Service</code>, which by default creates a DNS entry of the form <code>service-name.service-namespace.svc.cluster.local</code> where <code>service-name</code> and <code>service-namespace</code> are not those literal words, but the other 3 are literally that. However, if you just want to play around, and creating a <code>Service</code> is too much work, then you can use the current IP of the Pod in which your <code>SimpleHTTPServer</code> is running.</p>

<blockquote>
  <p><code>mainApplicationFile: https://0.0.0.0:8089/spark_k8s_airflow.jar</code></p>
</blockquote>

<p>Be aware that, at least as you have written the python example above, you cannot use <code>https:</code> since <code>SimpleHTTPServer</code> is just that <strong>HTTP</strong>. It's possible you can convince one of the built-in packages to serve https, but it'll be a lot more typing and arguably not worth the effort</p>
