title: When SSL is enabled, Spark UI doesn't use HTTPS, but forwards to port 0 over HTTP
body: <p>We are using Spark 2.0.2 and Hadoop 3.2.1. I have gotten SSL configured across Hadoop without any trouble. But Spark is having some trouble.</p>
  
  <p>Without SSL, I can launch a job and view the Spark UI, proxied through Yarn.</p>
  
  <p>When I enable SSL, I can still launch a job and run to completion, but I can not access the Spark Web UI. Yarn links to an ApplicationMaster proxy that fails with <code>java.net.ConnectException: Connection refused (Connection refused)</code></p>
  
  <p>Digging into the job's stdout, I see lines like this:</p>
  
  <pre><code>2020-04-14 16:21:57,258 INFO server.ServerConnector: Started ServerConnector@1484944f{HTTP/1.1}{0.0.0.0:40809}
  2020-04-14 16:21:57,303 INFO server.ServerConnector: Started ServerConnector@799e8b3a{SSL-HTTP/1.1}{0.0.0.0:36015}
  2020-04-14 16:21:57,303 INFO server.Server: Started @5614ms
  2020-04-14 16:21:57,304 INFO util.Utils: Successfully started service 'SparkUI' on port 40809.
  2020-04-14 16:21:57,309 INFO ui.SparkUI: Bound SparkUI to 0.0.0.0, and started at https://10.10.75.40:40809
  </code></pre>
  
  <p>If I call up <code>https://10.10.75.40:40809</code> I get an error that it isn't SSL.</p>
  
  <p>If I call up <code>http://10.10.75.40:40809</code> I get served a bogus redirect to port 10.10.75.40:0.</p>
  
  <p>If I call up <code>https://10.10.75.40:36015</code> and ignore certificate errors, or if I call via the correct hostname, I get redirected to the RM's <code>:8090/proxy/redirect</code> which throws the 500 error again.</p>
  
  <p>My configuration consists of:</p>
  
  <pre><code>spark.ssl.enabled                 true
  spark.ssl.keyStore                /etc/sslmate/STAR.mtv.qxxxxxxxxd.com.jks
  spark.ssl.keyStorePassword        _the password_
  spark.ssl.trustStore              /etc/sslmate/STAR.mtv.qxxxxxxxxd.com.jks
  spark.ssl.trustStorePassword      _the password_
  spark.ssl.protocol                TLSv1.2
  </code></pre>
  
  <p>I have also experimented with trying to set ports explicitly, to no avail:</p>
  
  <pre><code>spark.ssl.ui.port                 4440
  spark.ssl.history.port            18480
  spark.ui.driver.port              2020
  spark.ssl.ui.driver.port          2420
  </code></pre>