<p>I am looking for a way to introduce LDAP authentication to my Apache Spark Web UIs. This includes both the main Spark Web UI and the Spark history server. The Spark instances are running inside a Kubernetes cluster and are launched both via the Spark Operator and in Standalone mode.</p>
<p>In the official documentation, <a href="https://spark.apache.org/docs/latest/security.html#authentication-and-authorization" rel="nofollow noreferrer">it says the following</a>:</p>
<p>&quot;Enabling authentication for the Web UIs is done using <a href="https://docs.oracle.com/javaee/6/api/javax/servlet/Filter.html" rel="nofollow noreferrer">javax servlet filters</a>. You will need a filter that implements the authentication method you want to deploy. Spark does not provide any built-in authentication filters.&quot;</p>
<p>When searching online, I cannot find any such filters which will enable authentication through LDAP.</p>
<p>My questions are:</p>
<ol>
<li>Are there any such filters implemented that is open source and freely available?</li>
<li>Is there another solution to my problem which will enable this feature?</li>
</ol>

<ol>
<li>Add a Java Servlet filter. I didn't find a public available ldap filter, however here is a PAM Filter, we can use <a href="https://github.com/marccarre/pam-servlet-filter" rel="nofollow noreferrer">https://github.com/marccarre/pam-servlet-filter</a>.
Now, we can configure PAM to use SSSD which uses LDAP.</li>
</ol>
<p>Hint: Use thin client and use libpam4j-1.11.jar (not the official referenced version (1.9) since i could not get it work with 1.9) and jna-4.3.0.jar. This was a working setup for me.
Add all of these 3 jars to SPARK_DIR/jars/</p>
<ol start="2">
<li>On all spark nodes execute</li>
</ol>
<pre><code>yum install sssd authconfig -y
authconfig --update --enablesssd --enablesssdauth
</code></pre>
<ol start="3">
<li><p>Add / Edit /etc/sssd/sssd.conf.
Google how to configure this file.
Basically you can configure here variables like ldap_uri, ldap_search_base and ldap_access_filter</p>
</li>
<li><p>Add following Variables in spark-defaults.conf:</p>
</li>
</ol>
<pre><code>spark.ui.filters                    com.carmatechnologies.servlet.PamAuthFilter
spark.com.carmatechnologies.servlet.PamAuthFilter.param.realm spark
spark.com.carmatechnologies.servlet.PamAuthFilter.param.service system-auth
</code></pre>
<ol start="5">
<li>Start sssd</li>
</ol>
<pre><code>sssd -i -d 3
</code></pre>
<ol start="6">
<li>Start Spark UIs</li>
</ol>
<p>Authentication should work now.</p>
