<p>Running into the following error when use custom UDF</p>
<pre><code>Traceback (most recent call last):
  File &quot;/usr/local/lib/python3.8/dist-packages/pyspark/worker.py&quot;, line 603, in main
    func, profiler, deserializer, serializer = read_udfs(pickleSer, infile, eval_type)
  File &quot;/usr/local/lib/python3.8/dist-packages/pyspark/worker.py&quot;, line 449, in read_udfs
    udfs.append(read_single_udf(pickleSer, infile, eval_type, runner_conf, udf_index=i))
  File &quot;/usr/local/lib/python3.8/dist-packages/pyspark/worker.py&quot;, line 251, in read_single_udf
    f, return_type = read_command(pickleSer, infile)
  File &quot;/usr/local/lib/python3.8/dist-packages/pyspark/worker.py&quot;, line 71, in read_command
    command = serializer._read_with_length(file)
  File &quot;/usr/local/lib/python3.8/dist-packages/pyspark/serializers.py&quot;, line 160, in _read_with_length
    return self.loads(obj)
  File &quot;/usr/local/lib/python3.8/dist-packages/pyspark/serializers.py&quot;, line 430, in loads
    return pickle.loads(obj, encoding=encoding)
ModuleNotFoundError: No module named 'jobs'
</code></pre>
<p>The import spark scripts looks something like this</p>
<pre><code>from jobs.lib_a import a
from jobs.udf import udf_function #This is a UDF
</code></pre>
<p>The scripts itself is located in
<code>jobs/scripts/test_script.py</code>, the entire jobs folder is zipped and then added to spark using pyFiles.</p>
<p>The weird thing is that the other import from jobs module works, only fail for udf.</p>
<p>I have tried approach in this <a href="https://stackoverflow.com/questions/59741832/pyspark-custom-udf-modulenotfounderror-no-module-named">post</a>, creating a separate zip file called udf.zip, putting udf at top level and then add it to spark via pyFiles, but still run into ModuleNotFoundError when I try to import <code>udf</code>.</p>
<p>I have also tried <code>sys.path.append(&lt;the udf path&gt;)</code></p>
<p>The only approach works is when I copy the udf_function into the spark script <code>test_script.py</code>. This wouldn't work in reality as the udf_function can be shared by other spark script.</p>
<p>The underlying system is:
Python 3.8
Spark 3.2
Spark is running in kubernetes</p>

<p>I was able to make it work.</p>
<p>Some more context is that we are leveraging spark on k8s operator, so we pass in the zip file via pyFiles
<a href="https://github.com/GoogleCloudPlatform/spark-on-k8s-operator" rel="nofollow noreferrer">https://github.com/GoogleCloudPlatform/spark-on-k8s-operator</a></p>
<p>This work the same as the following</p>
<pre><code>spark_session.sparkContext.addPyFile('/opt/spark/pyfiles/python.zip')
</code></pre>
<p>if we set it up in spark script</p>
<p>Initially, we pass in as</p>
<pre><code>pyFiles:
- local:///opt/spark/pyfiles/spinner-python.zip
</code></pre>
<p>But local:/// point to the working directory, we have to change it to the following with extra slash(/) to point to absolute.</p>
<pre><code>pyFiles:
- local:////opt/spark/pyfiles/spinner-python.zip
</code></pre>
<p>When Pyspark serializes a UDF, it sends a copy of the UDF code to all the worker nodes, we have to point PyFiles to absolute path instead of relative path</p>
