<p>I'm exploring this python package <code>mrjob</code> to run MapReduce jobs in python. I've tried running it in the local environment and it works perfectly.</p>
<p>I have Hadoop 3.3 runs on Kubernetes (GKE) cluster. So I also managed to run <code>mrjob</code> successfully in the name-node pod from inside.</p>
<p>Now, I've got a Jupyter Notebook pod running in the same Kubernetes cluster (same namespace). I wonder whether I can run <code>mrjob</code> MapReduce jobs from the Jupyter Notebook.</p>
<p>The problem seems to be that I don't have $HADOOP_HOME defined in the Jupyter Notebook environment. So based on the documentation I created a config file called <code>mrjob.conf</code> as follows;</p>
<pre><code>runners:
 hadoop:
  cmdenv:
    PATH: &lt;pod name&gt;:/opt/hadoop
</code></pre>
<p>However <code>mrjob</code> is still unable to detect hadoop bin and gives the below error</p>
<pre><code>FileNotFoundError: [Errno 2] No such file or directory: 'hadoop'
</code></pre>
<p>So is there a way in which I can configure <code>mrjob</code> to run with my existing Hadoop installation on the GKE cluster? I've tried searching for similar examples but was unable to find one.</p>

Thisara Watawana
<p><code>mrjob</code> is a wrapper around <code>hadoop-streaming</code>, therefore requires Hadoop binaries to be installed on the server(s) where the code will run (pods here, I guess); including the Juptyer pod that submits the application.</p>
<p>IMO, it would be much easier for you to deploy PySpark/PyFlink/Beam applications in k8s than hadoop-streaming since you don't &quot;need&quot; Hadoop in k8s to run such distributed processes.</p>
<p>Beam would be recommended since it is compatible with GCP DataFlow</p>
