<p>I'm using Airflow with kubernetes executor and the <code>KubernetesPodOperator</code>. I have two jobs:</p>

<ul>
<li>A: Retrieve data from some source up to 100MB</li>
<li>B: Analyze the data from A.</li>
</ul>

<p>In order to be able to share the data between the jobs, I would like to run them on the same pod, and then A will write the data to a volume, and B will read the data from the volume.</p>

<p>The <a href="https://airflow.apache.org/kubernetes.html" rel="nofollow noreferrer">documentation</a> states: </p>

<blockquote>
  <p>The Kubernetes executor will create a new pod for every task instance.</p>
</blockquote>

<p>Is there any way to achieve this? And if not, what recommended way there is to pass the data between the jobs?</p>

matanper
<p>Sorry this isn't possible - one job per pod.</p>

<p>You are best to use task 1 to put the data in a well known location (e.g in a cloud bucket) and get it from the second task. Or just combine the two tasks.</p>
