title: Sharing large files between airflow operators
body: <p>Let's assume an airflow DAG consists of the following two operators:</p>
  <ul>
  <li><p>Op1 (<code>GCSToLocalFilesystemOperator</code>): Downloads the file XYZ from GCS</p>
  </li>
  <li><p>Op2 (<code>PythonOperator</code>): Needs the file XYZ to do something with it</p>
  </li>
  </ul>
  <p>The DAG will be executed by GCP's Composer. I know it is possible to combine the two operators, and implement them as one <code>PythonOperator</code>, but I want to do it more modular, and use builtin operators as much as possible.</p>
  <p><strong>Candidate Solutions:</strong></p>
  <ul>
  <li><p>Use a temporary file: temporary files cannot be used. If XYZ is stored as /tmp/XYZ, Op2 may or may not find it, since the two Operators might be run by different workers.</p>
  </li>
  <li><p>xcom: xcom cannot also be used here since the file is rather large.</p>
  </li>
  <li><p>Use GCS as a shared storage: It does not work in this particular example. We are back to where we are, since one operator still needs to do the downloading and availing the file for the second operator.</p>
  </li>
  </ul>
  <p>So, what is a good solution here? In general, is there a way to hand off a file (or a string) from one operator to another?</p>