title: 'pimp' airflow databricks_hook or some python library to create and get the cluster_id for downstream tasks
body: <p>I have similar questions like below, but i wonder there is an existing library work nicely with airflow to <code>create databricks cluster, return the cluster_id, and reuse for the downstream tasks</code>.</p>
  <p><a href="https://stackoverflow.com/questions/54561640/triggering-databricks-job-from-airflow-without-starting-new-cluster">Triggering Databricks job from Airflow without starting new cluster</a></p>
  <p>My study shows: the <code>DatabricksHook</code> class has quite some nice methods and api calls, but it does not have calls to <code>create cluster</code> and re-use the cluster in the same DAG.</p>
  <p>if i have to add the methods myself:
  In scala or other language, one could 'pimp' the library to add new methods to an 3rd party class. Any suggestion in <code>python</code> to do elegant way of adding extra methods?</p>
  <ul>
  <li>Do I have to either 'duplicate' some of the code which is written in the <code>Databrickshook</code> into my project, and add the missing methods</li>
  <li>or maybe contribute to the airflow project to 'add' the missing methods to the <code>DatabricksHook</code> class. This might take a longer time than i could wait for.</li>
  </ul>
  <p>Info:</p>
  <ul>
  <li>the existing code for the hook class, where it has rest API calls: <a href="https://airflow.apache.org/docs/apache-airflow/1.10.3/_modules/airflow/contrib/hooks/databricks_hook.html" rel="nofollow noreferrer">https://airflow.apache.org/docs/apache-airflow/1.10.3/_modules/airflow/contrib/hooks/databricks_hook.html</a></li>
  <li>curl interface to create cluster etc: <a href="https://docs.microsoft.com/en-us/azure/databricks/dev-tools/api/latest/clusters#--create" rel="nofollow noreferrer">https://docs.microsoft.com/en-us/azure/databricks/dev-tools/api/latest/clusters#--create</a></li>
  </ul>