title: Real Time Cluster Log Delivery in a Databricks Cluster
body: <p>I have some Python code that I am running on a Databricks Job Cluster. My Python code will be generating a whole bunch of logs and I want to be able to monitor these logs in real time (or near real time), say through something like a dashboard.</p>
  <p>What I have done so far is, I have configured my cluster log delivery location and my logs are delivered to the specified destination every 5 minutes.</p>
  <p>This is explained here,
  <br>
  <a href="https://docs.microsoft.com/en-us/azure/databricks/clusters/configure" rel="nofollow noreferrer">https://docs.microsoft.com/en-us/azure/databricks/clusters/configure</a></p>
  <p>Here is an extract from the same article,</p>
  <blockquote>
  <p>When you create a cluster, you can specify a location to deliver the
  logs for the Spark driver node, worker nodes, and events. Logs are
  delivered every five minutes to your chosen destination. When a
  cluster is terminated, Azure Databricks guarantees to deliver all logs
  generated up until the cluster was terminated.</p>
  </blockquote>
  <p>Is there some way I can have these logs delivered somewhere in near real time, rather than every 5 minutes? It does not have to be through the same method either, I am open to other possibilities.</p>