<p>Our organisation runs Databricks on Azure that is used by data scientists &amp; analysts primarily for Notebooks in order to do ad-hoc analysis and exploration.</p>
<p>We also run Kubernetes clusters for non spark-requiring ETL workflows.</p>
<p>We would like to use Delta Lakes as our storage layer where both Databricks and Kubernetes are able to read and write as first class citizens.<br />
Currently our Kubernetes jobs write parquets directly to blob store, with an additional job that spins up a databricks cluster to load the parquet data into Databrick's table format. This is slow and expensive.</p>
<p>What I would like to do is write to Delta lake from Kubernetes python directly, as opposed to first dumping a parquet file to blob store and then triggering an additional Databricks job to load it into Delta lake format.<br />
Conversely, I'd like to also leverage Delta lake to query from Kubernetes.</p>
<hr />
<p>In short, how do I set up my Kubernetes python environment such that it has equal access to the existing Databricks Delta Lake for writes &amp; queries?<br />
Code would be appreciated.</p>

<p>You can <em>usually</em> can write into the Delta table using <a href="https://delta.io/" rel="nofollow noreferrer">Delta connector for Spark</a>.  Just start a Spark job with <a href="https://docs.delta.io/latest/quick-start.html#set-up-apache-spark-with-delta-lake" rel="nofollow noreferrer">necessary packages and configuration options</a>:</p>
<pre class="lang-sh prettyprint-override"><code>spark-submit --packages io.delta:delta-core_2.12:1.0.0 \
  --conf &quot;spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension&quot; 
  --conf &quot;spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog&quot; 
...
</code></pre>
<p>and write the same way as on Databricks:</p>
<pre class="lang-py prettyprint-override"><code>df.write.format(&quot;delta&quot;).mode(&quot;append&quot;).save(&quot;some_location&quot;)
</code></pre>
<p>But by using OSS version of Delta you may loose some of the optimizations that are available only on Databricks, like, <a href="https://docs.databricks.com/delta/optimizations/file-mgmt.html#data-skipping" rel="nofollow noreferrer">Data Skipping</a>, etc. - in this case performance for the data written from Kubernetes <em>could be</em> lower (really depends on how do you access data).</p>
<p>There could be a case when you couldn't write into Delta table create by Databricks - when the table was written by writer with writer version higher that supported by OSS Delta connector (see <a href="https://github.com/delta-io/delta/blob/master/PROTOCOL.md#writer-version-requirements" rel="nofollow noreferrer">Delta Protocol documentation</a>). For example, this happens when you enable <a href="https://docs.databricks.com/delta/delta-change-data-feed.html" rel="nofollow noreferrer">Change Data Feed</a> on the Delta table that performs additional actions when writing data.</p>
<p>Outside of Spark, there are plans for implementing so-called <a href="https://github.com/delta-io/connectors/issues/85" rel="nofollow noreferrer">Standalone writer</a> for JVM-based languages (in addition to existing <a href="https://github.com/delta-io/connectors" rel="nofollow noreferrer">Standalone reader</a>).  And there is a <a href="https://github.com/delta-io/delta-rs" rel="nofollow noreferrer">delta-rs project</a> implemented in Rust (with bindings for Python &amp; Ruby) that should be able to write into Delta table (but I haven't tested that myself)</p>
<p>Update 14.04.2022: Data Skipping is also available in OSS Delta, starting with version 1.2.0</p>
