title: Best practice for layer concept with ADF and Databricks
body: <p>I want to build a data warehouse-like layer concept using Azure Data Factory and Databricks, for example Ingestion Layer, Propagation Layer, and Data Mart Layer. The idea was to create separate Databricks scripts for each layers transformations etc. and then orchestrate all of this in ADF pipelines.
  However, the challenge is then how to orchestrate the data loads from/to Databricks for each step, especially handling databricks in-memory data models, and handover to persistent storages for each layer (e.g. to an Azure SQLDB). If I need to load everything again to Databricks for each layer processing, this will result in a lot of I/O overhead and slow processing times. However, if I instead keep everything in Databricks until the final layer for processing, it will be hard to keep track of pipeline errors from within ADF, and to reprocess specific layers.</p>
  <p>I am looking for best practice how to handle a layer concept with ADF and Databricks, Key Design Principles, or similar.
  Thanks in advance!</p>