<p>This is a really interesting problem —and fundamental to how you implement data analytics at a scale where failures are inevitable.</p>

<h2>Hadoop V1 Algorithm</h2>

<p>HDFS: <code>O(1)</code> to commit a task, resilient to failure in task commit. Job commit is ~<code>O(files)</code> with lots of files; if it fails partway through, output status unknown.</p>

<p>S3: <code>O(data)</code> to commit task, very slow to commit job (<code>O(data)</code> for whole job's output). Lack of atomic rename potentially dangerous.</p>

<h2>Hadoop V2 commit algorithm</h2>

<p>HDFS: <code>O(files)</code> to commit a task, can't handle failure. Job commit is an O(1) <code>touch _SUCCESS</code> call. 
S3: <code>O(data)</code> to commit a task , can't handle failure, and with a longer COPY operation to commit, chance of task commit failure higher.</p>

<p>I don't personally think the failure semantics of the V2 algorithm work; both MapReduce and Spark assume a task which fails during the commit process can be repeated...this does not hold here.</p>

<p>There are some extra details which you don't want to know about, like how the drivers conclude a task has failed, how MapReduce decides that it has partitioned from YARN and so must not commit, but generally it is all down to heartbeats and the assumption that once a task has timed out its not going to resurface. If you are implementing a commit algorithm yourself, make sure that a task committer which has hung until after the entire job has committed <em>will not affect the output</em></p>

<h2>For object stores:</h2>

<ul>
<li>Databricks DBIO. Not seen the code, sounds like they use DynamoDB for the XAs.</li>
<li>IBM Stocator: read the paper, <a href="https://arxiv.org/abs/1709.01812" rel="noreferrer">
Stocator: A High Performance Object Store Connector for Spark</a>. Focus is on minimising HTTP requests and being able to roll back from failed job/task commits.</li>
<li>Hadoop 3.1's S3A Committers, read: <a href="https://github.com/steveloughran/zero-rename-committer/releases/download/tag_draft_003/a_zero_rename_committer.pdf" rel="noreferrer">A Zero Rename Committer</a>. Time to commit task depends on which committer is chosen; at worst time to upload data from VM to S3. Task failures recoverable. Job Commit: one *HTTP POST per file created, parallelizable, so <code>O(files/threads</code>). Failure during job commit not recoverable.</li>
</ul>

<p>To round things off; Azure and google cloud stores do have directory renames, though they are usually O(files), not O(1) —but at least not O(data), like S3. You can safely use the Hadoop V1 committer.</p>
Spark transactional write operation using temporary directories
<p>According to <a href="https://databricks.com/blog/2017/05/31/transactional-writes-cloud-storage.html" rel="nofollow noreferrer">this</a> blog at databricks, spark relies on commit protocol classes from Hadoop so if job is not finished because of some failure output directory does not change(partial output files do not occurs).</p>

<p>So my questions are ;</p>

<p>Does spark prevent partial writes to different storages in case of failures (HDFS,S3 etc)?</p>

<p>Is it possible for a different spark jobs to use same temporary location before final write operation ?</p>

<p>Is it possible for a same spark job which is submitted more than once to use same temporary location ?</p>