title: Does Spark lock the File while writing to HDFS or S3
body: <p>I have an S3 location with the below directory structure with a Hive table created on top of it:</p>
  
  <pre><code>s3://&lt;Mybucket&gt;/&lt;Table Name&gt;/&lt;day Partition&gt;
  </code></pre>
  
  <p>Let's say I have a Spark program which writes data into above table location spanning multiple partitions using the below line of code:</p>
  
  <pre><code>Df.write.partitionBy("orderdate").parquet("s3://&lt;Mybucket&gt;/&lt;Table Name&gt;/")
  </code></pre>
  
  <p>If another program such as "Hive SQL query" or "AWS Athena Query" started reading data from the table at the same time:</p>
  
  <p>Do they consider temporary files being written?</p>
  
  <p>Does spark lock the data file while writing into S3 location?</p>
  
  <p>How can we handle such concurrency situations using Spark as an ETL tool?</p>