title: Understanting file distribution and partitioning in HDFS when using Hive
body: <p>On the one hand, in HDFS docs they say:</p>
  
  <blockquote>
    <p>HDFS is designed to support very large files. Applications that are
    compatible with HDFS are those that deal with large data sets. These
    applications write their data only once but they read it one or more
    times and require these reads to be satisfied at streaming speeds.
    HDFS supports write-once-read-many semantics on files. A typical block
    size used by HDFS is 64 MB. Thus, an HDFS file is chopped up into 64
    MB chunks, and if possible, each chunk will reside on a different
    DataNode.</p>
  </blockquote>
  
  <p>Meaning every file will be splitted between nodes.</p>
  
  <p>On the other hand, when I use Hive or Spark SQL, I manage the partitions in such a way that there is a folder for each partition, and all the files inside belong to this partition. For example:</p>
  
  <pre><code>/Sales
     /country=Spain
        /city=Barcelona
           /2019-08-28.parquet
           /2019-08-27.parquet
        /city=Madrid
           /2019-08-28.parquet
           /2019-08-27.parquet
  </code></pre>
  
  <p>Let's say that each file's size is 1GB and the HDFS block size is 128 MB.</p>
  
  <p>So I am confused. I don't understand if <code>city=Barcelonav/2019-08-28.parquet</code> is saved on only one node as a whole (even together with <code>city=Barcelona/2019-08-27.parquet</code>), or each file is distributed between 8 nodes.</p>
  
  <p>If each file is distributed, then what is the benefit of the partitions?</p>
  
  <p>If the data is distributed according to the partitions I define, how does HDFS know to do that? Does it look for folders with a name in the form of <code>key=value</code> and make sure they will be saved intact?</p>