title: Efficient reading nested parquet column in Spark
body: <p>I have following (simplified) schema:</p>
  
  <pre><code>root
   |-- event: struct (nullable = true)
   |    |-- spent: struct (nullable = true)
   |    |    |-- amount: decimal(34,3) (nullable = true)
   |    |    |-- currency: string (nullable = true)
   |    |
   |    | ... ~ 20 other struct fields on "event" level
  </code></pre>
  
  <p>I'm trying to sum on nested field</p>
  
  <pre><code>spark.sql("select sum(event.spent.amount) from event")
  </code></pre>
  
  <p>According to spark metrics I'm reading 18 GB from disk and it takes 2.5 min.</p>
  
  <p>However when I select the top level field:</p>
  
  <pre><code> spark.sql("select sum(amount) from event")
  </code></pre>
  
  <p>I read only 2GB in 4 seconds.</p>
  
  <p>From the physical plan I can see that in case of nested structure the <strong>whole event</strong> struct with <strong>all fields</strong> are read from parquet, which is a waste. </p>
  
  <p>Parquet format should be able to provide the desired column from nested structure without reading it all (which is the point of columnar store). Is there some way to do this efficiently in Spark ?</p>