title: Spark df not selecting nested fields as column names
body: <p>I am trying to create a Spark df which contains both top level and nested fields from a list of dictionaries containing key and values which correspond to key and value of a json object and I am having issues with selecting nested columns.</p>
  <p>Here is what I have so far:</p>
  <p>Input is a list of dictionaries containing JSON values:</p>
  <pre><code>[{
    &quot;uid&quot;: 98763,
    &quot;estimatedGrade&quot;: {
      &quot;science&quot;: 10.03,
      &quot;english&quot;: 20.5,
     },
    &quot;actualGrade&quot;:  {
      &quot;science&quot;: 10.03,
      &quot;english&quot;: 20.5,
     }
  }]
  
  printed schema:
   |-- uid: long (nullable = true)
   |-- actualGrade: struct (nullable = true)
   |    |-- science: double(nullable = true)
   |    |-- english: double (nullable = true)
   |-- estimatedGrade: struct (nullable = true)
   |    |-- science: double(nullable = true)
   |    |-- english: double (nullable = true)
  </code></pre>
  <p>desired output:</p>
  <div class="s-table-container">
  <table class="s-table">
  <thead>
  <tr>
  <th style="text-align: left;">uid</th>
  <th style="text-align: center;">estm.science</th>
  <th style="text-align: center;">estm.english</th>
  <th style="text-align: center;">act.science</th>
  <th style="text-align: center;">estGrade.english</th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td style="text-align: left;">val</td>
  <td style="text-align: center;">val</td>
  <td style="text-align: center;">val</td>
  <td style="text-align: center;">val</td>
  <td style="text-align: center;">val</td>
  </tr>
  </tbody>
  </table>
  </div>
  <p>*please note I don't need to renamed the column but had to shorten them to fit in one line</p>
  <p>This is my code so far:</p>
  <pre><code>
      #jsons contains list of dict with the json key/values
      df = self._spark.sparkContext.parallelize(jsons).map(lambda x: json.dumps(x))
      df = self._spark.read.json(df, multiLine=True)
      
      logger.info(&quot;Df count: %s&quot;, df.count())
      logger.info(&quot;Df table schema: %s&quot;, df.printSchema())
      
      columns = ['uid', 'estimatedGrade.science', 'estimatedGrade.english', 'actualGrade.science', 'actualGrade.english']
      
      df.select([column_header for column_header in df.columns if column_header in columns])
  
  </code></pre>
  <p>I am able to select only uid which is top level field so I'm guessing I'm doing something wrong with selecting nested value.</p>
  <p>Please help.</p>