title: Pyspark: Convert datetime spark fields during read of JSON
body: <p>I have some JSON files with a very long and multi-level schemas like</p>
  <pre><code>{ &quot;a&quot;: {&quot;b&quot;: {&quot;c&quot;: [1, 2], &quot;d&quot;: 3, &quot;e&quot;: {...}}}}
  </code></pre>
  <p>But with hundreds of fields inside of each other. I do have a description of schema, but my problem is that datetime fields are stored like</p>
  <pre><code>&quot;x&quot;: {&quot;__encode__&quot;: &quot;datetime&quot;, &quot;value&quot;: &quot;12314342.123&quot;}
  </code></pre>
  <p>Since all of the datetime fields are scattered across the schema, is there an easier solution for converting them all into <code>TimestampType</code> instead of having to iterate trough them and add run an UDF to convert to that?</p>
  <p>Basically I would like to describe my schema with a UDF builtin, so it converts the field during the read time.</p>
  <pre><code>my_schema = StructType([
      StructField(&quot;a&quot;, StructType([...])),
      ...
      StructField(&quot;x&quot;, StructType([   # &lt;-- I would like to pass a transformation function somehow
         StructField(&quot;__encode__&quot;, StringType()),
         StructField(&quot;value&quot;, DoubleType()),
      ])
  ])
  df = spark.read.schema(my_schema).json(&quot;foo.json&quot;, mode=&quot;FAILFAST&quot;)
  </code></pre>
  <hr />
  <p><strong>Edit:</strong> I found issues with the iterative approach of editing fields that</p>
  <pre><code>df.withcolumn('foo', to_timestamp(from_unixtime('foo.value')))
  </code></pre>
  <p>will only work with top level columns and to edit inner struct columns I have to recreate the whole struct and that is complicated with the several levels and I have also arrays in the middle.</p>
  <p>I found: <a href="https://stackoverflow.com/q/48777993/754991">this</a> and <a href="https://stackoverflow.com/a/48448731/754991">this</a></p>
  <p>But they only care about a single level and not an arbitrarily nested struct. So seems that I need to reproduce all that schema with fields replaced?</p>
  <p>Isn't there any easier way?</p>
  <hr />
  <p><strong>Sample:</strong> I want to convert from <a href="https://pastebin.com/bM0yNsBn" rel="nofollow noreferrer">this schema</a> to <a href="https://pastebin.com/QPtS2xcA" rel="nofollow noreferrer">that schema</a> -- Specially the nested fields: Fields TimeA, TimeB, TimeC and TimeD</p>
  <p>I am extracting the fields using the metadata attribute with this code: <a href="https://pastebin.com/E7H7GV9A" rel="nofollow noreferrer">https://pastebin.com/E7H7GV9A</a></p>