title: How to change a attribute in dataframe column using pyspark which is nested array of struct?
body: <p>While I have found a lot of questions similar to this one, but none of them was exactly solving my problem.
  I have an input JSON as following</p>
  <pre><code>{
      &quot;Header&quot;: {
          &quot;file_schema_version&quot;: &quot;1.0&quot;
      },
      &quot;Records&quot;: [
          {
              &quot;sequence_number&quot;: 1,
              &quot;ID&quot;: &quot;9024147714&quot;
          },
          {
              &quot;sequence_number&quot;: 2,
              &quot;ID&quot;: &quot;9029218142&quot;
          }
      ],
      &quot;Footer&quot;: {
          &quot;record_count&quot;: 2
      }
  }
  </code></pre>
  <p>The output I expect is a new file with updated ID fields as following.</p>
  <pre><code>{
      &quot;Header&quot;: {
          &quot;file_schema_version&quot;: &quot;1.0&quot;
      },
      &quot;Records&quot;: [
          {
              &quot;sequence_number&quot;: 1,
              &quot;ID&quot;: &quot;9024&quot;
          },
          {
              &quot;sequence_number&quot;: 2,
              &quot;ID&quot;: &quot;9029&quot;
          }
      ],
      &quot;Footer&quot;: {
          &quot;record_count&quot;: 2
      }
  }
  </code></pre>
  <p>Records is an array of struct. I have got up to creating a new DataFrame with updated ID. I am not sure how to put it back in the file with Header and Footer intact.</p>
  <pre><code>from pyspark import SparkConf
  from pyspark.sql import SparkSession
  from pyspark.sql import functions as f
  from pyspark.context import SparkContext
  
  spark = SparkSession.builder.master(&quot;local&quot;).config(conf=SparkConf()).getOrCreate()
  sc = SparkContext.getOrCreate()
  
  df = spark.read.json('D:\user\Documents\DATA.json')
  df.printSchema()
  df.show()
  
  records = df.select(f.explode(&quot;Records&quot;).alias(&quot;RecordsExploded&quot;)).select(&quot;RecordsExploded.*&quot;)
  records.printSchema()
  records.show()
  
  records = records.withColumn(&quot;ID&quot;, f.substring(records[&quot;ID&quot;], 1, 4))
  records.show()
  
  </code></pre>
  <p>This is the output that I get from above.</p>
  <pre><code>root
   |-- Footer: struct (nullable = true)
   |    |-- record_count: long (nullable = true)
   |-- Header: struct (nullable = true)
   |    |-- file_schema_version: string (nullable = true)
   |-- Records: array (nullable = true)
   |    |-- element: struct (containsNull = true)
   |    |    |-- ID: string (nullable = true)
   |    |    |-- sequence_number: long (nullable = true)
  
  +------+------+--------------------+
  |Footer|Header|             Records|
  +------+------+--------------------+
  |   [2]| [1.0]|[[9024147714, 1],...|
  +------+------+--------------------+
  
  root
   |-- ID: string (nullable = true)
   |-- sequence_number: long (nullable = true)
  
  +----------+---------------+
  |        ID|sequence_number|
  +----------+---------------+
  |9024147714|              1|
  |9029218142|              2|
  +----------+---------------+
  
  +----+---------------+
  |  ID|sequence_number|
  +----+---------------+
  |9024|              1|
  |9029|              2|
  +----+---------------+
  </code></pre>