title: Convert a column from StringType to Json (object)
body: <p>Here is a sample data</p>
  <pre><code>val df4 = sc.parallelize(List(
    (&quot;A1&quot;,45, &quot;5&quot;, 1, 90),
    (&quot;A2&quot;,60, &quot;1&quot;, 1, 120),
    (&quot;A6&quot;, 30, &quot;9&quot;, 1, 450),
    (&quot;A7&quot;, 89, &quot;7&quot;, 1, 333),
    (&quot;A7&quot;, 89, &quot;4&quot;, 1, 320),
    (&quot;A2&quot;,60, &quot;5&quot;, 1, 22),
    (&quot;A1&quot;,45, &quot;22&quot;, 1, 1)
  )).toDF(&quot;CID&quot;,&quot;age&quot;, &quot;children&quot;, &quot;marketplace_id&quot;,&quot;value&quot;)
  </code></pre>
  <p>thanks to @Shu for this piece of code</p>
  <pre><code>val df5 = df4.selectExpr(&quot;CID&quot;,&quot;&quot;&quot;to_json(named_struct(&quot;id&quot;, children)) as item&quot;&quot;&quot;, &quot;value&quot;, &quot;marketplace_id&quot;)
  </code></pre>
  <pre><code>+---+-----------+-----+--------------+
  |CID|item       |value|marketplace_id|
  +---+-----------+-----+--------------+
  |A1 |{&quot;id&quot;:&quot;5&quot;} |90   |1             |
  |A2 |{&quot;id&quot;:&quot;1&quot;} |120  |1             |
  |A6 |{&quot;id&quot;:&quot;9&quot;} |450  |1             |
  |A7 |{&quot;id&quot;:&quot;7&quot;} |333  |1             |
  |A7 |{&quot;id&quot;:&quot;4&quot;} |320  |1             |
  |A2 |{&quot;id&quot;:&quot;5&quot;} |22   |1             |
  |A1 |{&quot;id&quot;:&quot;22&quot;}|1    |1             |
  +---+-----------+-----+--------------+
  </code></pre>
  <p>when you do <code>df5.dtypes</code></p>
  <pre><code>(CID,StringType), (item,StringType), (value,IntegerType), (marketplace_id,IntegerType)
  </code></pre>
  <p>the column item is of string type, is there a way this can be of json/object type(if that is a thing)?</p>
  <p>EDIT 1:
  I will describe what I am trying to achieve here, the above two steps remains same.</p>
  <pre><code>val w = Window.partitionBy(&quot;CID&quot;).orderBy(desc(&quot;value&quot;))
  val sorted_list = df5.withColumn(&quot;item&quot;, collect_list(&quot;item&quot;).over(w)).groupBy(&quot;CID&quot;).agg(max(&quot;item&quot;) as &quot;item&quot;)
  </code></pre>
  <p>Output:</p>
  <pre><code>+---+-------------------------+
  |CID|item                     |
  +---+-------------------------+
  |A6 |[{&quot;id&quot;:&quot;9&quot;}]             |
  |A2 |[{&quot;id&quot;:&quot;1&quot;}, {&quot;id&quot;:&quot;5&quot;}] |
  |A7 |[{&quot;id&quot;:&quot;7&quot;}, {&quot;id&quot;:&quot;4&quot;}] |
  |A1 |[{&quot;id&quot;:&quot;5&quot;}, {&quot;id&quot;:&quot;22&quot;}]|
  +---+-------------------------+
  </code></pre>
  <p>now whatever is inside <code>[ ]</code> is a string. which is causing a problem for one of the tools we are using.</p>
  <p>Sorry, pardon me I am new to scala, spark if this is a basic question.</p>