title: In pyspark how to convert rdd to json with a different scheme?
body: <p>How to convert the below code to write output json with pyspark DataFrame using, <code>df2.write.format('json')</code></p>
  
  <ol>
  <li>I have an input list (for sake of example only a few items).</li>
  <li>Want to write a json which is more complex/nested than input.</li>
  <li>I tried using <code>rdd.map</code></li>
  <li>Problem: Output contains apostrophes for each object in json.</li>
  <li>I cannot just string replace because data itself might contain it.</li>
  <li>If there is a better way to convert the scheme to the nested json with <code>DataFrame</code> in the example below can you show how? as this might resolve the aposthrophe alltogether.</li>
  </ol>
  
  <p>Here is what I tried:</p>
  
  <pre><code>import json 
  
  rdd = sc.parallelize([(1,2,3),(4,5,6),(7,8,9)])
  df = rdd.toDF(["a","b","c"])
  rddToJson = df.rdd.map(lambda x: json.dumps({"some_top_level_1": {"mycolumn1": x.a}})) // note that result json is complex and more nested than input
  rddToJson.collect()
  </code></pre>
  
  <p>result: contains apostrophes (cannot replace, it can appear anywhere in values) how to do it with proper scheme and dataframe and then df.json.write?</p>
  
  <p>result:</p>
  
  <pre><code>Out[20]: 
  ['{"some_top_level_1": {"mycolumn1": 1}}',
   '{"some_top_level_1": {"mycolumn1": 4}}',
   '{"some_top_level_1": {"mycolumn1": 7}}']
  </code></pre>
  
  <p>My target (unless can be done in another way) is to use df.write.format('json') in order to write the nested/complex json from the above input.</p>
  
  <p>PS: I saw this interesting post: <a href="https://medium.com/@mrpowers/adding-structtype-columns-to-spark-dataframes-b44125409803" rel="nofollow noreferrer">https://medium.com/@mrpowers/adding-structtype-columns-to-spark-dataframes-b44125409803</a> but as i'm a newbie I was not sure how could I transform the input that I have into that nested scheme that I need on output.</p>