title: How to cast string to ArrayType of dictionary (JSON) in PySpark
body: <p>Trying to cast StringType to ArrayType of JSON for a dataframe generated form CSV.</p>
  
  <p>Using <code>pyspark</code> on <code>Spark2</code></p>
  
  <p>The CSV file I am dealing with; is as follows -</p>
  
  <pre><code>date,attribute2,count,attribute3
  2017-09-03,'attribute1_value1',2,'[{"key":"value","key2":2},{"key":"value","key2":2},{"key":"value","key2":2}]'
  2017-09-04,'attribute1_value2',2,'[{"key":"value","key2":20},{"key":"value","key2":25},{"key":"value","key2":27}]'
  </code></pre>
  
  <p>As shown above, it contains one attribute <code>"attribute3"</code> in literal string, which is technically a list of dictionary(JSON) with exact length of 2. 
  (This is the output of function distinct)</p>
  
  <p>Snippet from the <code>printSchema()</code></p>
  
  <pre><code>attribute3: string (nullable = true)
  </code></pre>
  
  <p>I am trying to cast the <code>"attribute3"</code> to <code>ArrayType</code> as follows</p>
  
  <pre><code>temp = dataframe.withColumn(
      "attribute3_modified",
      dataframe["attribute3"].cast(ArrayType())
  )
  </code></pre>
  
  <blockquote>
  <pre><code>Traceback (most recent call last):
    File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  TypeError: __init__() takes at least 2 arguments (1 given)
  </code></pre>
  </blockquote>
  
  <p>Indeed, <code>ArrayType</code> expects datatype as argument. I tried with <code>"json"</code>, but it did not work. </p>
  
  <p>Desired Output -
  In the end, I need to convert <code>attribute3</code> to <code>ArrayType()</code> or plain simple Python list. (I am trying to avoid use of <code>eval</code>)</p>
  
  <p>How do I convert it to <code>ArrayType</code>, so that I can treat it as list of JSONs?</p>
  
  <p>Am I missing anything here? </p>
  
  <p>(The <a href="https://spark.apache.org/docs/latest/api/python/pyspark.sql.html" rel="nofollow noreferrer">documentation</a>,does not address this problem in straightforward way)</p>