title: PySpark parse csv with field having JSON string value
body: <p>I have a csv file with a row dictionary/json like format values where it is using comma to within a one string value and also field delimiter.</p>
  <pre><code>&quot;{&quot;&quot;ImportId&quot;&quot;:&quot;&quot;loc1&quot;&quot;,&quot;&quot;place&quot;&quot;:&quot;&quot;India/Mumbai&quot;&quot;}&quot;, &quot;{&quot;&quot;ImportId&quot;&quot;:&quot;&quot;loc2&quot;&quot;,&quot;&quot;place&quot;&quot;:&quot;&quot;America/Denver&quot;&quot;}&quot;, &quot;{&quot;&quot;ImportId&quot;&quot;:&quot;&quot;status&quot;&quot;}&quot;, &quot;{&quot;&quot;ImportId&quot;&quot;:&quot;&quot;ipAddress&quot;&quot;}&quot;, &quot;{&quot;&quot;ImportId&quot;&quot;:&quot;&quot;progress&quot;&quot;}&quot;, &quot;{&quot;&quot;ImportId&quot;&quot;:&quot;&quot;duration&quot;&quot;}&quot;, &quot;{&quot;&quot;ImportId&quot;&quot;:&quot;&quot;finished&quot;&quot;}&quot;, &quot;{&quot;&quot;ImportId&quot;&quot;:&quot;&quot;recDate&quot;&quot;,&quot;&quot;place&quot;&quot;:&quot;&quot;America/Denver&quot;&quot;}&quot;, &quot;{&quot;&quot;ImportId&quot;&quot;:&quot;&quot;_recordId&quot;&quot;}&quot;, &quot;{&quot;&quot;ImportId&quot;&quot;:&quot;&quot;recipientName&quot;&quot;}&quot;
  </code></pre>
  <p>I have tried following options with read method in spark:</p>
  <pre><code>spark.read.option('header','false').option(&quot;multiLine&quot;, 'true').option('delimiter', ',') \
  .option('quotes', '\&quot;').option('escape', '\&quot;').option('escape', '\&quot;&quot;,&quot;&quot;') \
  .option('escape', '\n').option('charset', 'UTF-8').option('doublequote', 'false') \
  .option('ignoreLeadingWhiteSpace', 'true') \
  .option('ignoreTrailingWhiteSpace', 'true').option('inferSchema', 'false').csv(file_path)
  </code></pre>
  <p>But while reading the csv file, spark is breaking down value like <code>&quot;{&quot;&quot;ImportId&quot;&quot;:&quot;&quot;loc1&quot;&quot;,&quot;&quot;place&quot;&quot;:&quot;&quot;India/Mumbai&quot;&quot;}&quot;</code> into two separate columns <code>{&quot;&quot;ImportId&quot;&quot;:&quot;&quot;loc1&quot;&quot;</code> and <code>&quot;&quot;place&quot;&quot;:&quot;&quot;India/Mumbai&quot;&quot;}&quot;</code> but it should be treated a one value.</p>
  <p>P.S. I have also tested it with Pandas and it was effectively parsing the values without breaking down the inner values of &quot;{}&quot;. But I have to only use spark in my project.</p>