title: pyspark split on delimiter ignoring double quotes using regex
body: <p>I am very new to spark. I am trying to play with rdds. So here is my basic rdd</p>
  
  <pre><code>rdd=sc.parallelize(['"ab,cd",9', 'xyz,6'])
  </code></pre>
  
  <p>Now if I want to split it on commas I do</p>
  
  <pre><code>rdd.map(lambda x:x.split(",")).collect()
  </code></pre>
  
  <p>which gives me</p>
  
  <p><code>[['ab', 'cd', '9'], ['xyz', '6']]</code></p>
  
  <p>Since I want to ignore the commas in between the text placed in "", I write</p>
  
  <pre><code>rdd.map(lambda x:x.split(",(?=([^\\\"]*\\\"[^\\\"]*\\\")*[^\\\"]*$)")).collect()   
  </code></pre>
  
  <p>which gives the output</p>
  
  <p><code>[['ab,cd,9'], ['xyz,6']]</code>
   (Thus this is not a duplicate question)</p>
  
  <p>But I want the output similar to what I get with <code>.split(",")</code> like so</p>
  
  <p><code>[['ab,cd','9'], ['xyz','6']]</code></p>
  
  <p>I am not very good with regex and so I do not know how to manipulate it to get that output.
  Any help will be greatly appreciated</p>