title: How to read CSV in pyspark with "," delimiter but not ", "
body: <p>I am using the following code to read the CSV file in PySpark</p>
  <pre><code>cb_sdf = sqlContext.read.format(&quot;csv&quot;) \
                          .options(header='true', 
                                   multiLine = 'True', 
                                   inferschema='true', 
                                   treatEmptyValuesAsNulls='true') \
                          .load(cb_file)
  </code></pre>
  <p>The number of rows is correct. But for some rows, the columns are separated incorrectly. I think it is because the current delimiter is &quot;,&quot;, but some cells contain &quot;, &quot; in the text as well.</p>
  <p>For example, the following row in the pandas dataframe(I used pd.read_csv to debug)</p>
  <div class="s-table-container">
  <table class="s-table">
  <thead>
  <tr>
  <th style="text-align: left;">Unnamed: 0</th>
  <th style="text-align: center;">name</th>
  <th style="text-align: center;">domain</th>
  <th style="text-align: center;">industry</th>
  <th style="text-align: center;">locality</th>
  <th style="text-align: center;">country</th>
  <th style="text-align: center;">size_range</th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td style="text-align: left;">111</td>
  <td style="text-align: center;">cjsc &quot;transport, customs, tourism&quot;</td>
  <td style="text-align: center;">ttt-w.ru</td>
  <td style="text-align: center;">package/freight delivery</td>
  <td style="text-align: center;">vyborg, leningrad, russia</td>
  <td style="text-align: center;">russia</td>
  <td style="text-align: center;">1 - 10</td>
  </tr>
  </tbody>
  </table>
  </div>
  <p>becomes</p>
  <div class="s-table-container">
  <table class="s-table">
  <thead>
  <tr>
  <th style="text-align: left;">_c0</th>
  <th style="text-align: center;">name</th>
  <th style="text-align: center;">domain</th>
  <th style="text-align: center;">industry</th>
  <th style="text-align: center;">locality</th>
  <th style="text-align: center;">country</th>
  <th style="text-align: center;">size_range</th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td style="text-align: left;">111</td>
  <td style="text-align: center;">&quot;cjsc &quot;&quot;transport</td>
  <td style="text-align: center;">customs</td>
  <td style="text-align: center;">tourism&quot;&quot;&quot;</td>
  <td style="text-align: center;">ttt-w.ru</td>
  <td style="text-align: center;">package/freight delivery</td>
  <td style="text-align: center;">vyborg, leningrad, russia</td>
  </tr>
  </tbody>
  </table>
  </div>
  <p>when I implemented pyspark.</p>
  <p>It seems the cell &quot;cjsc &quot;transport, customs, tourism&quot;&quot; is separated into 3 cells: |&quot;cjsc &quot;&quot;transport| customs|  tourism&quot;&quot;&quot;|.</p>
  <p>How can I set the delimiter to be exactly &quot;,&quot; without any whitespace followed?</p>
  <p><strong>UPDATE:</strong></p>
  <p>I checked the CSV file, the original line is:</p>
  <p><code>111,&quot;cjsc &quot;&quot;transport, customs, tourism&quot;&quot;&quot;,ttt-w.ru,package/freight delivery,&quot;vyborg, leningrad, russia&quot;,russia,1 - 10</code></p>
  <p>So is it still the problem of delimiter, or is it the problem of quotes?</p>