title: Remove new line from CSV file
body: <p>I want to remove the new line character in CSV file field's data. The same question is asked by multiple people in SO/other places. However the provided solutions are in scripting. I'm looking for a solution in programming languages like PYTHON or in Spark(not only these two) as I have pretty big files.</p>
  
  <p><strong>Previously asked questions on the same topic:</strong></p>
  
  <ul>
  <li><p><a href="https://stackoverflow.com/questions/18619939/remove-new-line-character-from-csv-files-string-column">Remove New Line Character from CSV file&#39;s string column</a></p></li>
  <li><p><a href="https://stackoverflow.com/questions/26406281/replace-new-line-character-between-double-quotes-with-space">Replace new line character between double quotes with space</a></p></li>
  <li><p><a href="https://stackoverflow.com/questions/48861331/remove-new-line-from-csv-files-string-column">Remove New Line from CSV file&#39;s string column</a></p></li>
  <li><p><a href="https://unix.stackexchange.com/questions/222049/how-to-detect-and-remove-newline-character-within-a-column-in-a-csv-file">https://unix.stackexchange.com/questions/222049/how-to-detect-and-remove-newline-character-within-a-column-in-a-csv-file</a></p></li>
  </ul>
  
  <p>I have a CSV file of size ~1GB and want to remove the new line characters in field's data. The schema of the CSV file varies dynamically, so I can't hard code the schema. The line break doesn't always appear before a comma, it appears randomly even within a field.</p>
  
  <p><strong>Sample Data:</strong></p>
  
  <pre><code>playerID,yearID,gameNum,gameName,teamName,lgID,GP,startingPos
  gomezle01,1933,1,Cricket,Team1,NYA,AL,1
  ferreri01,1933,2,Hockey,"This is 
  Team2",BOS,AL,1
  gehrilo01,1933,3,"Game name is 
  Cricket" 
  ,Team3,NYA,AL,1
  gehrich01,1933,4,Hockey,"Here it is 
  Team4",DET,AL,1
  dykesji01,1933,5,"Game name is 
  Hockey"
  ,"Team name 
  Team5",CHA,AL,1
  </code></pre>
  
  <p><strong>Expected Output:</strong></p>
  
  <pre><code>playerID,yearID,gameNum,gameName,teamName,lgID,GP,startingPos
  gomezle01,1933,1,Cricket,Team1,NYA,AL,1
  ferreri01,1933,2,Hockey,"This is Team2",BOS,AL,1
  gehrilo01,1933,3,"Game name is Cricket" ,Team3,NYA,AL,1
  gehrich01,1933,4,Hockey,"Here it is Team4",DET,AL,1
  dykesji01,1933,5,"Game name is Hockey","Team name Team5",CHA,AL,1
  </code></pre>
  
  <p>Newline character can be in any field's data.</p>
  
  <p><strong>Edit:</strong>
  Screenshot as per the code:</p>
  
  <p><a href="https://i.stack.imgur.com/uJ1mz.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/uJ1mz.png" alt="enter image description here"></a></p>