title: How to read pipe-separated string into dataframe?
body: <p>I have to read a file into <code>spark</code> (<code>databricks</code>) as <code>bytes</code>, and convert it to a string.</p>
  
  <pre><code>file_bytes.decode("utf-8")
  </code></pre>
  
  <p>This is all fine, and I have my data, as a pipe delimited string, including carriage returns etc. It all looks good.  Something like:</p>
  
  <pre><code>"Column1"|"Column2"|"Column3"|"Column4"|"Column5"
  "This"|"is"|"some"|"data."|
  "Shorter"|"line."|||
  </code></pre>
  
  <p>I want this in a <code>dataframe</code> though so that I can manipulate it, and initially I was attempting to use the following:</p>
  
  <pre><code>df = sqlContext.read.format("csv")
    .option("header", "true")
    .option("inferSchema", "true")
    .option("delimiter", '|')
    .load(???)
  </code></pre>
  
  <p>I appreciate that the <code>load()</code> portion is really meant to be a path to a location on the <code>filesystem</code> ... so have been struggling with that one.</p>
  
  <p>I have therefore reverted to using <code>pandas</code> as it makes life a lot easier:</p>
  
  <pre><code>import io
  import pandas
  
  temp = io.StringIO(file_bytes.decode("utf-8"))
  df = pandas.read_csv(temp, sep="|")
  </code></pre>
  
  <p>This is a <code>pandas</code> <code>dataframe</code>, and not a <code>spark</code> <code>dataframe</code>, which as far as I am aware (and it's a very loose awareness) has pros and cons relating to where it lives (in memory) which relates to scaleability/ cluster-usage etc.</p>
  
  <p>Initially, is there a way for me to get my string into a <code>spark</code> <code>dataframe</code> using <code>sqlContext</code>? Maybe I am missing some parameter or switch etc., or should I just stick with <code>pandas</code>?</p>
  
  <p>The main thing I am worried about is that right now files are quite small (200 kb or so), but they might not be forever, and I'd like to reuse a pattern that will allow me to work with larger things (which is why I am marginally concerned about using pandas).</p>