title: How to separate a UTF-8 file into separate rows (comma delimited) using python?
body: <p>I am trying to convert a UTF-16 file to a UTF-8 file (as I am using the python csv module which apparently does not handle UTF-16 files). I then want to delimit this UTF-8 file so I can import it into a postgres table using a simple row.strip() method. This python file looks like:</p>
  
  <pre><code>with codecs.open(sourcefile, 'rU', 'UTF-16') as infile:
     with open(sourcefile + '.utf8', 'wb') as outfile:
         for line in infile:
             outfile.write(line.encode('utf8'))
  
  
  with open(sourcefile + '.utf8', 'rb') as f:
      reader = csv.reader(f, delimiter=',')
  
      for row in reader:
          print row[1]
  </code></pre>
  
  <p>I am unable to separate the rows though as row appears to only have one index in it and printing row[1] gives me in an index range - how do I split this file?</p>
  
  <p>Excel row:</p>
  
  <pre><code>15,"1/2 TYPE A","98","MCDS, TX","XA","852","TX","955","148","HAPPY, TX",,"$0.00","0","0.00","$1,504","179","0.00%","100.00%","0"
  32,"1/2 TYPE B","98","MCDS, MI","XA","252","MI","72","925","HAPPY, MI",,"$0.00","0","0.00","$2,504","225","0.00%","100.00%","0"
  </code></pre>
  
  <p>My apologies for not being very descriptive. Basically the input file is a UTF-16 file. I used to open the file with excel, separate the one column into many columns with delimiter ',' and save it as a csv file. I then ran that manipulated csv file through a python script that was able to read a csv file and strip the rows and import the data into a postgres database. </p>
  
  <p>The original import part of the python script (when I separate by delimiter ',') looks like this (simplified version):</p>
  
  <pre><code> for row in reader:
      arg = {
              'item_number': row[0].strip(),
              'item_size': row[1].strip(),
              'description': row[2].strip(),
              #etc...
          }
          cur.execute(
              """INSERT INTO 
              "Sales"("ITEM_NUMBER","ITEM_SIZE","DESCRIPTION"")
               select
                  %(item_number)s, 
                  %(item_size)s )
                  %(description);""", arg)
  </code></pre>
  
  <p>However, I now want to be able to simply run the UTF-16 file with my python script to import the data into postgres so I do not have to open the file at all in excel. I would like to do this by transforming the file into a UTF-8 file then somehow stripping each row and importing it into my database. </p>
  
  <p>I have been able to successfully transform the file into UTF-8 but now I have the issue where the UTF-8 file is essentially a bunch of lines that are seen as "one column". How would I go about stripping each row? I can not do a simple row[0].strip() as there are some commas in the files that belong to descriptions.</p>