title: Custom Delimiters in CSV Data while exporting data from BigQuery to GCS bucket?
body: <p><strong>Background:</strong></p>
  
  <p>I have a GA Premium account. I currently have the following process setup:</p>
  
  <ol>
  <li>The raw data from the GA account flows into BigQuery. </li>
  <li>Query the Bigquery tables.</li>
  <li>Export query results to a GCS bucket. I export it in a CSV and gzipped format.</li>
  <li>Export the CSV gzipped data from the GCS bucket to my Hadoop cluster HDFS.</li>
  <li>Generate hive tables from the data on the cluster by using the comma as the field delimiter.</li>
  </ol>
  
  <p>I run Steps 1-3 programmatically using the BigQuery REST API.</p>
  
  <p><strong>Problem:</strong></p>
  
  <p>My data contains embedded commas and newlines within quotes in some of the fields. When I generate my hive tables, the embedded commas and newlines are causing shifts in my field values for a row or are causing nulls in the records in the hive table.</p>
  
  <p>I want to clean the data by either removing these embedded commas and newlines or by replacing them with custom delimiters within the quotes.</p>
  
  <p>However, the catch is that I would like to do this data cleaning at Step 3 - while exporting to GCS. I looked into the possible query parameters that I can use achieving this but did not find any. The possible parameters that you can use to populate the configuration.extract object are listed at: <a href="https://cloud.google.com/bigquery/docs/reference/v2/jobs#configuration.extract" rel="nofollow">https://cloud.google.com/bigquery/docs/reference/v2/jobs#configuration.extract</a></p>
  
  <p>Here is snippet of the code that does the data export from Bigquery tables to GCS bucket.</p>
  
  <pre><code>query_request = bigquery_service.jobs()
  
  DATASET_NAME = "#######";
  PROJECT_ID = '#####';
  DATASET_ID = 'DestinationTables';
  
  DESTINATION_PATH = 'gs://bucketname/foldername/'
  query_data = {
  'projectId': '#####',
  'configuration': {
    'extract': {
      'sourceTable': {
              'projectId': PROJECT_ID,
              'datasetId': DATASET_ID,
              'tableId': #####,
       },
      'destinationUris': [DESTINATION_PATH + my-files +'-*.gz'],
      'destinationFormat': 'CSV',
      'printHeader': 'false',
      'compression': 'GZIP'
     }
   }
  }
  
  query_response = query_request.insert(projectId=constants.PROJECT_NUMBER,
                                       body=query_data).execute()
  </code></pre>
  
  <p>Thanks in advance.</p>