title: Write Hive Table using Spark SQL and JDBC
body: <p>I am new to Hadoop and I am using a single node cluster (for development) to pull some data from a relational database.</p>
  
  <p>Specifically, I am using Spark (version 1.4.1), Java API, to pull data for a query and write to Hive.  I have run into various problems (and have read the manuals and tried searching online) but I think I might be misunderstanding some fundamental part of this because I am having problems.</p>
  
  <p>First, I thought I'd be able to read data into Spark, optionally run some Spark methods to manipulate the data and then write it to Hive through a HiveContext object.  But, there doesn't seem to be any way to write straight from Spark to Hive. Is that true?</p>
  
  <p>So I need an intermediate step.  I have tried a few different methods of storing the data first before writing to Hive and settled on writing an HDFS text file since it seemed to work best for me.  However, writing the HDFS file, I get square brackets in the files, like this:  [A,B,C]</p>
  
  <p>So, when I load the data into Hive using the "LOAD DATA INPATH..." HiveQL statement, I get the square brackets in the Hive table!!  </p>
  
  <p>What am I missing?  Or more appropriately, can someone please help me understand the steps I need to do to:</p>
  
  <ul>
  <li>Run a SQL on SQL Server or Oracle DB</li>
  <li>Write the data out to a Hive table that can be accessed by a dashboard tool.</li>
  </ul>
  
  <p>My code right now, looks something like this:</p>
  
  <pre><code>    DataFrame df= sqlContext.read().format("jdbc").options(getSqlContextOptions(driver, dburl, query)).load(); // This step seem to work fine.
  JavaRDD&lt;Row&gt; rdd = df.javaRDD();
  rdd.saveAsTextFile(getHdfsUri() + pathToFile);  // This works, but writes the rows in square brackets, like:  [1, AAA].
  hiveContext.sql("CREATE TABLE BLAH (MY_ID INT, MY_DESC STRING) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' LINES TERMINATED BY '\n' STORED AS TEXTFILE");
  hiveContext.sql("LOAD DATA INPATH '" + getHdfsUri() + hdfsFile + "' OVERWRITE INTO TABLE `BLAH`");  // Get's written like:  
  
  MY_INT  MY_DESC
  ------  -------
           AAA]
  </code></pre>
  
  <p>The INT column doesn't get written at all because the leading [ makes it no longer a numeric value and the last column shows the "]" at the end of the row in the HDFS file.</p>
  
  <p>Please help me understand why this isn't working or what a better way would be.  Thanks!</p>
  
  <p>I am not locked into any specific approach, so all options would be appreciated.</p>