title: Will hadoop be faster than mySQL
body: <p>I am facing a big data problem. I have a large MySQL (Percona) table which joins on itself once a day and produces about 25 billion rows. I am trying to to group together and aggregate all the rows to produce a result. The query is a simple join:</p>
  
  <pre><code>--This query produces about 25 billion rows
  SELECT t1.colA as 'varchar(45)_1', t2.colB as 'varchar(45)_2', count(*)
  FROM table t1
  JOIN
  table t2
  on t1.date = t2.date
  GROUP BY t1.colA, t2.colB
  </code></pre>
  
  <p>The problem is this process takes more than a week to complete. I have started reading about hadoop and wondering if the map reduce feature can improve the amount of time to process the data. I noticed HIVE is a nice little add-on to allow SQL like queries for hadoop. This all looks very promising, but I am facing an issue where I will only be running on a single machine:</p>
  
  <pre><code>6-core i7-4930K
  16GB RAM
  128 SSD
  2TB HDD 
  </code></pre>
  
  <p>When I run the query with MySQL, my resources are barley being used, only about 4Gb of ram and one core is only working 100% while the other are working close to 0%. I checked into this and found MySQL is single threaded. This is also why Hadoop seems to be promising as I noticed it can run multiple mapper functions to better utilize my resources. My question remains is hadoop able to replace MySQL in my situation in which it can produce results within a few hours opposed to over a week even though hadoop will only be running on a single node (although I know it is meant for distributed computing)?</p>