Take the 2-minute tour ×

I was wondering if there are any good solutions for recording data in a NoSQL database and then converting them over to an RDBMS?

For example, if you wanted to capture some data quickly, like session logs, but then you want to be able to create reports on them later.

My favorite database is Postgres, so if your answer is relevant in Postgres that would be great.

share|improve this question
add comment

2 Answers

up vote 9 down vote accepted

The Quick Answer - Yes. Happens all the time.

There are plenty of good solutions. What solutions are already in your environment? I am helping one client that takes their web site/session activity information from their web application, they write it to xml then deserialize that xml into Hadoop. They then use Hive on top of Hadoop to create aggregations and facts (by joining this log data to reference data from their website that they store in a Postgres database). These aggregtions and facts are then loaded into an RDBMS (they use SQL Server, because of the rich analytics provided in the SQL Server stack).

I would explore Hadoop, look at Hive which is a query engine on top of Hadoop that allows you to write declarative queries that closely resemble SQL and get translated into MapReduce jobs behind the scenes. Then look at Sqoop which is another tool in the Hadoop family that assists you with exporting/importing data.

http://hadoop.apache.org/ http://www.amazon.com/Hadoop-Definitive-Guide-Tom-White/dp/1449389732/ref=sr_1_1?s=books&ie=UTF8&qid=1319160463&sr=1-1 http://www.cloudera.com/blog/2009/06/introducing-sqoop/ http://hive.apache.org/

There are other options out there too...

share|improve this answer
Well that is an extremely complete answer. Doesn't sound too simple, but sounds like it uses a lot of cool technologies. –  Joe Oct 21 '11 at 15:47
Well it isn't "simple" in terms of the number of pieces, etc. but it actually really isn't that complex once you get going. What tool you select for the pieces depends on what you have in your environment and what you are comfortable with. Take a look at those links, check out that book recommendation link. –  Mike Walsh Oct 22 '11 at 2:20
add comment

Not all NoSQL are the same. For instance MongoDB has very good query capabilities. it is very easy to retrieve data, sort data in a manner that reminds you of SQL. So what I am saying is, that you shouldn't be thinking - SQL for reporting, unless you have off-the-shelf tools that require SQL compliant data, of course. Some of the other NoSQL are a bit more involved as far as getting SQL-like results from your store, CouchDB for instance, is not trivial to define a simple where-clause type query. MongodDB - I have used the Python driver - is a cinch. So I am finding it less important to worry about porting my data to a relational store.

share|improve this answer
Do you have any link or affiliation to MongoDB? Just joined, four answers, all very positive about MongoDB that sound like marketing. You may be correct, but it's poor form if you have an interest... –  gbn Dec 15 '11 at 12:37
Apparently you scared him off there GBN –  jcolebrand Feb 21 '12 at 16:43
add comment

Your Answer


