Tuesday, May 3, 2011

Time for a SQL/NoSQL Group Hug

European Open Database Camp 2011 is this weekend in the hills above Cagliari, Sardinia.  In honor of the increasing number sites that use both NoSQL and SQL databases, I am going to be running a MySQL to NoSQL Hackathon to prototype Tungsten Replicator support for transferring data from MySQL to MongoDB.  The conference will have at least one well-informed MongoDB expert, so we should have enough critical mass to get this done.  It helps that I'll be completely jet-lagged after flying in from the US and unable to sleep anyway.  

Over the past year SQL vs. NoSQL rants have started to abate as people get down to the practical work to make both types of systems work most effectively, often within the same site if not the same application.   I have spoken to a small but growing number of users who want to move data between MySQL and Cassandra, HBase, MongoDB, and others.  (Here's a typical example from StackOverflow.)  Some of them are even willing to pay money for Tungsten Replicator implementations.  You know there is a real need when that happens.  I expect such requests will grow as more applications implement NoSQL stores and have to set up feeds to or from SQL.   Heterogeneous replication in this sense is a good proxy for solution maturity, and at least a couple of the NoSQL stores seem to be getting there.

The MySQL to MongoDB hackathon is our first crack at this problem.  There is no guarantee the results will be especially usable, because Tungsten will not have a built-in solution for SQL to JSON mapping.  (I'm still noodling about that problem--suggestions welcome.)  Still, we will start to understand the problem space better and pave the way for more robust solutions over the course of the next year or so.  I'm looking forward to learning more a lot more about non-SQL stores along the way. 

If you are attending the conference you can hear about our results in person.  You can also tune into the #tungsten IRC channel at irc.freenode.net starting Friday May 5th around 10pm GMT.  I'll post regular updates over the weekend.  Feel free to drop by on IRC to see how we are doing or even to help out.  We will not single-handedly end the SQL/NoSQL wars, but at least we can help them learn how to share. 


Baron said...

Anticipate the demand and write the code now before it's too late: write tools to export data from all the NoSQL databases into MySQL and PostgreSQL :-)

(Sorry, I just couldn't resist the opportunity to be completely immature!)

Robert Hodges said...

The thought has crossed my mind. However, I think it would be better to support real-time replication from the winners in the NoSQL race (should be 2-3 I would guess) back to SQL for applications like reporting, billing & accounting, etc.

Scaling Databases Using Commodity Hardware and Shared-Nothing Design