

Cassandra Usurping MySQL on Twitter

  • submit to reddit

King says that Twitter needs a system that can scale as quickly as the website is growing.  The number of tweets per day rose from 2 million to over 50 million in 2009 alone.  The MySQL + Memcached system just isn't cutting it, King said.  The costs of operation are getting too high and a more automated, highly available, and scalable data system is needed, King said.  

King's team tested several other solutions before arriving at Cassandra.  One option was a more automated sharded MySQL model.  They also tested the best-known open source NoSQL databases including HBase, Project Voldemort, MongoDB, MemcacheDB, Redis, Cassandra, and HyperTable.  King looked for low requirements, a healthy open source community, scalability, and any single points of failure.  Cassandra, notably, has no single point of failure.  The system also had highly scalable writes for Twitter's extremely variable write traffic.  Finally, King was impressed by the Cassandra community, which is now encapsulated as a Top Level Project at Apache.

Twitter is known for adopting new technologies if they can dramatically help improve their site's architecture.  The company rewrote some of its backend code in Scala not too long ago.  Now Twitter plans to completely replace the current solution with Cassandra over the coming months.  King says they are currently moving their largest table (the statuses table, which includes tweets and retweets) over to Cassandra architecture.  King explains how Twitter is making this transition:


So to roll out the new data store we do this:

  3. Find a bug :)
  4. Turn the feature off
  5. Fix the bug and deploy
  6. GOTO #2


  1. Take a backup from the mysql databases

--Excerpt from MyNoSQL

Cassandra makes a lot of sense for Twitter and other large sites that deal with tremendous amounts of data.  Facebook, the biggest social networking site in existence, was the architect of Cassandra, so it's no surprise that other social networking sites, like DIgg, find it to be extremely helpful.  With Facebook and now Twitter using the Apache Cassandra system, the project, and the NoSQL movement, have been given the the seal of approval by the two great pillars of social networking.  

For more information on Cassandra and the NoSQL databases, check out this article.