Tuesday, September 30, 2014

More on Facebook's "Cold Storage"

So far this year I've attended two talks that were really revelatory; Krste Asanović's keynote at FAST 13, which I blogged about earlier, and Kestutis Patiejunas' talk about Facebook's cold storage systems. Unfortunately, Kestutis' talk was off-the-record, so I couldn't blog about it at the time. But he just gave a shorter version at the Library of Congress' Designing Storage Architectures workshop, so now I can blog about this fascinating and important system. Below the fold, the details.








After the off-the-record talk I was thinking about the synergistic effects that Facebook got from the hard limit the system provides on the total power consumption of the data center. This limit is fixed, the system schedules its activities to stay under the limit. I connected this idea to the Internet Archive's approach to operating their data center in the church in San Francisco without air conditioning. Most of the time, under the fog, San Francisco is quite cool, but there are occasional hot days. So the Internet Archive team built a scheduling system that, when the temperature rises, delays non-essential tasks. Since hot days are rare, these delays do not significantly reduce the system's throughput although they can impact the latency of the non-essential tasks.

The interesting thing about renewable energy sources, such as solar and wind, is that these days the output of these sources can be predicted with considerable accuracy. Suppose Facebook's scheduler could modulate the power limit dynamically, it could match the data center's demand to the power available from the solar panels or wind turbines powering it. This could enable, for example, off-the-grid cold storage data centers in the desert, eliminating some of the possible threats to the data.


Wes Felter said...

Didn't Copan MAID also strictly limit the number of disks that could be simultaneously on?

David. said...

Wes: I don't think so but I could be wrong - do you have a source for this info?

In any case, as far as I know MAID was never deployed at a scale that would enable the synergies that Facebook is harvesting. The point here is that unless you can build an entire data center only for cold storage you miss out on many of the potential savings.

David. said...

Yesterday's "Top Comments" post by lotac at Daily Kos is a worthwhile summary of the issues around cloud storage for consumer use.

The only thing I take serious issue with is this:

"It seems that AWS is losing huge piles of money, while increasing its usage by leaps and bounds, and slashing prices continually. It's hard to filter out exactly what is going on though, as Amazon lumps AWS along with a bunch of other things called "Other" in its accounting statements. Amazon may be following a strategy of increasing market share, taking big losses, in order to squeeze out competitors."

Amazon is making at lot of money from AWS, and especially from S3.

David. said...

A team from Microsoft Research has a paper at OSDI entitled Pelican: A Building Block for Exascale Cold Data Storage describing a prototype implementation of a disk-based cold storage system similar to the upper layer of Facebook's cold storage.

David. said...

I've now posted about Facebook's warm storage, the layer above these two.

Unknown said...

Thanks for sharing cold storage technology of FB. I am interesting in the storage technology of optical disc, but I don't find any articles, do you know any details about it? As we can see in the video, the arm loads 12 pieces of disc in a cartridge and puts it into a disc changer on the top of the rack. Do the 12 drive read/write 12 discs simultaneously to make a large throughput or only one piece of disc in the cartridge will be loaded into a certain drive? If yes, what technology does it use to make it work simultaneously, is it something about blu-ray RAID archive? I am looking forward to your answer.

David. said...

Winston, the answer to your questions is in the post:


The goal is write bandwidth.

David. said...

Chris Mellor at The Register has another post with details of the disk layer of Facebook's cold storage.