User:Tim/NFS server build

From Van Essen Lab

< User:Tim
Revision as of 00:50, 19 July 2011 by Tim (Talk | contribs)
Jump to: navigation, search

The goal of this project is to create 2 identical capacity NFS servers from standard parts, having roughly 45TB of robust storage each, for a cost of around $5000 each (alternatively, roughly 30TB of storage for around $4000 each). One of the two will be built first to ensure compatibility of parts, and to make any necessary adjustments to the build, following which the other will be built to the final working specifications of the first.


Parts List

51TB configuration with i7-2600k and 16GB RAM for running processing - $5,500 - based on plan for my workstation (idea being that in case of severe problems, my workstation could lend parts temporarily)

For the second NFS server that doesn't need the extra processing power (or if we decide we don't want to have one with processing capability), the processor, cpu cooler, RAM, boot drives, and graphics card can all be downgraded (the motherboard can be downgraded somewhat, but we should stick with Sandy Bridge, due to the amount of bandwidth its northbridge and PCIe 2.0 interfaces have). Additionally, if the servers are in the same room (for instance, using the second ethernet port as a dedicated link for backup), a KVM switch can be used, so monitor, keyboard, and mouse can be removed. All of these together will probably shave around $1000 off the build cost of a single machine.

A higher performance processing machine is possible by using server motherboards (that can seat 2 CPUs) and Xeon processors, but will cost at least an extra thousand dollars. If we just want more RAM, we would need to change CPUs to either the previous generation (still has very nice performance and another 2 physical cores) or to Xeon, Sandy Bridge CPUs don't support more than 4 slots for RAM.


Storage: between 10 and 20 of one of the following:

3TB drives:

Western Digital Caviar Green 3TB - 5400rpm, $140, cheapest 3TB on newegg - appears to come with a minimal 2 port SATA controller card, but likely will not serve our purposes (would need to use up to 6 of them)

Hitachi Deskstar 3TB - 7200rpm, $180, cheapest 7200rpm 3TB on newegg

Hitachi Deskstar 3TB - 5400rpm, $140, $130 on tigerdirect, currently on sale at newegg for $120

Seagate Barracuda XT 3TB - 7200rpm, $200, cheapest Seagate 3TB on newegg




Hitachi Deskstar 2TB - 5400rpm, $80, possible rebate of $10

Seagate Barracuda Green 2TB - 5900rpm, $80

Boot drives: 0 or 2 of one of the following:





Seagate Barracuda 250GB - $37, cheap (slow) hard drive to let the OS boot without the main array operating


SATA Controller Card


1 of the following:

HighPoint RocketRAID mSASx3 PCIe x8 - $350, has hardware raid capability (which we don't plan on using), uses different cables (mSAS, not included, $50, there's also a $22 one out of stock) to connect 4 disks to one port on the card, but more importantly supports 12 disks in a single card, allowing us to use only 1 SATA card, in case having multiple controller cards installed runs into conflicts and keeps us from using more than 1 card at a time



OR 1 or 2 of the following:


AND 1 of the following, if needed:





Lian Li aluminum ATX toolless full tower - $320, toolless case, 12 5.25" bays.

Lian Li aluminum ATX full tower - $290, very similar case, requires tools for assembly, but drive bays are probably easier to add to tooled cases due to their more standard mounts, recommended by Jon S.


Drive bays









My preferred combinations:

4 5 in 3 3.5" bays, 2 2.5" in PCI bays - 20 disks

3 5 in 3 3.5" bays, 2 2.5"/3.5" combo bays, 1 single 3.5" bay - 18 disks

3 5 in 3 3.5" bays, 1 triple 3.5" bay, 1 double 2.5" bay (with 3.5" to 5.25" adapter) - 18 disks

2 2.5"/3.5" combo bays, 3 5 in 3 3.5" bays, 1 optical drive - 17 disks

3 5 in 3 3.5" bays, 1 triple 3.5" bay, 2 2.5" in PCI bays, 1 optical drive - 18 disks, optical drive for cd booting (as opposed to usb booting)

internally mounted 2.5" drives, 4 5 in 3 3.5" bays - 20 disks

10 single 3.5" bays, 2 2.5"/3.5" combo bays - 12 disks, for more room around disks, less chance of multiple disk failure (because fewer disks)

internally mounted 2.5" drives, 12 single 3.5" bays - 12 disks, fewer small (noisy) fans

11 single 3.5" bays, 1 2.5" dual bay (with 3.5" to 5.25" adapter) - 11 disks

10 single 3.5" bays, 1 2.5" dual bay (with 3.5" to 5.25" adapter), 1 optical drive - 10 disks

Power supply

PC Power and Cooling "Silencer" 910W - $150, massive single 74 amp 12V rail (82 amps peak), for simultaneous spinup of large numbers of drives



System configuration



The most robust and high performance configuration would be to have two SSD drives, connected directly to the motherboard, in software raid 1, for booting and operating system (possibly with a third to act as ZFS cache if we use ZFS), with as many 3TB (or 2TB) drives as can fit in the case as a RAID 6 array or ZFS filesystem (two drives from this array can fail without losing any data), with a hot spare (a disk already in the machine, but unused until a disk fails, used immediately to rebuild when a new disk is required to keep the array fully redundant). 10 disks would yield 21TB, 12 would yield 27TB, 18 gives 45TB, and 20 gives 51TB.





Backup plan


Backup between the two servers may be accomplished over a separate connection from the WUSTL network, if desired, and the servers are located in the same lab. The upgrade to gigabit network may make this irrelevant, depending on whether rsync can saturate the network (considering gigabit network clock is only a third of cpu clock, it may not be possible to saturate the network with 1 machine). If they can saturate the network, having a separate connection would leave the machines more responsive to their WUSTL network during backup, especially for operations not involving the main storage array.

Build plan




Miscellaneous Notes






For processing involving very high file IO, especially creating large (hundreds of MB) files and then reading from them, it would be good to write the file on the machine running the script, so that it doesn't need to write things over the network. This is a general rule, not something specific to this project, because even with a gigabit network, the network can saturate before a modern rotational drive (and will saturate long before a single SSD would). As such, if we build a high end computer or two to run intensive tasks on, we may want to set them up with SSD local storage for the IO performance, or a RAID array of decent performance, modestly sized disks, and have processing scripts use it as a scratch directory (under a subdirectory of their process ID, and delete stuff from it when they are done). An even higher performance, but smaller size option would be to make a ramdisk, that is, use a portion of the RAM as a temporary filesystem, but space is severely limited. Alternatively, we could build the primary NFS server with a high end processor and lots of RAM, and run processing locally on it, since the RAID array could have throughput above that of an SSD due to its parallel nature.


Personal tools
Sums Database