Take the 2-minute tour ×

I'd like to set up a software raid 10 array (4x2TB) for a workstation using a fresh install, but I'm finding non applicable/conflicting/old resources online. I'd like to get the community's advice on setting up such an array as there appears to be a plethora of possible configurations.

Specifically, I'll be using the workstation for image analysis (i.e., typical file sizes from several hundred MB-several GB), so if there are optimizations that can be made for such purposes that would be great.


share|improve this question
add comment

3 Answers

up vote 1 down vote accepted

I don't remember exactly what resources I followed when setting up my RAID on my server, but I think this article was the main point of information. Some important things:

  • Use mdadm and not dmraid.
  • Use /dev/disk/by-id/ paths to point at the disks, instead of /dev/sda etc… It's easier to map them to the physical devices, in case you need to replace a disk or such.
  • Be patient. At first I thought my RAID would be ready after the 5 hours initial setup time. Then it took another day to rebuild itself and actually be ready. (4x2TB)
  • Use a separate disk/partition/RAID for primary OS installs. It's easier to manage everything if you keep the large data RAID separate from primary OS and its data. Much easier to recover a small OS disk, than rebuild a huge multi-terabyte array, if something goes bad on the OS side.
share|improve this answer
Not entirely, you still need to identify your drives in advance, preferably by serial number, so when one is reported failed it's easy to remove just that one. I'll take a label and copy the serial # to the cable end of the drive. The by-id name usually includes the serial number. –  ppetraki Dec 14 '12 at 14:21
@ppetraki Ok, fair enough. It would save you some time in mapping the physical disk visually to the name given. However, are you sure this actually works like this? I seems not to work for me when doing auto-assembling by array-UUID. It reports plain names in the array configuration (e.g. sda1) no matter what the original paths to disks were on the time of creation. –  gertvdijk Dec 14 '12 at 14:30
You don't need to use the by-id names in assembling the array, but does help you organize your plan, especially if you have tons of disks, they're just symlinks to SD devices. MD just scans for metadata, it doesn't care about disk names, neither does the admin tool, which is unfortunate as now I have to spend the extra time to lookup sda => scsi-SATA_TOSHIBA_THNS128_10US10BTT02Z, which is a lot more meaningful. It would be a great feature to add I just haven't got around to it. –  ppetraki Dec 14 '12 at 14:35
@dobey - so far, that's the best description for a RAID 10 setup I've seen. Thx. Pity that it's dated 2008. –  Prophet60091 Dec 14 '12 at 14:48
@Prophet60091 Regarding the mdadm commands, nothing has really changed. The Grub part is a bit outdated, though. –  gertvdijk Dec 14 '12 at 14:49
show 6 more comments

With RAID10 in the given situation I see only two variables candidate for optimization:

  • Chunk size

    Set it to something larger than the default of 512KiB to minimize the overhead for linear reads/writes of large files. You should try it on a small partition on your machine to see what gives the best performance, e.g. 1MB, 2MB, 5MB, 10MB...

  • Near vs Far layout

    comparable to RAID1+0 vs RAID0+1. Far is a bit faster as the performance for reading is more like RAID0. Yet a Near layout is the default because it has a slightly higher chance of surviving the unlikely event that all mirrored disks are broken (some probability math here). Some more visual idea of the differens is below, happily stolen from SLES mdadm documentation:

    Near looks like

    sda1 sdb1 sdc1 sdd1
      0    0    1    1
      2    2    3    3
      4    4    5    5

    Far looks like

    sda1 sdb1 sdc1 sdd1
      0    1    2    3
      4    5    6    7       
      . . .
      3    0    1    2
      7    4    5    6

Update about far vs near redundancy from the discussion in the comments. Suppose sda fails:

sda1 sdb1 sdc1 sdd1
  -    0    1    1
  -    2    3    3
  -    4    5    5

then sdc or sdd can still fail, while in far:

sda1 sdb1 sdc1 sdd1
  -    1    2    3
  -    5    6    7       
  . . .
  -    0    1    2
  -    4    5    6

now only sdc can fail, as a failed sdb drive make block 4 inaccessible and a failed sdd drive will make block 3 inaccessible.

Conclusion: chances of surviving a 2-disk failure are higher when using a near layout. (can someone do the math here for a quantitative number?)

share|improve this answer
thx for that link. I like the distinction made between "Complex vs. Nested RAID 10". That particular nuance was never clear to me. –  Prophet60091 Dec 14 '12 at 14:46
far doesn't have any better chance of survival ( data is on the same disk, just different location ), it is just optimized for reading rather than writing. There is also the offset layout, which is more of a balance between the two. –  psusi Dec 14 '12 at 15:28
@psusi Suppose one drive has failed in a 4-disk RAID10. Then only one out of the working three could fail without blowing up the array in a far-layout, whereas two out of three can fail in near-layout. At least this is what I understood a year ago when I set up my RAID10 array. I am unable to find that source again. –  gertvdijk Dec 14 '12 at 15:34
@gertvdijk, no.. once again, the backup copy is on the same drive with far layout, just all towards the end of the drive rather than mixed throughout. –  psusi Dec 14 '12 at 23:54
@psusi No, that's exactly making the difference! Suppose sda fails. In near, then sdc or sdd can fail, but not sdb. In far, only sdc can fail because sdb can't fail (missing block 4) and sdd can't either (missing block 3). –  gertvdijk Dec 15 '12 at 0:01
show 2 more comments

Picking up some hotspares in advance would be a good idea. Also taking these notes into account.

Recommended storage scheme for home server? (LVM/JBOD/RAID 5...)

See footnote [1] in above link to see what happens with cheap storage when you need it the most.

This is all a moot point however until you profile how your the target application actually uses storage. You might find that parallelism is possible, so one block can be used for reading results, and one for writing them. This could be further abstracted behind a RAID0 (until the HBA reports QUEUE_FULL) with the results backed up via rsync.

It really depends, saying "I'm doing image analysis" without defining the workload or level of service just isn't enough; even if you did, that level of performance analysis is real work, I know it's something "I" wouldn't do in my spare time. My intentions are to get you thinking about your application to create your own solutions. Spindles are always the slowest part of your system, plan accordingly.

One idea if you wish to do the multi-array approach would be creating two RAID 1's, on separate controllers, and adding those MD devices to a LVM VG for management. Sure a RAID 10 is fast, but it's still one storage queue, now you have two, and with separate controllers there's no HBA queue sharing either.

Performance Notes:

Remember, SW RAID is no different than HW RAID, understand how it works or when it fails you might end up being more at risk as opposed to say spending your energies creating a regular backup strategy (rsync.net) instead. I've lost count of the number of users who've lost everything because they didn't read the manual and actually test the failure modes.

share|improve this answer
Yes, some amazing IO requirements are definitely in mind (image analysis). If I had the money, this would be purely an SSD affair. Also, it's my understanding that SW RAID is more flexible and cheaper that HW RAID...I've even read that it's faster, but that's neither here nor there for this particular question. Thx for the link though. –  Prophet60091 Dec 14 '12 at 14:38
In which case, asynchronous backup is cheaper, start with a RAID0 Rsync to replicate your work to a safe location. I recommend rsync.net, works well with open standards: rsync, duplicity, & deja-dup. You might want to have two arrays, on separate controllers. One that does all the reading of the original data, and one that serves as the destination of the processed result; nice for multi-threaded processing (pipeline), You also might want to consider using XFS or similar fs that excels at large files, the storage array is just one piece of the puzzle. –  ppetraki Dec 14 '12 at 14:47
interesting: so one RAID0 dedicated to reading, and another RAID0 dedicated to writing, all the while backing up the processed results. I'll have to explore this option as well. Thx. –  Prophet60091 Dec 14 '12 at 15:01
If you're doing batch processing, doing something as simple as dividing the list into 4 parts and spawning a pipeline for each segment for instant parallelism. You can get really cute here and chunk the list by total size and spawn a job for every 500GB for example. –  ppetraki Dec 14 '12 at 15:08
Several things are wrong with this answer: 1) lvm over raid10 is better than lvm over two raid1s, 2) you don't need to replace with the exact same model with or without partitions, 3) hardware or software raid does not matter; raid is no substitute for having regular backups! –  psusi Dec 14 '12 at 15:23
show 5 more comments

Your Answer


