I need advice for how to setup RAID for a server that has an array of 16 disks (boot/OS disks are outside this array). In this 16-disks array, all disks are identical (12.8TB SSDs).

The ideal solution for me would be to have at least half the space usable (so, at least 102.4TB out of 204.8TB), and the capacity to resist at least 8 disks failures (any 8 disks should be able to fail without any data loss).

There's no need to see the 102.4TB usable space as a single contiguous volume, so the disks can also be grouped into separate RAID volumes, etc. as long as any 8 disks can break without any data loss.

I was thinking of a couple of solutions, but no solution really satisfies me completely:

  1. The first solution was a single RAID 1+6 volume with 8 spans (76.8TB of capacity, and in the worst case any 5 disks can break without data loss). Scheme: RAID6( RAID1(D1,D2), RAID1(D3,D4), RAID1(D5,D6), RAID1(D7,D8), RAID1(D9,D10), RAID1(D11,D12), RAID1(D13,D14), RAID1(D15,D16) )

  2. Second solution was a single RAID Z3+1 volume with 2 spans (64TB of capacity, and in the worst case any 7 disks can break without data loss). Scheme: RAID1( RAID_Z3(D1,D2,D3,D4,D5,D6,D7,D8), RAID_Z3(D9,D10,D11,D12,D13,D14,D15,D16) )

The perfect RAID solution for me would be an octuple parity RAID (like RAID 5 is single parity, RAID 6 is double parity, and RAID Z3 is triple parity), because that would mean 102.4TB of usable space and any 8 disks can break without data loss. But this hypothetical "RAID Z8" doesn't exist, unfortunately.

Thanks in advance for any advice.

P.S.: performance is really not an issue in this case, so it is not a parameter to take into consideration.

P.P.S.: the reason why I seek resistance to any 8-disk failures is because these disks will contain highly mission critical data. Backups are already planned, but still, if the RAID volume stops working, several mission-critical servers and services that depend on it will stop working as well.

P.P.P.S.: high availability of the server itself has already been taken care of.

  • high availability of the server itself has already been taken care of. THE server? There are many, many, many single points of failure on any one server... May 16 at 21:50
  • The server in question has 4 independent motherboards, and each motherboard has a different OS, RAM, double network cards, etc. Breaking the server is equivalent to break 4 servers, all at the same time. So, yes, I don't think it's a single point of failure. And that's why I added that high availability of the server itself was already been taken care of. May 16 at 22:22
  • I suspect X-Y problem here. What you are trying to achieve in the end? Why you need this "8 any simultaneously failed disks", why don't you employ a proper monitoring and replacement of disks in a more "traditional" array if you have such tight availability requirements? The true redundancy of RAID comes from proper service, without it you'll eventually lose all your data no matter how strong your construction was. May 17 at 7:07
  • @NikitaKipriyanov The 16 disks are distributed across the 4 nodes of the server. If one node goes down (broken motherboard, OS crash, broken network cards, etc.) you lose 4 disks. If two nodes go down, you lose 8 disks. If one node goes down, and at the same time two disks break on the other nodes, you lose 6 disks. Another possible solution I thought of is a RAID 6+0 with 4 spans, with each span distributed on the 4 nodes, so that if you lose 2 nodes and 8 disks, it keeps working (but this only protects against any 2 disks failures in the worst case). May 17 at 8:52
  • I only ue R1/10 and R6/60, I know a few people who swear by RAID Z too but I personally have no experience - anything else, especially R5/50 is basically trash :)
    – Chopper3
    May 18 at 13:19

Try network-based redundant storage, e.g. CEPH. You may set it to store 9 copies of each block and set up so it'll be storing all of them on different OSDs, so each copy will be on the another device; in this case you truly can remove 8 OSDs and still have at least one copy of each block on the remaining system.

Yes, very inefficient in terms of redundant storage, but those was exactly by your requirement. I consider it as very exaggerated, up to the meaninglessness. The world seem to reach consensus nobody really needs so many copies. Martian rovers have three computers, and that's enough even in the place where conditions are extremely harsh and somebody to fix them is at least half-year away.

Better invest into the system that has live repairing ability than having so many static copies. CEPH does exactly that: you specify you need, say, 3 copies of each block, and those copies must not be co-located. Now, should some device become inaccessible, the system finds that out, and it knows which blocks were stored on it; so it immediately begins redistributing them, so to reach required redundancy again. You may set it up so it'll block access if there is only 1 copy left, so so it will have a chance to repair (distribute that copy and resume access). You may create several pools with different requirements in a single cluster. Should you extend storage, you just add more OSDs.

  • Thanks, I'll take a look at a possible architecture with Ceph. P.S.: yes, it may be exaggerated to have this level of redundancy, but the storage server will be the central point in a mission-critical infrastructure, and it cannot go down or even lose just 10 minutes of data. May 18 at 14:11

You did not specify the operating system, which is an essential requirement for the answer.

Personally I would not adopt such a mechanism, because there is not sufficient redundancy and too slow recovery time.


I would therefore operate with a zfs pool on FreeBSD (v12, not 13 with openzfs, still not mature enough for me) replicated on another pool (using for example syncoid/sanoid) on a different machine (if possible) or even cheap iSCSI NAS-based device

A sort of "hardware-RAID", but "full" (if one machine goes down, the other can be used)

In fact, a single breaking/failure point is not only the volume, but also and above all the machine to which it is connected.

Also, always to minimize recovery time during any problems, I never recommend volumes of more than 8 disks, because it is much, much easier to find RAID controllers (flashed in non-RAID mode for zfs) for up to 8 drives.

What do you do if, for example, a SAS controller with 16 connectors fails?

You can't take the disks and connect them to the SATA controllers of a $500 computer bought in the emergency.

Your Answer

