He did give an answer, and from that answer I think he probably does know. In short:
it depends.
I know, it sucks that we can't sum up all aspects of a technology in a single paragraph. Computers aren't easy and are still for people who don't mind doing their own homework.
I used to own a backup company and we did a lot of testing, testing both software and the major hardware vendors. Including some $2,000+ raid cards.
Once upon a time there were plusses and minuses.
Software raid is better. Specifically the Linux MD raid, with LVM on top. It's much more flexible / featurefull and doesn't make your data dependent on a specific controller.
Back in the day, hardware raid had the advantage of not using CPU cycles. At full write bandwidth, raid could use up 10% or even 20% (during a rebuild) of your 500 MHz CPU. That was a consideration.
10% of a 500 MHz CPU is, of course, 50 MHz. Less than 1% of any modern CPU, in the worst case. That's 50 MHz on one core, so if you have four cores @ 2.5 GHz the raid will max out at 0.5% of your total CPU.
Excellent arguments, and for the most part I agree with you.
However, in those rare situations when we're still building actual physical servers instead of deploying cloud-based infrastructure it sometimes does make sense to use hardware RAID, if only because we don't have the chance to use a software-based solution.
In my case this happens when deploying hypervisors. If I'm installing a VMware vSphere cluster on bare metal, the software won't give me any way to set up a software RAID and we have to rely on the hardware RAID controller in the machine.
This is just an example, and I'm sure there are several others (people using different operating systems, or reusing older hardware for home-based file/media servers, etc).
Thanks for pointing that out. I had actually intended to include a footnote mentioning that sometimes decent software raid isn't available, such as with some type 1 hypervisors.
There are also bound to be a few weird situations where hardware is better in some way for some very specific application.
The bigger footnote would probably be that I didn't test Windows software raid. Our solution was based on the Linux stack that I mentioned. I would assume the Windows implementation isn't 100X worse than the Lin
Wait, vSphere doesn't support software raid? I know HyperV can sit on top of a windows storage space, and I know Proxmox supports both LVM and ZFS arrays directly, I just assumed that vSphere, being the expensive dedicated hypervisor it is, would support some kind of multi-disk array in software. TIL
vSphere will do this but it requires multiple nodes and licensing costs almost as much (or more, in some circumstances) per socket as the base vSphere license.
Sure, vSAN is a lot more than just software RAID, but that's what they offer. I'll also note that S2D on Windows requires Datacenter licensing, which is also not cheap.
Uh...why would they. The storage tier handles RAID at the hardware level with software. Point your goofy little VM at a volume...config the volume to your requirements. This is bog standard 101 shit.
Couldn't software RAID be implemented in the host system and just inherited by the virtual conatiner disks that the VM's see?
I'm curious, though. I always thought one of the advantages of hardware RAID was that it incorporated a battery-backed cache so that sensitive write operations don't have to wait for disk I/O before 'completing'. How is that accomplished with software RAID - battery backed cache on each individual drive controller? Or is that just not deemed important in most use cases?
I've seen enough bespoke systems with (admittedly excellent, until they weren't) ancient controllers break after 10 years and someone having to scrape Ebay for a replacement to say yes, this is the correct answer in most all current use cases.
While slightly true, ssh caching and encryption at rest are still far better performers with hardware raid with dedicated processors for crypto. I've also seen far more examples of ZFS pools going missing after a reboot than I have had logical volumes go missing with hardware raid. Feel free to substitute MDRAID in for ZFS. SSD caching with ZFS does perform fairly well but in most cases with hardware raid you are using a hypervisor at which point you are already using a large portion of the fancy new proce
Both processors will be bottlenecked by the interface that they are connected to. The difference is the hardware raid card doesn't have to send everything over pcie and system memory to do operations between the disks.
Still, the advantages of a real software raid solution like zfs can't be discounted. You're not going to be able to run it on that super advanced 4ghz i9 though since intel hates ECC.
PS you also mentioned sending "drive to drive" data (rebuild?) through system memory. System memory bandwidth is 640 Gbps. The SAS connection is 6 Gbps.
Going through system memory is literally over a 100 times faster than reading or writing the disk. It's also several times faster than the memory onboard the raid card, in most cases.
You should have a plan in place to age out old hardware, and have new hardware ready to take over production. While also maintaining backups the entire time, which can use to pre-load your spares and dramatically reduce the replication time when you go live.
One of the reasons to go with a hardware RAID controller in the past was also port density - when even server mainboards only have 4 SATA ports on them, it makes software RAID of any density harder. Now you can grab a cheap 16 port SATA RAID card and just pass the disks through as JBOD and you're in business for mdraid, LVM, ZFS, or whatever.
It was 10% of a 500MHz CPU... But it was when hard drives had platters. I guess the math works different if you set up a raid of fast SSDs. Particularly if you go for the risky types of stripped fast raids. I haven't done the math though.
> It was 10% of a 500MHz CPU... But it was when hard drives had platters. Particularly if you go for the risky types of stripped fast raids. I haven't done the math though.
Let's do the math real quick. If you're actually moving the same amount of data, the CPU usage doesn't change much. Except you've gone from 32 bits per op to 64-bit, so that cuts the number of CPU operations in half.
The SSDs can typically handle about 4X as much data. So that would be 4X as much CPU usage. Divided by two because it's
In my experience, there are a few places where hardware RAID 1 is useful:
A RAID card for a RAID 1 pair of OS drives. Everything else can be software RAID, but having the OS boot on hardware RAID can save headaches.
An OS that doesn't have software RAID. ESXi comes to mind.
A need for encryption on the hardware layer. Even though this can be handled by a NAS/SAN appliance, sometimes there are some requirements for it, so having a RAID controller that uses self-encrypting drives and handles the encryption be
> A RAID card for a RAID 1 pair of OS drives. Everything else can be software RAID, but having the OS boot on hardware RAID can save headaches.
Software RAID 1 works fine for the boot partition. I always used software RAID 1 on the boot partition with the OS. The kernel goes into raid mode before it writes anything to the drive. The boot loader (grub) doesn't know it's raid, but that doesn't matter since it's raid 1, a mirrored pair. Grub can boot from either drive - they are identical.
All my systems boot from RAID 1, and on most of them I have more then 2 drives in the RAID. I have not seen any hardware RAID controllers that can do RAID1 over more then 2 drives. Linux MD software has no limit on how many active drives you place in a RAID 1 array.
My rationale for having 3 drive RAID 1 is simple. As a drive fails, I still have redundancy while I replace it.
You may be missing the purpose of NVRAM(Non-volatile or battery-backed-up ram) on a RAID card. The idea is during a write, the controller cannot correctly acknowledge the write until the data is committed to durable device like disk. This vastly slows down writing to disk. With NVRAM, as soon as the data is copied to NVRAM, you may acknowledge the write. Software raid cannot do this unless it has access to NVRAM. When power is restored, the writes still in NVRAM are committed to disk. This is critical
> the controller cannot correctly acknowledge the write until the data is committed to durable device like disk
And if you don't use a hardware raid card, the write isn't acknowledged until it's written to disk. So that's not made you any safer. It's only preserved the safety, if the system is properly maintained and works right.
So then we can ask, does it increase performance? For either SSDs or for sequential writes on HDs, it cannot possibly increase the sustained data rate. Simply because if data was
The NVRAM cache on a RAID controller is meant to speed up random writes to a spinning disk. The speed up for most spinning disks is more than 20 times because seek speed hasn't significantly increased in 30 years. Yes, NVRAM doesn't help much for a SSD. I don't know what you mean by "your card". Most RAID controllers are available PCIe cards and run at full bus speed. PCIe x16 version 6 is 128GB/sec. You could make a software raid system that uses a NVMe M2 card as a write back cache but I'm not awar
> You could make a software raid system that uses a NVMe M2 card as a write back cache but I'm not aware of software RAID that does this but it may be out there.
Linux can do that. Had a fairly short lifespan for much use because the NVMe cache isn't faster than an array of SSDs, and it ends up more complex. (Or much faster). So people just use SSDs for nearline in most cases if 1 GB/s isn't fast enough.
> Most RAID controllers are available PCIe cards and run at full bus speed. PCIe x16 version 6 is
I was lead engineer on a NAS product and engineer on a NAS/SAN. Both software RAID. it's just way more flexible if you have the CPU to throw at it. And when you're making an appliance you can put as much CPU at it that you want. What was especially attractive was on one of them we found a 24-port SATA/SAS chip that could interface directly with HyperTransport (we were using Opteron at the time). Performance of that software-based system couldn't be beat with an off-the-shelf hardware RAID on a PCI-X slot (y
A mathematician is a device for turning coffee into theorems.
-- P. Erdos
How is this on Slashdot? (Score:-1, Flamebait)
Re: (Score:5, Insightful)
If you don't know, why post?
I'm interested as well, and I want to read suggestions from people who have been there.
Re: (Score:0, Flamebait)
He did give an answer, and from that answer I think he probably does know. In short:
it depends.
I know, it sucks that we can't sum up all aspects of a technology in a single paragraph. Computers aren't easy and are still for people who don't mind doing their own homework.
Used to be "it depends". Now software is better (Score:5, Insightful)
I used to own a backup company and we did a lot of testing, testing both software and the major hardware vendors. Including some $2,000+ raid cards.
Once upon a time there were plusses and minuses.
Software raid is better. Specifically the Linux MD raid, with LVM on top. It's much more flexible / featurefull and doesn't make your data dependent on a specific controller.
Back in the day, hardware raid had the advantage of not using CPU cycles. At full write bandwidth, raid could use up 10% or even 20% (during a rebuild) of your 500 MHz CPU. That was a consideration.
10% of a 500 MHz CPU is, of course, 50 MHz. Less than 1% of any modern CPU, in the worst case. That's 50 MHz on one core, so if you have four cores @ 2.5 GHz the raid will max out at 0.5% of your total CPU.
Re:Used to be "it depends". Now software is better (Score:5, Informative)
Excellent arguments, and for the most part I agree with you.
However, in those rare situations when we're still building actual physical servers instead of deploying cloud-based infrastructure it sometimes does make sense to use hardware RAID, if only because we don't have the chance to use a software-based solution.
In my case this happens when deploying hypervisors. If I'm installing a VMware vSphere cluster on bare metal, the software won't give me any way to set up a software RAID and we have to rely on the hardware RAID controller in the machine.
This is just an example, and I'm sure there are several others (people using different operating systems, or reusing older hardware for home-based file/media servers, etc).
Re: (Score:2)
Thanks for pointing that out. I had actually intended to include a footnote mentioning that sometimes decent software raid isn't available, such as with some type 1 hypervisors.
There are also bound to be a few weird situations where hardware is better in some way for some very specific application.
The bigger footnote would probably be that I didn't test Windows software raid. Our solution was based on the Linux stack that I mentioned. I would assume the Windows implementation isn't 100X worse than the Lin
Traditionally Softwrare RAID was buggy (Score:2)
Especially on Windows.
You were far more likely to lose data due to software bugs than due to a disk failure.
Hopefully things have got better.
Re: (Score:2)
Wait, vSphere doesn't support software raid? I know HyperV can sit on top of a windows storage space, and I know Proxmox supports both LVM and ZFS arrays directly, I just assumed that vSphere, being the expensive dedicated hypervisor it is, would support some kind of multi-disk array in software. TIL
Re:Used to be "it depends". Now software is better (Score:4, Insightful)
Remember, VMware and EMC are the same company. They like selling storage hardware.
Re: (Score:2)
vSphere will do this but it requires multiple nodes and licensing costs almost as much (or more, in some circumstances) per socket as the base vSphere license.
Sure, vSAN is a lot more than just software RAID, but that's what they offer. I'll also note that S2D on Windows requires Datacenter licensing, which is also not cheap.
Re: Used to be "it depends". Now software is bette (Score:2)
Uh...why would they. The storage tier handles RAID at the hardware level with software. Point your goofy little VM at a volume...config the volume to your requirements. This is bog standard 101 shit.
Re: (Score:2)
Couldn't software RAID be implemented in the host system and just inherited by the virtual conatiner disks that the VM's see?
I'm curious, though. I always thought one of the advantages of hardware RAID was that it incorporated a battery-backed cache so that sensitive write operations don't have to wait for disk I/O before 'completing'. How is that accomplished with software RAID - battery backed cache on each individual drive controller? Or is that just not deemed important in most use cases?
Re:Used to be "it depends". Now software is better (Score:5, Insightful)
Software raid is better.
I've seen enough bespoke systems with (admittedly excellent, until they weren't) ancient controllers break after 10 years and someone having to scrape Ebay for a replacement to say yes, this is the correct answer in most all current use cases.
Re: (Score:2)
Hardware raid for best performance.
Software raid for most flexibility.
So it all comes down to what you need.
Re: (Score:2)
That was true in the mid 1990s. Consider these two processors:
8-core i9 @ 4 GHz per core
Single core Celeron @ 500 MHz
Which do you think will give better performance?
The raid card has the Celeron on-board.
The software raid uses the i9.
Re: (Score:2)
While slightly true, ssh caching and encryption at rest are still far better performers with hardware raid with dedicated processors for crypto. I've also seen far more examples of ZFS pools going missing after a reboot than I have had logical volumes go missing with hardware raid. Feel free to substitute MDRAID in for ZFS. SSD caching with ZFS does perform fairly well but in most cases with hardware raid you are using a hypervisor at which point you are already using a large portion of the fancy new proce
Re: (Score:2)
Both processors will be bottlenecked by the interface that they are connected to. The difference is the hardware raid card doesn't have to send everything over pcie and system memory to do operations between the disks.
Still, the advantages of a real software raid solution like zfs can't be discounted. You're not going to be able to run it on that super advanced 4ghz i9 though since intel hates ECC.
Re: (Score:2)
> The difference is the hardware raid card doesn't have to send everything over pcie and system memory to do operations between the disks.
Yeah it doesn't send it over the 64 GBps PCIe, bus it sends it over the 0.75 GBps sas or sata connection. (6 Gbps is 0.75 GBps).
Yep, the hardware raid doesn't send it over a super fast connection, it instead uses a connection that runs 85 times slower.
Re: (Score:2)
PS you also mentioned sending "drive to drive" data (rebuild?) through system memory. System memory bandwidth is 640 Gbps. The SAS connection is 6 Gbps.
Going through system memory is literally over a 100 times faster than reading or writing the disk. It's also several times faster than the memory onboard the raid card, in most cases.
Re: (Score:2)
If you're using some oddball RAID controllers. I don't recall every having an issue mounting Adaptec RAID volumes on new controllers.
Re: (Score:3)
RAID is not backup. Why should you have to go and buy an ancient controller if the card fails? Start fresh and restore from backup.
Re: (Score:2)
ding ding ding.
You should have a plan in place to age out old hardware, and have new hardware ready to take over production. While also maintaining backups the entire time, which can use to pre-load your spares and dramatically reduce the replication time when you go live.
Re: (Score:2)
One of the reasons to go with a hardware RAID controller in the past was also port density - when even server mainboards only have 4 SATA ports on them, it makes software RAID of any density harder. Now you can grab a cheap 16 port SATA RAID card and just pass the disks through as JBOD and you're in business for mdraid, LVM, ZFS, or whatever.
But it wasn't always that way.
Re: (Score:1)
Re: (Score:2)
> It was 10% of a 500MHz CPU... But it was when hard drives had platters. Particularly if you go for the risky types of stripped fast raids. I haven't done the math though.
Let's do the math real quick. If you're actually moving the same amount of data, the CPU usage doesn't change much. Except you've gone from 32 bits per op to 64-bit, so that cuts the number of CPU operations in half.
The SSDs can typically handle about 4X as much data.
So that would be 4X as much CPU usage. Divided by two because it's
Re: (Score:2)
In my experience, there are a few places where hardware RAID 1 is useful:
A RAID card for a RAID 1 pair of OS drives. Everything else can be software RAID, but having the OS boot on hardware RAID can save headaches.
An OS that doesn't have software RAID. ESXi comes to mind.
A need for encryption on the hardware layer. Even though this can be handled by a NAS/SAN appliance, sometimes there are some requirements for it, so having a RAID controller that uses self-encrypting drives and handles the encryption be
A couple of those ... (Score:2)
A couple thoughts.
> A RAID card for a RAID 1 pair of OS drives. Everything else can be software RAID, but having the OS boot on hardware RAID can save headaches.
Software RAID 1 works fine for the boot partition.
I always used software RAID 1 on the boot partition with the OS. The kernel goes into raid mode before it writes anything to the drive. The boot loader (grub) doesn't know it's raid, but that doesn't matter since it's raid 1, a mirrored pair. Grub can boot from either drive - they are identical.
Re:A couple of those ... RAID 1 with 3+ drives (Score:1)
All my systems boot from RAID 1, and on most of them I have more then 2 drives in the RAID.
I have not seen any hardware RAID controllers that can do RAID1 over more then 2 drives. Linux MD software has no limit on how many active drives you place in a RAID 1 array.
My rationale for having 3 drive RAID 1 is simple. As a drive fails, I still have redundancy while I replace it.
Re:A couple of those-NVRAM is to speed up writes (Score:2)
Re: (Score:2)
> the controller cannot correctly acknowledge the write until the data is committed to durable device like disk
And if you don't use a hardware raid card, the write isn't acknowledged until it's written to disk. So that's not made you any safer. It's only preserved the safety, if the system is properly maintained and works right.
So then we can ask, does it increase performance?
For either SSDs or for sequential writes on HDs, it cannot possibly increase the sustained data rate. Simply because if data was
Re: (Score:2)
Re: (Score:2)
> You could make a software raid system that uses a NVMe M2 card as a write back cache but I'm not aware of software RAID that does this but it may be out there.
Linux can do that. Had a fairly short lifespan for much use because the NVMe cache isn't faster than an array of SSDs, and it ends up more complex. (Or much faster). So people just use SSDs for nearline in most cases if 1 GB /s isn't fast enough.
> Most RAID controllers are available PCIe cards and run at full bus speed. PCIe x16 version 6 is
Re: (Score:2)
I was lead engineer on a NAS product and engineer on a NAS/SAN. Both software RAID. it's just way more flexible if you have the CPU to throw at it. And when you're making an appliance you can put as much CPU at it that you want. What was especially attractive was on one of them we found a 24-port SATA/SAS chip that could interface directly with HyperTransport (we were using Opteron at the time). Performance of that software-based system couldn't be beat with an off-the-shelf hardware RAID on a PCI-X slot (y