Triple M.2 NVMe RAID-0 Testing Proves Latency Reductions 73
Vigile writes: The gang over at PC Perspective just posted a story that looks at a set of three M.2 form factor Samsung 950 Pro NVMe PCIe SSDs in a RAID-0 array, courtesy of a new motherboard from Gigabyte that included three M.2 slots. The pure bandwidth available in this configuration is amazing, breaching 3.3 GB/s on reads and 3.0 GB/s on writes. But what is more interesting is a new testing methodology that allows for individual storage IO latency capturing, giving us a look at performance of SSDs in all configurations. What PC Perspective proved here is that users often claiming that RAIDs "feel faster" despite a lack of bandwidth result to prove it, are likely correct. Measurements now show that the latency of IO operations improves dramatically as you add drives to an array, giving a feeling of "snappiness" to a system beyond even what a single SSD can offer. PC Perspective's new testing demonstrates the triple RAID-0 array having just 1/6th of the latency of a single drive.
All about IOPS (not bandwidth) (Score:3)
On VM's in a home lab I see no difference between raid 0 and not with SATA on Samsung pros outside of benchmarks.
However, my VM's do boot quicker with the Samsung than the sansdisk that replaced them. The IOPS were better and that made them boot quicker and shutdown. I imagine at work it is the same with database or production VM's too
Re:All about IOPS (not bandwidth) (Score:4, Insightful)
Usually for this kind of stuff people are wanting it for gaming. Given that a lot of the "load" times these days are work for the GPU, I'm not sure you'd see a massive improvement.
Re: (Score:2)
You would notice a difference as some of the files are gigs in size. This is bandwidth. But besides gamers professionals like developers, sql guys, Sys admins, and those setting a home virtual lab. These are mostly IOP based and wouldn't benefit. Gamers would for bandwidth but are a nich as many young folks who play do not have that nice job to pay for a $3500 system with raid 0 NVE drives.
Re: (Score:3)
Having just built a brand new Z170 system (with a 950 PRO), it doesn't take $3500 to build. Not even close. It's closer to $1200 for a single 512GB SSD system, and an additional $600 to put two more 512GB drives in it.
Re: (Score:2)
Mine is a haswell I built just over a year ago. I spent about $3500 for an i7, 32 gigs of ram, and 3 Samsung pro 840 SSD and a case and platnium grade power supply and an nvidia gtx 770.
This is not even a high end gaming system but is decent. Mostly for VM lab work for my certs and the fact I wanted a nice system :-). But if you only spend $1200 for such a system you wouldn't use a low end dual core i5 with just 8 gigs of ram and a cheap case with integrated or low end graphics?
Raid 0 NVMe users typically w
Re: (Score:2)
It was a skylake i7-6700k ($350), 16GB of DDR4 ram ($109), GA-Z170X-GAMING GT ($249), Corsair H90 ($105), Samsung 950 PRO 512GB ($298) = $1,111. I reused the case/powersupply/video card. I wouldn't need a video card for myself because the built-in video card is fine for office use (programming/server).
My gaming PC is a different one. i7-3930k, 64GB ram, 2 Samsung SSDs in a RAID-0, and a 980. (and a LSI 9280-16i4e with 12 3TB drives in a RAID-6) in which I run my games from a RamDisk, so the NVMe SSDs woul
Re: All about IOPS (not bandwidth) (Score:2)
Games and most non-DB applications are very poorly written for opening large files.
Most libraries have some very bad "defaults" for modern systems as the common denominator (2TB drives) still have 512B blocks and thus programs will often read very large files 512B at a time.
Re: (Score:3)
my experience too. the jump from disk to good quality sata ssd was noticeable. jump from sata ssd to raided pcie ssds felt like painting go-faster stripes on a car. it's nice to look at but in a blind test, i wouldn't be able to tell the difference.
1 word: (Score:2)
Re: (Score:2)
or a futile attempt to make Intellij IDEA run at usable speeds?
Re:This is retarded. (Score:5, Informative)
The SSD controller already does a form of this, as it is talking to multiple flash memory dies over multiple channels. RAID is just another layer to get even more performance out of more parallelism (and as we figured out in testing, to considerably drop the latency under load).
Allyn Malventano
Storage Editor, PC Perspective
Re: (Score:2)
Re: (Score:2)
The other consideration is heat, to read/write data that fast generates a lot of heat (as you can see in this page of TFA [pcper.com]). Fitting the same heat load in a small enclosure would probably require some cooling (although this is a problem the computing industry has had to solve with practically every other component).
RAID 0 is not for anything you don't want to lose (Score:4, Informative)
Re: RAID 0 is not for anything you don't want to l (Score:2)
But does raid 0 support sharing? It is the secret ingredient in the async sauce?
Re:RAID 0 is not for anything you don't want to lo (Score:4, Interesting)
They did post a quick speed graph regarding raid-5 on the drives; obviously writes took an impact but reads were almost exactly the same as raid-0 x3 drives.
Re: (Score:2)
Hopefully we start seeing boards transitioning over to offering connectivity for ever-greater numbers of M.2 drives.
For RAID-0, the big issue is "lose a drive and you're fucked".
For RAID-5, the big issue is "lose a drive on a large-enough array and you could be looking at an unrecoverable read error during the array recovery".
Granted, most of the people who are using these setups are frothing gamers and hardware junkies who aren't keeping anything truly valuable within those filesystems.
But for those who ar
Re: (Score:2)
For RAID-5, the big issue is "lose a drive on a large-enough array and you could be looking at an unrecoverable read error during the array recovery".
This gets repeated a lot, but isn't a problem for any halfway decent RAID setup because they slowly read data from the drives in the background (called patrol read on LSI/Dell controllers). The chances of a problem with a drive not turning up in one of the numerous patrol reads yet happening during a recovery are astronomically small.
Re:RAID 0 is not for anything you don't want to lo (Score:4, Informative)
Not completely true. With 6 or 8tb drives, you are looking at a few days to a week or so of the raid rebuilding. During this time, you have the protection of raid 0 without the speed.
Re: (Score:2)
When was the last time you actually had to rebuild an array with large (4GB+) constituent disks?
And they don't read THAT slowly. Indeed, the increased (and sustained) load during the rebuild can cause additional drives in the array to fail.
Re: (Score:2)
Rebuild just finished... Yesterday. Took 5 days, and that is with 3TB disks.
Re: (Score:2)
5 days is a long time to hold one's breath...
Re: (Score:2)
And is why it's a RAID-6 and not a RAID-5. I've had a second disk fail during rebuilds twice so far.
Re: (Score:2)
Astronomically small? It's happened to me TWICE in a couple years and I only have a single large raid array. It happens quite often -- and I'm using one of your LSI controllers (9280-16i4e).
Re: (Score:1)
For RAID-5, the big issue is "lose a drive on a large-enough array and you could be looking at an unrecoverable read error during the array recovery".
This gets repeated a lot, but isn't a problem for any halfway decent RAID setup because they slowly read data from the drives in the background (called patrol read on LSI/Dell controllers). The chances of a problem with a drive not turning up in one of the numerous patrol reads yet happening during a recovery are astronomically small.
I'm not sure how you define "astronomically", but I've seen this more than a few times in my career. And it has become increasingly common with larger disks and larger arrays.
RAID 5 is decent for availability... but you'd better be able to restore from your backups. RAID 6 should be the default these days (though I prefer ZFS RAIDZ2 or RAIDZ3). And don't be one of those idiots who makes a 32-disk, 192 TB RAID5 (or 6 for that matter).
Re: RAID 0 is not for anything you don't want to l (Score:2)
Any classic RAID level is useless if you want data safety. So one of your drives in RAID10/5/6 returns garbled data (without an error), which copy/parity do you trust?
Also many 5/6 implementations won't actually calculate the parity chunk on reads, only for rebuilds. There are some pricey controllers that do full checksumming ala ZFS on chip but as with most hardware systems the SPOF becomes your controller.
With the drives becoming ever larger and faster, more data is being read but the errors per terabyte
Re: (Score:2)
Is the rebuild issue for SSD RAID-5 arrays the same stratum of risk it is for spinning rust?
I would presume not, both because of speed and because there's not nearly as much added stress from the intensive reads necessary to rebuild the array.
Double parity and/or hot spare is better, but I kind of wonder as SSDs gain write durability (or it becomes more accepted they just have it, as some endurance tests have noted) and they start popping up in more budget minded arrays if maybe RAID-5 might make a comeback
Re: (Score:2)
RAID0 is any drive failing is a loss, so multi-drive failures don't matter so much, but they're also much less likely until it's a firmware bug or other pathological issu
Re: (Score:3)
I have found these calculators work well for projecting the performance and capacity of various RAID levels:
http://wintelguy.com/raidperf.pl/?formid=1&raidtype=R0&ndg=2&ng=1 [wintelguy.com]
http://wintelguy.com/raidperf.pl/?formid=2&raidtype=R0&ndg=2&ng=1 [wintelguy.com]
Some other guy mentioned RAID 10 isn't a backup strategy; he's correct (no RAID level is), however one thing to keep in mind is that when his RAID 0 array dies, he'd better hope his back-ups are all up-to-date and restorable. With just about a
Re: (Score:2)
Not smart to run your OS on RAID 0.
Why? You're assuming the OS is something that I can't just re-install? Remember that you're only slightly more than doubling the failure rate. Given the incredibly low failure rates it's not like you're guaranteed to lose things constantly.
Re: (Score:2)
Most SSD do RAID internally across several dies already.
Re: (Score:2)
RAID0 is unsuitable for situations that require very high uptime but there is nothing inherently dangerous about storing real data on a RAID0 array. I know this gets said frequently but, RAID, at any level, isn't a backup. It's a reliability/performance feature. Even if you had configured these three disks as a triple mirrored RAID1, you would be insane to not run SMART monitoring tools on the disks and even more insane to not have good backups. I don't know if I've ever had a disk fail without plenty o
Re: (Score:3)
Ummm...ok. So when your SMART detects a failing drive in your RAID0 array and you decide you want to replace it, how do you do that exactly? Oh, that's right, wipe the entire array and restore from backup, which, depending on the size of your array can take anywhere from several hours to days, more if you decided to use your array to run the OS as well. RAID0 is just a plain terrible idea, period. It doesn't matter if you don't think you need uptime, an N disk RAID0 is N times more likely to fail catastroph
Re: (Score:2)
It depends on how you've setup your RAID0 array. If you are using mdraid, you can simply take the array offline, dd the contents of a failing disk onto a new disk, remove the old disk and bring the array back online (you can do this with a USB boot stick if you need to take the root filesystem offline). That's certainly more work than popping a failing disk out of a hotswap bay, screwing a new disk into the drive tray and pushing it back in. But, it's not that prohibitive.
Now, having said that, I certain
Re: (Score:2)
What do I do? Well, I run a RAID-0 of SSD for my OS. I drop both the failing drive and a new drive into my drive duplicator, hit a button, and approximately 5 minutes later I put the new drive into the box and it's running again. That is of course if SMART detects it before failure which is ~50% of the time. Otherwise I wipe and restore from backups.... I'm guessing about 2-3 hours as I haven't had to do it yet.
Hmmmm ... (Score:2)
So, basically it partly takes seek time out of the equation, or something similar?
Because then in theory I guess you can be serving multiple requests instead of ju
Re: (Score:2)
That's pretty much it. The trick was showing it properly, which has not previously been possible without our new test method.
Allyn Malventano
Storage Editor, PC Perspective
A welcome improvement in the age of PC stagnation (Score:5, Insightful)
I've been building PCs long enough to remember a time when things were improving so quickly that it made no sense to keep a computer for more than 4 years. But since then, the progress in CPU performance has reached a plateau. People like me, who bought a good Sandy Bridge system in 2011, still have a system that doesn't come close to feeling crippled and lazy. We don't have much reason to envy the people who bought the latest generation of i5/i7 systems. Five years used to mean an order of magnitude improvement in performance. Now it's not even a doubling. I've sometimes wondered when I will finally start feeling the urge to upgrade my system.
These SSD latency numbers are the first thing I've seen that gave me the feeling that there is some truly worthwhile trick that my present computer can't come close to matching. I'm not saying that I now want to upgrade, but on reading this, I have become upgrade-curious for the first time in many years.
Re: (Score:1)
You're right. I/O is where the best improvements have been coming lately.
I'd still be using a Core2Quad if it wasn't for the platform's outdated I/O features.
PCI 3.0, SATA 3, USB3, DDR4, and surrounding technologies like M.2 and NVME are the the real reason to upgrade. The newer platforms are faster in most practical applications simply because they can feed the data to the CPU faster.
Intel's also been focusing on power saving. The new parts sip power in comparison to the decade old core2.
To bad the skylack only has 16+4 PCI-E 3.0 (Score:2)
To bad the skylack only has 16+4 PCI-E 3.0 lanes from the CPU. That why intel needs to put QPI in all cpus and drive the chipset off of QPI and not DMI.
Re: (Score:2)
I can totally agree with these sentiments. My first computer was a Commodore 64. Since then, I've been chasing the performance dragon, upgrading to a new computer every couple years. C128, Amiga, XT, 286, 386, 486, Athlon, P60, P2, P3, P4, P5... up until my last two computers.
My previous PC was a Core2Duo e6600 (circa 2006) and its performance was great and then still good. Had it for 5 years. I assembled my current desktop, a Core i7 2600k, in mid 2011. I've upgraded the video card (I'm a gamer) a couple t
Questionable Results (Score:5, Interesting)
These results seem to be very questionable. Their graphs claim that in some configurations almost all 4k read requests are handled within 100 ns. But getting even a single DRAM burst from a random DRAM location already takes almost 100ns, even through the memory controller is connected with a much tighter interface, optimized for low latency and PCIe is much slower than DRAM interface. Even without overhead 4 PCIe 3.0 lanes ( 8 GB/s) can only transfer 8 KB per s. Transfering a 4 KB Block should thus take at least 0.5 s or 500ns and that does not include any overhead nor the time needed to actually send the request to the SSD, open the page from the NAND flash, run ECC and decompression.
Re: (Score:2)
I think they must have made a scale error when rendering their graph. In the article they introduced this latency tool in (Oct 2015); you'll see that even the best PCIe SSDs tested had their 85th percentile at 100 microseconds: http://www.pcper.com/image/vie... [pcper.com]
Additionally; the far right of the graph in this article is labeled as "1ms", when if the scale was reasonable, it'd be 1us. I suspect they just forgot about microseconds when doing their conversion.
They also cite a 6ns overhead due to the RAID contro
Re: (Score:2)
Yup, it's been corrected. Should have been 6 micro (u) sec.
Re: (Score:2)
Even without overhead 4 PCIe 3.0 lanes ( 8 GB/s) can only transfer 8 KB per s.
Did you mean 8 KB per ms? Rinse and repeat for 0.5s/500ns being equivalent.
Re: (Score:2)
Yes, unfortunately the mu did not show up. Let's see if the html entity works: nope
Seems you have to use u for mu on slashdot.
Re:Questionable Results (Score:4, Informative)
Yup, we had a scale error as our Excel-fu was not as strong as we'd hoped when we made the custom format for the axis, and I totally fell for the error. I've updated the article with corrections.
Damn! (Score:2)
That Gigabyte board is looking more and more attractive.
And I'm actually due for a system rebuild this year...
You forgot something (Score:3)
PC Perspective's new testing demonstrates the triple RAID-0 array having just 1/6th of the latency of a single drive.
That was with a queue depth of 16. Not exactly representative of a normal desktop user.
Re:You forgot something (Score:4)
PC Perspective's new testing demonstrates the triple RAID-0 array having just 1/6th of the latency of a single drive.
That was with a queue depth of 16. Not exactly representative of a normal desktop user.
It's reasonable for peak power user load. Folks running / considering triple SSD RAIDs are not exactly 'typical desktop users' :)
Re: (Score:2)
All testing was performed with default (disabled cache). Further, cache settings have little effect on NVMe RAIDs on Z170. Additionally, our minimum latencies were 6us *longer* in an array vs. single SSD, so clearly no caching taking place.
RAID0 of SSDs will be faster than a single SSD, ya (Score:1)
No Shit Sherlock?! (Score:2)
Let me get this straight. When you have more devices available to service read or write requests, the time that it takes to service the request goes down.
What next? Are we going to be told that RAID5 gives better read performance than a single drive too?
you missed the real news here! (Score:3)
The gang over at PC Perspective...
gangs, presumably roving, are taking over websites now! YOUR SITE COULD BE NEXT!
This would seem obvious (Score:2)
There are several things that affect the latency. You will get about 200us latency on program on a die, but that can be reduced by using some caching and acknowledging the writes before the program finishes, but that cache can be saturated by sustained writes. Especially with random writes that get high level of map updates. As that fills the write latency on sustained random writes will eventually climb to the 2 program instructions latency, which is 400us. With multiple controllers, only every third write