Follow Slashdot blog updates by subscribing to our blog RSS feed

 



Forgot your password?
typodupeerror
Check out the new SourceForge HTML5 internet speed test! No Flash necessary and runs on all devices. ×
Data Storage Stats Upgrades

Triple M.2 NVMe RAID-0 Testing Proves Latency Reductions 73

Vigile writes: The gang over at PC Perspective just posted a story that looks at a set of three M.2 form factor Samsung 950 Pro NVMe PCIe SSDs in a RAID-0 array, courtesy of a new motherboard from Gigabyte that included three M.2 slots. The pure bandwidth available in this configuration is amazing, breaching 3.3 GB/s on reads and 3.0 GB/s on writes. But what is more interesting is a new testing methodology that allows for individual storage IO latency capturing, giving us a look at performance of SSDs in all configurations. What PC Perspective proved here is that users often claiming that RAIDs "feel faster" despite a lack of bandwidth result to prove it, are likely correct. Measurements now show that the latency of IO operations improves dramatically as you add drives to an array, giving a feeling of "snappiness" to a system beyond even what a single SSD can offer. PC Perspective's new testing demonstrates the triple RAID-0 array having just 1/6th of the latency of a single drive.
This discussion has been archived. No new comments can be posted.

Triple M.2 NVMe RAID-0 Testing Proves Latency Reductions

Comments Filter:
  • by Billly Gates ( 198444 ) on Monday February 01, 2016 @01:52PM (#51415415) Journal

    On VM's in a home lab I see no difference between raid 0 and not with SATA on Samsung pros outside of benchmarks.

    However, my VM's do boot quicker with the Samsung than the sansdisk that replaced them. The IOPS were better and that made them boot quicker and shutdown. I imagine at work it is the same with database or production VM's too

    • by ArmoredDragon ( 3450605 ) on Monday February 01, 2016 @02:26PM (#51415661)

      Usually for this kind of stuff people are wanting it for gaming. Given that a lot of the "load" times these days are work for the GPU, I'm not sure you'd see a massive improvement.

      • You would notice a difference as some of the files are gigs in size. This is bandwidth. But besides gamers professionals like developers, sql guys, Sys admins, and those setting a home virtual lab. These are mostly IOP based and wouldn't benefit. Gamers would for bandwidth but are a nich as many young folks who play do not have that nice job to pay for a $3500 system with raid 0 NVE drives.

        • Having just built a brand new Z170 system (with a 950 PRO), it doesn't take $3500 to build. Not even close. It's closer to $1200 for a single 512GB SSD system, and an additional $600 to put two more 512GB drives in it.

          • Mine is a haswell I built just over a year ago. I spent about $3500 for an i7, 32 gigs of ram, and 3 Samsung pro 840 SSD and a case and platnium grade power supply and an nvidia gtx 770.

            This is not even a high end gaming system but is decent. Mostly for VM lab work for my certs and the fact I wanted a nice system :-). But if you only spend $1200 for such a system you wouldn't use a low end dual core i5 with just 8 gigs of ram and a cheap case with integrated or low end graphics?

            Raid 0 NVMe users typically w

            • It was a skylake i7-6700k ($350), 16GB of DDR4 ram ($109), GA-Z170X-GAMING GT ($249), Corsair H90 ($105), Samsung 950 PRO 512GB ($298) = $1,111. I reused the case/powersupply/video card. I wouldn't need a video card for myself because the built-in video card is fine for office use (programming/server).

              My gaming PC is a different one. i7-3930k, 64GB ram, 2 Samsung SSDs in a RAID-0, and a 980. (and a LSI 9280-16i4e with 12 3TB drives in a RAID-6) in which I run my games from a RamDisk, so the NVMe SSDs woul

      • Games and most non-DB applications are very poorly written for opening large files.

        Most libraries have some very bad "defaults" for modern systems as the common denominator (2TB drives) still have 512B blocks and thus programs will often read very large files 512B at a time.

    • my experience too. the jump from disk to good quality sata ssd was noticeable. jump from sata ssd to raided pcie ssds felt like painting go-faster stripes on a car. it's nice to look at but in a blind test, i wouldn't be able to tell the difference.

  • by PeeAitchPee ( 712652 ) on Monday February 01, 2016 @02:04PM (#51415481)
    Remember kids, losing just one drive dumps the entire array. It's really not appropriate for anything besides completely transient data (scratch disks, stuff like this benchmark, etc.). Not smart to run your OS on RAID 0. RAID 10, OTOH . . . now we're talking.
    • But does raid 0 support sharing? It is the secret ingredient in the async sauce?

    • by karnal ( 22275 ) on Monday February 01, 2016 @02:14PM (#51415555)

      They did post a quick speed graph regarding raid-5 on the drives; obviously writes took an impact but reads were almost exactly the same as raid-0 x3 drives.

      • by Chas ( 5144 )

        Hopefully we start seeing boards transitioning over to offering connectivity for ever-greater numbers of M.2 drives.

        For RAID-0, the big issue is "lose a drive and you're fucked".

        For RAID-5, the big issue is "lose a drive on a large-enough array and you could be looking at an unrecoverable read error during the array recovery".

        Granted, most of the people who are using these setups are frothing gamers and hardware junkies who aren't keeping anything truly valuable within those filesystems.
        But for those who ar

        • For RAID-5, the big issue is "lose a drive on a large-enough array and you could be looking at an unrecoverable read error during the array recovery".

          This gets repeated a lot, but isn't a problem for any halfway decent RAID setup because they slowly read data from the drives in the background (called patrol read on LSI/Dell controllers). The chances of a problem with a drive not turning up in one of the numerous patrol reads yet happening during a recovery are astronomically small.

          • by sr180 ( 700526 ) on Monday February 01, 2016 @08:21PM (#51418063) Journal

            Not completely true. With 6 or 8tb drives, you are looking at a few days to a week or so of the raid rebuilding. During this time, you have the protection of raid 0 without the speed.

          • by Chas ( 5144 )

            When was the last time you actually had to rebuild an array with large (4GB+) constituent disks?

            And they don't read THAT slowly. Indeed, the increased (and sustained) load during the rebuild can cause additional drives in the array to fail.

          • Astronomically small? It's happened to me TWICE in a couple years and I only have a single large raid array. It happens quite often -- and I'm using one of your LSI controllers (9280-16i4e).

          • For RAID-5, the big issue is "lose a drive on a large-enough array and you could be looking at an unrecoverable read error during the array recovery".

            This gets repeated a lot, but isn't a problem for any halfway decent RAID setup because they slowly read data from the drives in the background (called patrol read on LSI/Dell controllers). The chances of a problem with a drive not turning up in one of the numerous patrol reads yet happening during a recovery are astronomically small.

            I'm not sure how you define "astronomically", but I've seen this more than a few times in my career. And it has become increasingly common with larger disks and larger arrays.

            RAID 5 is decent for availability... but you'd better be able to restore from your backups. RAID 6 should be the default these days (though I prefer ZFS RAIDZ2 or RAIDZ3). And don't be one of those idiots who makes a 32-disk, 192 TB RAID5 (or 6 for that matter).

        • Any classic RAID level is useless if you want data safety. So one of your drives in RAID10/5/6 returns garbled data (without an error), which copy/parity do you trust?

          Also many 5/6 implementations won't actually calculate the parity chunk on reads, only for rebuilds. There are some pricey controllers that do full checksumming ala ZFS on chip but as with most hardware systems the SPOF becomes your controller.

          With the drives becoming ever larger and faster, more data is being read but the errors per terabyte

        • by swb ( 14022 )

          Is the rebuild issue for SSD RAID-5 arrays the same stratum of risk it is for spinning rust?

          I would presume not, both because of speed and because there's not nearly as much added stress from the intensive reads necessary to rebuild the array.

          Double parity and/or hot spare is better, but I kind of wonder as SSDs gain write durability (or it becomes more accepted they just have it, as some endurance tests have noted) and they start popping up in more budget minded arrays if maybe RAID-5 might make a comeback

          • by Bengie ( 1121981 )
            Correct. SSDs fail for different reasons than spinning rust. Most mechanical HDs fail for physical reasons and physical reasons tend to be highly correlated for all drives in an array, even if they're different models or even brands. There is a very high risk that if one drive fails, another is right behind it. RAID5, I'm looking at you.

            RAID0 is any drive failing is a loss, so multi-drive failures don't matter so much, but they're also much less likely until it's a firmware bug or other pathological issu
      • I have found these calculators work well for projecting the performance and capacity of various RAID levels:

        http://wintelguy.com/raidperf.pl/?formid=1&raidtype=R0&ndg=2&ng=1 [wintelguy.com]

        http://wintelguy.com/raidperf.pl/?formid=2&raidtype=R0&ndg=2&ng=1 [wintelguy.com]

        Some other guy mentioned RAID 10 isn't a backup strategy; he's correct (no RAID level is), however one thing to keep in mind is that when his RAID 0 array dies, he'd better hope his back-ups are all up-to-date and restorable. With just about a

    • Not smart to run your OS on RAID 0.

      Why? You're assuming the OS is something that I can't just re-install? Remember that you're only slightly more than doubling the failure rate. Given the incredibly low failure rates it's not like you're guaranteed to lose things constantly.

    • by aralin ( 107264 )

      Most SSD do RAID internally across several dies already.

    • RAID0 is unsuitable for situations that require very high uptime but there is nothing inherently dangerous about storing real data on a RAID0 array. I know this gets said frequently but, RAID, at any level, isn't a backup. It's a reliability/performance feature. Even if you had configured these three disks as a triple mirrored RAID1, you would be insane to not run SMART monitoring tools on the disks and even more insane to not have good backups. I don't know if I've ever had a disk fail without plenty o

      • Ummm...ok. So when your SMART detects a failing drive in your RAID0 array and you decide you want to replace it, how do you do that exactly? Oh, that's right, wipe the entire array and restore from backup, which, depending on the size of your array can take anywhere from several hours to days, more if you decided to use your array to run the OS as well. RAID0 is just a plain terrible idea, period. It doesn't matter if you don't think you need uptime, an N disk RAID0 is N times more likely to fail catastroph

        • It depends on how you've setup your RAID0 array. If you are using mdraid, you can simply take the array offline, dd the contents of a failing disk onto a new disk, remove the old disk and bring the array back online (you can do this with a USB boot stick if you need to take the root filesystem offline). That's certainly more work than popping a failing disk out of a hotswap bay, screwing a new disk into the drive tray and pushing it back in. But, it's not that prohibitive.

          Now, having said that, I certain

        • What do I do? Well, I run a RAID-0 of SSD for my OS. I drop both the failing drive and a new drive into my drive duplicator, hit a button, and approximately 5 minutes later I put the new drive into the box and it's running again. That is of course if SMART detects it before failure which is ~50% of the time. Otherwise I wipe and restore from backups.... I'm guessing about 2-3 hours as I haven't had to do it yet.

  • What PC Perspective proved here is that users often claiming that RAIDs "feel faster" despite a lack of bandwidth result to prove it, are likely correct. Measurements now show that the latency of IO operations improves dramatically as you add drives to an array, giving a feeling of "snappiness" to a system beyond even what a single SSD can offer.

    So, basically it partly takes seek time out of the equation, or something similar?

    Because then in theory I guess you can be serving multiple requests instead of ju

    • by AllynM ( 600515 ) *

      That's pretty much it. The trick was showing it properly, which has not previously been possible without our new test method.

      Allyn Malventano
      Storage Editor, PC Perspective

  • by Dr. Spork ( 142693 ) on Monday February 01, 2016 @02:17PM (#51415593)

    I've been building PCs long enough to remember a time when things were improving so quickly that it made no sense to keep a computer for more than 4 years. But since then, the progress in CPU performance has reached a plateau. People like me, who bought a good Sandy Bridge system in 2011, still have a system that doesn't come close to feeling crippled and lazy. We don't have much reason to envy the people who bought the latest generation of i5/i7 systems. Five years used to mean an order of magnitude improvement in performance. Now it's not even a doubling. I've sometimes wondered when I will finally start feeling the urge to upgrade my system.

    These SSD latency numbers are the first thing I've seen that gave me the feeling that there is some truly worthwhile trick that my present computer can't come close to matching. I'm not saying that I now want to upgrade, but on reading this, I have become upgrade-curious for the first time in many years.

    • by Anonymous Coward

      You're right. I/O is where the best improvements have been coming lately.

      I'd still be using a Core2Quad if it wasn't for the platform's outdated I/O features.

      PCI 3.0, SATA 3, USB3, DDR4, and surrounding technologies like M.2 and NVME are the the real reason to upgrade. The newer platforms are faster in most practical applications simply because they can feed the data to the CPU faster.

      Intel's also been focusing on power saving. The new parts sip power in comparison to the decade old core2.

      • To bad the skylack only has 16+4 PCI-E 3.0 lanes from the CPU. That why intel needs to put QPI in all cpus and drive the chipset off of QPI and not DMI.

      • by Gondola ( 189182 )

        I can totally agree with these sentiments. My first computer was a Commodore 64. Since then, I've been chasing the performance dragon, upgrading to a new computer every couple years. C128, Amiga, XT, 286, 386, 486, Athlon, P60, P2, P3, P4, P5... up until my last two computers.

        My previous PC was a Core2Duo e6600 (circa 2006) and its performance was great and then still good. Had it for 5 years. I assembled my current desktop, a Core i7 2600k, in mid 2011. I've upgraded the video card (I'm a gamer) a couple t

  • Questionable Results (Score:5, Interesting)

    by tempmpi ( 233132 ) on Monday February 01, 2016 @02:18PM (#51415599)

    These results seem to be very questionable. Their graphs claim that in some configurations almost all 4k read requests are handled within 100 ns. But getting even a single DRAM burst from a random DRAM location already takes almost 100ns, even through the memory controller is connected with a much tighter interface, optimized for low latency and PCIe is much slower than DRAM interface. Even without overhead 4 PCIe 3.0 lanes ( 8 GB/s) can only transfer 8 KB per s. Transfering a 4 KB Block should thus take at least 0.5 s or 500ns and that does not include any overhead nor the time needed to actually send the request to the SSD, open the page from the NAND flash, run ECC and decompression.

    • by Fwipp ( 1473271 )

      I think they must have made a scale error when rendering their graph. In the article they introduced this latency tool in (Oct 2015); you'll see that even the best PCIe SSDs tested had their 85th percentile at 100 microseconds: http://www.pcper.com/image/vie... [pcper.com]

      Additionally; the far right of the graph in this article is labeled as "1ms", when if the scale was reasonable, it'd be 1us. I suspect they just forgot about microseconds when doing their conversion.

      They also cite a 6ns overhead due to the RAID contro

    • by Junta ( 36770 )

      Even without overhead 4 PCIe 3.0 lanes ( 8 GB/s) can only transfer 8 KB per s.

      Did you mean 8 KB per ms? Rinse and repeat for 0.5s/500ns being equivalent.

    • by AllynM ( 600515 ) * on Monday February 01, 2016 @04:03PM (#51416375) Journal

      Yup, we had a scale error as our Excel-fu was not as strong as we'd hoped when we made the custom format for the axis, and I totally fell for the error. I've updated the article with corrections.

  • by Chas ( 5144 )

    That Gigabyte board is looking more and more attractive.

    And I'm actually due for a system rebuild this year...

  • by jon3k ( 691256 ) on Monday February 01, 2016 @02:35PM (#51415743)

    PC Perspective's new testing demonstrates the triple RAID-0 array having just 1/6th of the latency of a single drive.

    That was with a queue depth of 16. Not exactly representative of a normal desktop user.

    • by AllynM ( 600515 ) * on Monday February 01, 2016 @04:06PM (#51416381) Journal

      PC Perspective's new testing demonstrates the triple RAID-0 array having just 1/6th of the latency of a single drive.

      That was with a queue depth of 16. Not exactly representative of a normal desktop user.

      It's reasonable for peak power user load. Folks running / considering triple SSD RAIDs are not exactly 'typical desktop users' :)

  • A RAID0 of 3 SSDs will be faster than a single SSD drive for multiple reasons, the primary one being that the kernel reads from all three devices at the same time, and (secondarily) both the SSDs and the kernel are doing read-ahead, so that once hundreds (if not thousands) of sectors are in memory, you're only looking at the time to copy them into the destination buffer. For more speed, set your read-ahead buffers up - /sys/devices/(device)/hostX/targetX:0:0/X:0:0:0/block/sde/queue/read_ahead_kb, in Linux.
  • Let me get this straight. When you have more devices available to service read or write requests, the time that it takes to service the request goes down.

    What next? Are we going to be told that RAID5 gives better read performance than a single drive too?

  • by Gravis Zero ( 934156 ) on Monday February 01, 2016 @07:24PM (#51417787)

    The gang over at PC Perspective...

    gangs, presumably roving, are taking over websites now! YOUR SITE COULD BE NEXT!

  • There are several things that affect the latency. You will get about 200us latency on program on a die, but that can be reduced by using some caching and acknowledging the writes before the program finishes, but that cache can be saturated by sustained writes. Especially with random writes that get high level of map updates. As that fills the write latency on sustained random writes will eventually climb to the 2 program instructions latency, which is 400us. With multiple controllers, only every third write

All extremists should be taken out and shot.

Working...