Forgot your password?
typodupeerror
Data Storage Power Hardware

How Power Failures Corrupt Flash SSD Data 204

Posted by Soulskill
from the not-so-solid-state dept.
An anonymous reader writes "Flash SSDs are non-volatile, right? So how could power failures screw with your data? Several ways, according to a ZDNet post that summarizes a paper (PDF) presented at last month's FAST 13 conference. Researchers from Ohio State and HP Labs researchers tested 15 SSDs using an automated power fault injection testbed and found that 13 lost data. 'Bit corruption hit 3 devices; 3 had shorn writes; 8 had serializability errors; one device lost 1/3 of its data; and 1 SSD bricked. The low-end hard drive had some unserializable writes, while the high-end drive had no power fault failures. The 2 SSDs that had no failures? Both were MLC 2012 model years with a mid-range ($1.17/GB) price.'"
This discussion has been archived. No new comments can be posted.

How Power Failures Corrupt Flash SSD Data

Comments Filter:
  • by X0563511 (793323) on Friday March 01, 2013 @06:12PM (#43049765) Homepage Journal

    Seriously... slap in some basic power circuitry and some caps - enough that the drive can finish the cycle it is on and do whatever it needs to do to power off safely.

    • Re: (Score:2, Insightful)

      by Anonymous Coward

      I'll quote the great CliffyB: Vote with your dollars!

      What? It's valid thinking, not at all 9:th grade.

      • by AliasMarlowe (1042386) on Friday March 01, 2013 @06:41PM (#43050073) Journal

        We encountered extensive and progresssive file corruption on SSDs in an industrial device. It used the FAT file system, and after every loss of power, it ran its equivalent of chkdsk/f at the next boot. If power was lost again while this command was running, then it was guaranteed that the file system would become corrupt (despite the fact that we were writing nothing to the SSD; it held only files which were opened for reading). The window of opportunity was described as "very short", and the possibility of corruption was "very small" according to the vendor. In our experience in the field, and in our internal testing, the window of opportunity exceeded 20 seconds, and the possibility of corruption was "utter certainty".

        The vendor fixed the problem in a very easy way. They changed the file system from FAT to a commercial journaling FS. In our subsequent tests, we never found any file corruption, even on iterated power loss at random intervals after power on.

        • by TheRealMindChild (743925) on Friday March 01, 2013 @06:52PM (#43050199) Homepage Journal
          First, running an SSD on an "industrial device"

          Second, using FAT

          Third, "commercial journaling FS". What does that even mean?

          If you are industrial, where is your UPS?
          • by yurtinus (1590157) on Friday March 01, 2013 @07:13PM (#43050391)
            Likely as part of an embedded system - monitoring or control software. Systems where you just flip the power switch on when you need them and off when you're done, so an UPS wouldn't apply.

            I'm not saying their implementation was right, just saying that you can't imply from his post that it was wrong :P
          • Re: (Score:3, Informative)

            by thejynxed (831517)

            If it was a drive being used to read schematics for CNC for instance, there isn't a manufacturer out there that currently offers a machine-tied UPS for the CNC machine. If the CNC machine loses power, then so does the drive, and vice versa, since it's all on the same circuit (usually you'll find the power stuff hidden in a cabinet along a nearby wall, and that stuff takes power directly from the mains).

        • by certsoft (442059) on Friday March 01, 2013 @06:58PM (#43050289) Homepage
          We use USB flash drives for a data logger. Most of the time the data is being buffered in the ARM based Linux board's RAM to save power. Once we get a complete file's worth (4MB at the present) we power up, validate, write the file, and power down. Supercaps have been a lifesaver. There's even enough capacity to do the write cycle if the flash was powered down when a power fail is detected. That allows to not lose whatever what was already in the RAM buffer.
          • by Darinbob (1142669) on Friday March 01, 2013 @07:48PM (#43050715)

            I hate a lot of USB drives and CompactFlash. They're all designed as dumb commodity devices for the undiscriminating user, and trying to get any solid spec sheets out of the manufacturers is impossible if you're not also a giant corporation. Instead their data sheets are just marketing literature (you rarely get anything more technical than "8x speed"). Almost all are designed to work with Windows with no concern to work with embedded systems or production automation, etc. So you end up buying a wide variety to test with and see which ones are barely adequate to work with your system.

            • by certsoft (442059)
              Fortunately the client has facilities to test various drives over a wide temperature range (down to -40, not sure how hot they test) while running. And yes, a lot of them are crap.
      • by Dunbal (464142) *
        The problem with voting has always been that the idiots get to vote too. So while you might "vote with your dollars" to select the most reliable drive, they will vote for the one with the cute name, or the shiny case, or the "free gift", or the special price, etc.
    • by v1 (525388) on Friday March 01, 2013 @06:23PM (#43049853) Homepage Journal

      space is at an extreme premium in those drives. There's a reason they feel so heavy/dense. Given the quilting layout of the chips, adding a single cap would prevent several memory chips from fitting. So you may as well then fill that remaining space with more caps. But you will reduce capacity, and that's what sells SSDs.

      There's already a substantial amount of circuitry in them, far from "basic". It's essentially a CPU. I'd be interested to see some numbers as to average power drain during idle, read, and write.

      The ones that did the best during the power blips probably did have caps and a bit more in their power system to handle it though. It certainly does surprise me that the mid-range, not the high-end, were the best performers in this test.

      • space is at an extreme premium in those drives. There's a reason they feel so heavy/dense.

        I don't know what SSDs you've been using, but I've never picked up an SSD (OCZ Vertex 2/3, Intel X25-M/320/330/335/510/520) that didn't feel light and sound nearly hollow.

        • by Mashiki (184564) <mashiki@gmail. c o m> on Friday March 01, 2013 @07:05PM (#43050341) Homepage

          I don't know what SSDs you've been using, but I've never picked up an SSD (OCZ Vertex 2/3, Intel X25-M/320/330/335/510/520) that didn't feel light and sound nearly hollow.

          Consumer drives are usually lightweight, they don't need the extra cooling. Enterprise drives depending on who they're made by and what they're for can have heatspreaders or heatsinks within, or attached to each chip adding to the weight.

        • by thegarbz (1787294)

          but I've never picked up an SSD (OCZ Vertex 2/3, Intel X25-M/320/330/335/510/520) that didn't feel light and sound nearly hollow.

          Rip it open and have a look. There's not much weight at all to a piece of fibreglass and some plastic resin encasing some silicon. Circuit boards and components are really quite light when they don't require cooling or even large bits of metal for simple thermal mass.

          You'll find that even though it's light and looks hollow it'll be packed quite full. Now combine that with the problems associated with creating some form of energy storage. Storage can come in some electrical form, i.e. battery which would be

      • There is all kinds of extra space in a 2.5" SSD. They have a lil' CPU, some flash chips, and that's it more or less. They are quite small. In smaller form factors, then ya space can become an issue but there's plenty in a 2.5" unit.

      • by AmiMoJo (196126) *

        Most SSDs are 2.5" so there would be plenty of room for a large capacitor or small battery. You really don't need a lot of energy to finish flushing a small RAM buffer.

        • by edmudama (155475)

          Most of the enterprise grade SSDs on the market that are outfitted with power-loss protection circuitry fit these capacitors within the 2.5" form factor.

      • >space is at an extreme premium in those drives.
        So put them in a desktop drive form. The first thing I do with SSDs is put them in one of those adaptors to make them fit in a normal drive tray.

    • by Guspaz (556486)

      Most enterprise SSDs do have small supercapacitors or capacitor arrays onboard for exactly this reason. Some of the higher-end consumer drives do too. But most consumer drives don't.

      The answer? Get a UPS.

      • by sjames (1099)

        The answer? Get a UPS.

        Because those never fail.

      • by dgatwood (11270)

        The answer? Get a UPS.

        You're assuming a desktop-sized drive in a desktop computer, yet nearly all computers sold today are portables, and laptop users are more likely to buy bus-powered external drives than mains-powered drives.

        So the five most likely causes of power failure in a consumer hard drives (and presumably, in the future, SSDs), ordered from most likely to least likely, are probably:

        • Somebody yanking a USB cable before the device is fully unmounted.
        • The laptop's battery dying earlier than expecte
      • by AmiMoJo (196126) *

        Or maybe attach a capacitor or battery to the power connector (with diodes so you don't try to power the entire PC).

      • Most enterprise SSDs do have small supercapacitors or capacitor arrays onboard for exactly this reason. Some of the higher-end consumer drives do too. But most consumer drives don't.

        The answer? Get a UPS.

        A UPS is no panacea: I experience grid failure very rarely.

        However, relatively speaking I experience many more kernel lockups that require an ACPI-initiated poweroff by holding down the power button until the machine abruptly powers off. What do you do when a reboot/poweroff command causes your Linux/BSD machine to hang? I/O handle leaks in the Samba SMB client (ie. *not* the smbd daemon) and the Samba Winbind code are notorious for this. The only times I have ever had to "yank power" from a production Linu

        • by adolf (21054)

          Instead of a supercap I'd rather there be a couple of replaceable Lithium coin cells inside of an SSD, to just finish finish writes after the power unexpectedly dips for some reason. They're cheap commodities, they seem to have predictable failure rates, and I don't remember the last time I changed one in any computer (though it used to be a fairly frequent repair).

          By using them only once in a blue moon and occasionally monitoring the voltage and setting a SMART error if they're getting worn out, I'd estim

        • by Vairon (17314)

          Assuming you have sysrq keys enabled, you can hit alt-sysrq-s, wait for the sync to complete, alt-sysrq-u, alt-sysrq-b. This performs a filesystem sync then remounts all filesystems read-only then boots the system. Also if you have a stuck mount point you can always use a lazy umount (umount -l) to remove it from filesystem hierarchy so you don't need to reboot in the first place.

    • by WillgasM (1646719)
      You would think. The only SSD I'm running is on my computer at home and my house is sufficiently UPS'd. It's always cool when the power goes out at my apartments but all my electronics keep going. I just wish there was a battery on that Time Warner box outside my door.
      • by PRMan (959735)

        I just wish there was a battery on that Time Warner box outside my door.

        Strange. My DirecTV DVRs just keep on working...

    • That was my first thought as well, throw in one supercap and you'll solve this problem.

    • enterprise-class SSDs have capacitors designed to last long enough for the SSD to finish any writes if the power fails.

      Capacitors cost money though.. so this is one of the things that gets stripped out of consumer-level drives to reduce the price.

    • by hairyfeet (841228)

      Uhhh...we solved this problem ages ago with UPS. If you care about your data put the machine on a UPS. I've had my business customers on UPS systems for years, showed them how to test the batteries and swap 'em when they get worn out, no problems. I just had to swap the PSU and HDD out of my netbox at the shop because a transformer blew on my block and managed to give the old gal enough of a shock even through a surge protector that it cooked the PSU and the HDD, but since its just a netbox I don't care eno

      • I've had lots more failures due to UPSs going tits up than through data loss on SSDs.

    • Seriously... slap in some basic power circuitry and some caps

      A small, stupid, retro NiMH battery might work even better.

    • by Darinbob (1142669)

      You can design the firmware to avoid corruptions as well, it doesn't need a hardware solution. The manufacturers just have to be aware that power failures will occur and take that into account during the design. Extra capacitors won't fix the problem of shortcuts in the design.

    • by gweihir (88907)

      High-end SSDs have supercaps for that. Low-end SSD customers are to cheap to pay a few USD/EUR more for the added protection.

  • Before you ask. (Score:5, Informative)

    by eddy (18759) on Friday March 01, 2013 @06:12PM (#43049767) Homepage Journal

    The paper doesn't disclose the brands.

    • Of course it doesn't. Naming/Shaming is not allowed.
      I was sarcastic, of course. They don't do it, though, because it'd probably put them in a crossfire of lawsuits coming from powerful companies. Nobody wants that. They will lose simply by being bullied financially. It's all about who brings more lawyers to the table, not who's right or wrong.

    • MLC == Intel. But they were the good ones.

    • by greg1104 (461138)

      I created a Reliable Writes [postgresql.org] page for PostgreSQL that talks about this and gives some known good and bad examples. Intel's 320 and 710 drives are the only two SATA SSDs still on the market that have survived the tests for clean shutdown I've advocated everyone run. They are units with a supercapacitor to enable power failure cleanup. If a drive doesn't have a battery for that sort of purpose, you will lose data at shutdown one day. And, no, a UPS is no cure, because all it takes to ruin a system on one is

    • by DragonTHC (208439)

      Which makes it completely useless for 90% of us who just wasted our 3 minutes.

  • Last time I checked, standard platter-based disks had the same issue -- a problem that is solved in server/enterprise environments by placing a write-cache battery in the RAID controller.

    In a desktop environment I suppose one could embed a write cache battery into the SSDs to abate the issue, but in a laptop environment it'd be unlikely you'd even encounter it since you'd have to be writing data while running out of battery, in which case, you might well deserve it :)
    • A capacitor could hold enough power to finish a write cycle on SSD no problem. It wouldn't even have to be very large.
      • True, but, then when the cap dries out and eventually bursts open it'd probably be a major cause of drive failure and lack of longevity.
    • OR the battery fails, is taken out, or falls out.

  • by preflex (1840068) on Friday March 01, 2013 @06:23PM (#43049855)
    ... Power failure corrupts absolutely.
  • by rossdee (243626)

    Why should a power failure corrupt anything? The UPS will shut the computer off if there is a prolonged outage.

  • Unsurprising (Score:3, Insightful)

    by Anonymous Coward on Friday March 01, 2013 @06:27PM (#43049893)

    These devices have an elaborate internal database for the management of block remapping. For this to survive power failures it needs to use transactional updates. Getting this right is hard - it takes years for file systems and databases to become robust. I'd guess that many devices don't even attempt to do it and the ones that do probably have obscure failure modes. A UPS is essential.

  • by Dishwasha (125561) on Friday March 01, 2013 @06:28PM (#43049907)

    I had some original Vertex drives from OCZ that kept absolutely corrupting when my laptop got accidentally unplugged and I powered on the machine. I had to RMA them over and over and over again. I finally figured out that my battery was getting old and, although everything was functional even on battery power and it would boot, the initial large draw of power on boot must have created a voltage drop (i.e. brownout) which the SSDs weren't designed to compensate for. Within an hour of boot (even back on plugged power) they would choke, freeze the OS, and be rendered unusable from then on out.

    Several SSD manufacturers are probably not engineering well for fluctuating power. Rather than fixing the problem with better engineering, OCZ simply changed their warranty policy to void the warranty if the customer is not providing proper power which, correct me if I'm wrong, I don't think rotating disk hard drive manufacturers have had that in their warranty clauses.

    • Re: (Score:2, Insightful)

      by citylivin (1250770)

      Well thats probably becuase you were using OCZ crap. I have never had a quality product from that company.

      However that said, I have noticed the same thing with the crucial m4s I have. In one particular laptop, it keeps bricking drives becuase the battery doesnt hold much of a charge any more. Luckily, i can "unbrick" them by plugging in the power (but not data) for 20 minutes, then plugging in the data connection, then rebooting the machine. Has worked more than once.

      and crucial has put out a bunch of firmw

      • The OCZ Vertex 2 series were prone to spontainiously bricking. The original OEM Samsungs that Dell used would exhibit all sorts of strange disk I/O issues that led to strange Windows hard locking. You would think it would be causes by CPU, RAM, Video, or chipset. No, it was the Samsungs.

        When it comes to reliability, I'll stick with Intel offerings. Not the fastest things on the block, but good enough in its class. Personally, I gambled on a OCZ Vertex 4 because Newegg slashed some insane 30% off MSRP via Sh

    • I think you make a good point about warranty clauses, and it would be hard to imagine HD manufacturers singling out their SSDs with an inferior warranty in this respect.

      Considering the paper cited by TFA won't spill the beans on which models were tested, it may be a safer bet to purchase SSDs from traditional HD makers (at least I hope that is the case with my Samsung).

    • by ckthorp (1255134)
      I had some fun with trying to mount some Crucial M4 drives in USB external enclosures. They kept getting unmounted and the SMART block remap count kept running up, and up, and up. One of the drives outright failed and the other was at 55% spare sectors remaining when I figured out the issue. When there was a write, the current consumption from programming the FLASH chip would cause a voltage sag and the write would fail but it wasn't usually enough of a drop to make the drive reset. Once I bought the "Y
  • by citizenr (871508) on Friday March 01, 2013 @06:52PM (#43050201) Homepage

    Useless paper/test.

    • by Theovon (109752)

      If they do that, they won't get any more free SSDs to test, and that'll impact their ability to write papers criticizing SSDs. What would you prefer? A paper biased towards SSDs too small/cheap to be useful to you, or one that doesn't name names? Anonymity is VERY important in this kind of research.

      • by citizenr (871508)

        If they do that, they won't get any more free SSDs to test, and that'll impact their ability to write papers criticizing SSDs. What would you prefer?

        I would prefer research to be done by someone who is not manufacturers bitch.
        You dont need a ton of money to test commodity hardware, the trick is to SELL stuff after the test, not take home and pretend it wasnt a bribe.

  • by h8sg8s (559966)

    What some of folks don't realize is its the seesaw nature of many power events that's primarily behind both data corruption and SSD failure. It's a rare rack system that has its own power conditioning and UPS these days (HP NonStop comes to mind) and without it you're subject to whatever the event provides in the way of under/over voltage, spikes, drops, etc. Many times these happen in timeframes too fast for power switching equipment to react and in some cases its that stuff that gets fried first.

  • There is a protection mechanism that I know exists in Crucial SSDs which makes the drive appear dead after some unclean shutdowns of the drive while it performs a firmware-level integrity check of the drive. It may exist in other brands as well. Sometimes it takes 2 runs of 30-60 minutes to get the drive to re-enumerate via SATA. I'd be curious to know if the "dead" drive was affected by this bug.
    • by drinkypoo (153816)

      There is a protection mechanism that I know exists in Crucial SSDs which makes the drive appear dead after some unclean shutdowns of the drive while it performs a firmware-level integrity check of the drive.

      I don't know if they're violating a spec or not and it's probably a life's work to find out, but that seems very rude to me. They really ought to identify as busy or something, so that they don't just scare the piss out of you. If you almost-brick an Xperia phone by scragging the bootloader so bad you can't even reflash it, whatever handles the comms is still working and lurking in the background and it will enumerate via USB with the service interface. That way you know whether you should even bother. Woul

      • by ckthorp (1255134)
        I agree. The first time one of our engineer's laptops HD's did this, it was rather uncomfortable to say the least. I think a good compromise solution would be to have it enumerate with a "useful" drive textual model identifier like "M4 ERROR CHECKING, LEAVE ON 30 MIN" or some such. I'm sure it violates some standard, too, but it would at least give the user some indication of what is happening.
    • You got this too? I just ordered a Crucial M4 on sale a few weeks ago. the day after I installed and cloned it, I had the same situation where it wouldn't start. I called Crucial, expecting to need an RMA. Luckily I got an informed gentleman on the phone who told me to leave it at the failed POST screen for 20 minutes, reboot, and give it another 20 minutes, and reboot again. It worked. Supposedly it's not so much a 'bug' as an 'obscure feature'. ...I'm keeping my spinning rust drive around just in case.

      • by ckthorp (1255134)
        Overall, it is a good thing. The data isn't organized linearly for wear leveling purposes, so a power outage can leave the metadata in an inconsistent state. Also, make sure you have the latest firmware on the drive. They had a fun one earlier that caused a drive lockup hourly after the power on counter hit about 35k hrs (or some such). I've got about 2 dz M4 drives in service, so I've seen a lot of the bugs.
  • by thue (121682) on Friday March 01, 2013 @07:16PM (#43050417) Homepage

    This is old news; see fx Wikipedia's coverage [wikipedia.org]. Only buy SSDs with a battery or capacitor, or whatever is the in DRAM cache of the SSD will be lost on power failure.

  • My Personal Policy (Score:3, Insightful)

    by wisnoskij (1206448) on Friday March 01, 2013 @07:54PM (#43050765) Homepage

    This is why I don't use prototype tech that is really not ready to be used in the real world. And if you do, expect loads of bugs and bricking.

    But either way, thanks for funding the development of something I am excited to try out in 2-4+ years when it will be a mature usable technology.

    • by edmudama (155475)

      SSDs are way past prototype technology at this point. The products from high quality vendors are both fast and robust.

  • Not occur anywhere on this page?

  • by rabtech (223758) on Saturday March 02, 2013 @02:00AM (#43052523) Homepage

    Power loss protection (super capacitors) was stated on four of the drives (the four least expensive to boot). Only three performed flawlessly in the unserialized writes test. Those aren't great odds. In fact only two drives passed all tests with no errors, and it wasn't necessarily the SLC "enterprise" drives, though those two also passed the serialized writes test.

    In case you aren't aware, unserialized writes invalidate *every* assumption, including write ahead, journaling, even your fancy BTRFS/ZFS. His example is a database where the transaction log write was sync'd before the data page write, then after a power failure the data page is persisted but the log write is gone.

    You can recover from many of the other errors or at least detect them but unserialized writes can silently corrupt data or even ruin the entire filesystem.

    Obviously the metadata/dead failures are the exception... Those render the whole SSD useless.

What this country needs is a good five dollar plasma weapon.

Working...