How Does Flash Media Fail? 357
bhodge writes "Aside from the obvious 'it stops working' answer, how does flash media — such as USB, SD, and CF — fail? Unlike with traditional hard drive, where anyone who's worked with computers for a while knows what a drive failure looks like, I don't know anyone who has experienced such a failure with flash. I've haven't been able to find more than scant evidence of what such failures look like at the OS level. The one account I have found detailed using a small USB drive for /var/log storage; it failed very quickly, and then utterly (0 byte unformatted device), after five years of service in the role. This runs contrary to other anecdotal claims that you should still be able to read the media after you can no longer write to it. So my question is: what have you seen of the nature of flash media failure, if anything?"
Burnt out (Score:3, Informative)
From what I gather, the most common cause of failure is the flash getting fried. Dodgy card readers, pulling the card out when a voltage is running through it, the chips are very sensative to spikes in current or voltage and burn out because of it.
Failure to Write (Score:5, Informative)
Had two finally wear out. Both started giving "could not write to device" sort of errors. The system (Windows 2K or XP) would still recognize the drive, would show the files, etc. Indeed, I could still access (read) the files, so the data was there and copyable. But I'd get a file write error every time I read anything, because Windows was trying to update the flash drive's file directory with "last accessed" or some such, and that write would fail.
No biggie; copied the data to a replacement, threw the old ones away, after hitting them several times with a hammer to "clear" the memory :-)
Flashmemory (Score:4, Informative)
As I understand it aluminium alloys created without lead and then used in computers degenerate several magnitudes quicker than alloys with lead. The process is apparently that the aluminium start sprouting tiny tiny "hairs" and when one of these connects to another one of these coming from somewhere else in the machine then it's thank you and good night for that part.
Anyway the reason I mentioned this is because apparently with intensive use 5-7 years is how long parts in your computer takes to make a connection and after that it is LED OFF (see what I did there?) Of course unless you have a computer constructed before the mid nineties (I think that was the point); since they use lead in their alloys this isn't something that will affect them (though a range of other issues will).
FAT (Score:5, Informative)
Without knowing more about this specific situation, I'd say this failure sounds like it pre-dates wear leveling. Prior to wear leveling, the most used sectors were likely to fail the fastest. And what sector gets written to more than the file allocation table?
If the file allocation table was lost, that would explain why the device became completely inaccessible. The card might not be a total loss if the card contains firmware or circuitry to remove bad blocks from usage. In that case it might be possible to reformat it. (Of course, if it lacks wear leveling I wouldn't count on it.)
Wear leveling neatly solves this issue by shifting writes to different free blocks with every write. This assures that the maximum use of the card is obtained prior to failure. Should any given block fail the card will detect the checksum error, mark the block as bad, then attempt to rewrite to a different block. This is communicated back to the reader in a transparent way. As far as the reader knows, nothing happened.
As you can imagine, wear leveling makes it incredibly rare to see Flash failures these days. It can still happen, but the results are likely to be unpredictable. The card will need to chew through all free blocks before it starts returning errors. In that case you may be able to continue reading the media. Or it may fail like the USB drive you mentioned. It all depends on the importance of the block on which the erasure was attempted. Since you only know about a failure *after* the block erasure, you're at the mercy of the quality of the card's electronics and algorithms to protect against a dangerous erasure.
like a CPU (Score:3, Informative)
I've been booting linux servers off of flash for a few years. For some of them, the whole OS, even /var/log, is on the flash drive.
I've had one drive fail, and it basically got hot and stopped being recognized as being connected by the computer. It was older generation technology, though. Newer flash technology designed for computers doesn't fail, as far as I have experienced. I'm talking about the flash SATA drives from name-brand manufactures.
Flash mail server (Score:4, Informative)
I had a 4GB FAT32 flash drive that I used as storage for a mail server attached to an OpenWRT router. It required renaming and deleting files all the time (every time it got an e-mail)--so I think it wore down pretty quickly.
One day, the storage for the flash drive stopped working (from one hour to the next, without being touched, the computer acted like I had just yanked the drive out)--it would be recognized but report a "no media in drive" error when you tried to access it, like an empty CD drive. In fact I think Windows would say "Insert CD" or "No disc in drive F"
Re:Flashmemory (Score:5, Informative)
Re:Was there a point to this article? (Score:5, Informative)
Comment removed (Score:5, Informative)
gracefully... (Score:2, Informative)
Re:In my case (Score:5, Informative)
Corrosion.
Being repowered while the internal circuit board is still damp with soap-contaminated water (shorting).
Physical stress ("agitate" cycle, "spin" cycle, Tumble Dry...).
Heat stress (which heat cycle did you use/did it go through the dryer too).
Need I go on?
CF (Score:5, Informative)
Big mistake.
The install took a whole day. I happily ran some programs the next day and crash - kernel screams of i/o errors in the swap partition.
Formated the card MS-DOS - it found a few bad sectors. Then i ran Norton Disk Doctor and at every run it was founding more and more bad sectors. But each time i was re-formating the card using a camera, the bad sectors were shifting around. Unusable.
FYI: IBM PC110 is a 486 Palmtop with a CF slot to be used as hard-drive. The CF interface is IDE.
Re:FAT (Score:5, Informative)
Most devices that do hardware wear leveling are not power fail safe. And get corrupted beyond repair, random data corruption may follow, or an unreadable device.
(I've done extensive testing with SD and Compact Flash devices in power fail cases. Because not all manufactures deliver what they promise)
The short answer... (Score:3, Informative)
Your flash memory is fine, the controller is hosed.
This kind of (essentially unrecoverable) failure will continue to be an issue wherever the logic is integrated with the storage.
If it's any consolation, except for those who are always forgetting to "eject" or turn off their device before removing the media this kind of failure should be quite rare*.
Enjoy.
*Mfr's producing shoddy products not withstanding.
Depends on the Filesystem I suppose (Score:5, Informative)
On a modern filesystem, your writes should essentially be atomic and in theory it shouldn't be possible to leave the drive in an inconsistent state when the write fails.
Of course most camera memory cards end up being formatted with fat32 which can be a little less forgiving.
Re:Fail on write (Score:4, Informative)
It just seems like the traditional drives only fail on reads: they mostly do reads, so when they fail, it's more likely on a read.
I've had many a drive fail during writes though, usually at the worst possible time (deadlines, when the machines are getting read/write hammered, and then bam, drive goes down and RAID performace goes to shit, and people start whinging.)
I've had flash drives die all at once. It's not the norm, but there are things that can happen that will take them from "fine" to "dead" with no steps in between. Usually it's thumbdrives that that happens with; I haven't had a full flash harddrive fail at all yet, so I don't have any insight there.
Re:Burnt out (Score:5, Informative)
I am currently taking a class on solid state devices, and we just talked about how MOSFETs would fail. Basically, a high voltage to the gate would create these electrons that have so much kinetic energy that they create pairs of opposing charges (electron-hole pairs) in what was supposed to be the insulator. These pairs of charges would create an internal electric field inside the insulator. This process reduces the barrier for tunneling to occur, so more electrons are able to tunnel through the insulator and do the same thing, creating a runaway effect.
For more information, look up "Time-Dependent Dielectric Breakdown" and refer to pages 293 and 294 of Streetman and Banerjee's "Solid State Electronic Devices" (6th ed).
Because people focus on the GB... (Score:5, Informative)
...and quality and longevity take a back seat. So companies stopped offering SLC Flash RAM (+100.000 writes) and only offer MLC (5000 writes), and are now pushing even eight-level MLC, which will be even less reliable than standard 4-level MLC Flash RAM. But who cares, the consumer will be slightly fucked after a while, but that will be much later, after they enjoyed the happiness of getting slightly more GB for their buck.
The only manufacturer that I know of, that is an exception, if Kingston, which still offers SLC Flash products - namely their elite pro line of SD and CF cards, and the Data traveler USB drives. But that's it, everyone else has not completely transitioned to MLC.
Re:Was there a point to this article? (Score:5, Informative)
If a cell fails, you can't read or write that cell.
If a gate fails in a page, you lose access to the page.
If a gate fails in the overall control logic, you lose access to the whole device.
Is there something I'm missing? Did you think there were oil changes or brake shoes? It's one silicon chip with metal on it.
Conceptually at least, there are several parts to worry about:
1 - the OS & storage driver
2 - the USB driver
3 - the flash controller
4 - the flash memory
At the flash memory cell level the usual failures are breakdown of the dielectric materials and trapping charges in the memory cell that prevent an erase from happening and yield 'stuck' cells. This is normal for /all/ flash chips and is why they all have an erase cycle rating. There are certainly more exceptional ways for the chips to fail (soldering, wire bond failure, static damage, etc).
The flash controller is supposed to be doing wear leveling, error detection and correction on the flash, to get around those problems with the flash chips, and also talking USB. These chips usually have a microcontroller in them somewhere, and there's probably bugs in that code, no doubt more in the parts that get exercised the least, like error paths :-)
The OS and drivers just have the garden variety bugs and features that we all know and love...
Re:Was there a point to this article? (Score:5, Informative)
There's no redundancy or self healing in the hardware of a common USB flash stick. The illusion that there is comes from a flash controller chip that does a mapping between disk sectors and flash sectors and shuffles things in and out so you don't notice the failures until it can't compensate for them anymore.
Have done some extensive testing... (Score:5, Informative)
First off, when the actual flash media itself wears out, it takes longer and longer to erase individual sectors.
A flash device such as a USB stick or a CF card is slight more complicated because it has something known as an FTL (Flash Translation Layer). The FTL has the job of implementing the virtual media to flash sector translations, implementing wear leveling, and handling the awkward page erases. (Multiple sectors in a page, but you can only erase full pages.)
The FTL obviously must store some mapping information in the media in addition to your data.
If you start writing flash media, and time those writes, you see an initial rapid growth in the write timing that evetually levels off as the FTL tables swell to their constant operational size.
The over all flash write speed will level off to some average value that follows slow growth over a very very long tail as the media wears.
Early flash chips supported about 10,000 erases per page, and modern chips shipped by Samsung and others support a couple million erases per page. When you consider this is spread over say 4GB of media, you can understand that tail is very very long and flash media are probably comperable to hard drives in their MTBF these days.
Secondly, when flash actually does begin to fail, the media itself tends to exhibit a small number of different symptoms.
The flash may stat to show occasional data corruption when read. You might also have instances where data persists in the media only so long as power is applied. And then of course you have the fact that erases take longer and longer to achieve. Eventually erases or programming start timing out occasionaly.
With the FTL between you and the flash, you don't directly observe these effects. Presumably the FTL is smart enough to try and re-map your data elsewhere. In most cases there's ECC to attempt correction of moderately corrupted data. The real killers are when the data fails to persist after power cycling, when ECC fails to recover critical FTL data tables, or when there are no more spare sectors to re-map data too.
Those first two critical errors are likely to produce the lightbulb effect where your flash card or USB stick one day simply fails to come up when probed after device insertion. In more rare cases, the lack of spares may show up as some sort of reported write failure in your kernel logs assuming the flash device reports proper IDE/ATAPI/??? error data.
One final note -- please don't leave your USB stick inserted in the PC as you power it off! USB ports supply power and use a FET device to control that power. When you turn off the PC, the gates float and significant leakage current goes to the USB device. Some of the cheaper USB drives lack a key resistor that bleads this current away and protects the flash memory chips. This leads to data corruption. I have seen the FTL break in such sticks simply by doing POR on the PC.
Oh...almost forgot. When you put you flash stick through the washer and dryer, always use fabric softner or Bounce strips to reduce the static.
Re:Was there a point to this article? (Score:3, Informative)
Having a broken SD card in my pocket, I will describe how it behaves (which I think is what the article is asking). It is a 1GB SVP.
In Windows (XP and Vista), it asks me to format the drive, chkdsk fails because the partition type is raw. Using recovermyphotos on it I get between 10 and 200 photos found before the card reader decides it is not in their anymore, and I can't recover the ones found (perhaps if I paid I could recover as it scanned).
On Linux cat /dev/sdb returns no media found (I assume this is card to card reader trouble again).
Interestingly, on a different reader that gives IO errors with every other card I use I get the raw partition do you want to format it issue.
The fact that I can't read the drive at all from Linux ended my exploration.
Re:Flash mail server (Score:3, Informative)
After about 6 months of fairly heavy use (with only 32Mb RAM I needed to swap to flash), one day the USB flash drive just stopped working, and it's no longer even detected when I plug it into any system now.
I'd done all the obvious things such as mounting with noatime and have the swapiness to 0, but ultimately discovered that flash really doesn't like being constantly written to.
Fortunately, even large capacity USB sticks are pretty cheap, so they're still quite good for as long as they last.
Quiet failure... (Score:5, Informative)
Fortunately, the drive was mostly used for "sneaker net" use, and did not contain any irreplaceable data. This use exposed the issue quickly too (had it been a backup device, the backup would have been useless and I wouldn't know until I needed it.)
A typical failure was to zip up a software installation on a dev machine, then take it to a clean target machine, where the zip would fail to unpack, or the installer exe, once unpacked, would fail to run with various errors.
I finally got to the point where I simply copied several megabytes of plain text data to the memory key, then copied it back and diffed the files to see the corruption (large areas of nulls, as I recall.)
Never heard a peep from the OS.
It was a 1 1/2 year old Patriot XT 2GB, and, after a couple of emails and a PDF of my NewEgg receipt, a new drive showed up in the mail under the lifetime warranty.
I also had an expensive Lexar CF card for a digital SLR that failed. In that case pictures that I know I took simply weren't on the card... but could be "recovered" with the Lexar utility (along with EVERYTHING else on the card, so it was a PITA.) Since that was nearly $200 when it was new, I figured getting my lifetime warranty honored would be easy, since the cards were down to about $20. No dice. Just got the run-around and finally gave up. Lexar lost a customer.
Re:In my case (Score:5, Informative)
You are only partially correct.
The company I work for sells USB flash drives and going through a washer is rather common and they survive more often than not.
The question is: Did you use soap?
Water is practically harmless if you allow the device to dry completely before using it. The problem is water in washing machines isn't just water, its almost always water AND detergent, and probably some fabric softener as well.
When the device dries, the detergent and fabric softener are left behind and are conductive, not like metal, but the resistance is low enough in the tiny spaces between the pins on surface mount chips to make all the difference in the world.
The main reason devices fail however is simply abuse, or poor manufacturing depending on the device. Most of our returns are due to the USB connector pulling the solder pads off the circuit board because of the stresses during insertion/removal. Sometimes the pads don't come off at the USB connector but the board flexes enough to eventually break the connection at one of the flash chips or the controller. When that happens you go from working perfectly to 0 byte unformatted device in an instant as the controller can no longer talk to the actual flash.
We have on occasion successfully retrieved customer data for them by removing the case from the device and flexing the board while its plugged in to get it to work or if that doesn't work, reflowing the solder where possible. Most of the time, thats all it takes.
The heating and cooling is bad, but its not that bad. The temps in a dryer aren't as bad as one might think. My personal device has been washed and dried at least a dozen times in the last couple of years. When I find it in the dryer I simply pull it out of the case, clean the PCB with some PCB cleaner, let it dry, reassemble and life goes on. If its a good quality device doing it once will probably be okay, but as has been stated, doing it too many times and the heat expansion will certainly come into play and destroy solder joints or start making the board lamination fail.
Now ... don't take that as a recommendation to wash your thumb drives, my stick is trying to get into the record books or something, I think it just refuses to die.
Re:Was there a point to this article? (Score:5, Informative)
This is a silent failure, much like hard drives marking blocks as bad. Capacity is reduced without any obvious signs. Not sure if OS tools can recognize it unless the controller reports bad cells as bad blocks. This will eventually result in "disk full" messages when there appears to be space on the drive. Reformatting won't recover the cells but it will likely result in your OS being aware of the flash's reduced capacity.
Very similar to above, but larger amounts of data. I want to say there's 64 cells to the page but don't take that as gospel.
Hello failed/unreadable/size 0 disk error. The data storage mechanism is intact but there's no way to access them. As people stated above, a lot of the time it is not the failure of a transistor so much as a trace or solder point failing. If you know your device has been abused physically, you can try the low-tech approach of gently squeezing or bending the stick while it's in the USB port (use an extension cable so you don't damage your mobo!!) to try and get the contacts to reconnect long enough to retrieve data. If that fails you can pop the case apart and use a magnifying glass to look for breaks in the solder or traces; if you're handy with a soldering iron you can try to bridge the connection. Again, temporary fix.
Actually most of them are several silicon chips; one controller plus a variable amount of memory chips. The increase in traces and board assembly is offset by the ability to reuse components and the overall design while memory chip prices fall. It also cuts down on the impact of failed chips, since you aren't losing controller+memory for one bad gate on the controller.
Overview (Score:2, Informative)
So, I will pass on what I have discussed with my brother-in-law who is an Electrical Engineer that writes software to test flash memory:
1. Flash memory is built with additional fail over storage (so a 1GB SD card actually has a certain % more memory than 1GB).
When a section of memory fails it is marked bad by the flash controller and some of the fail over memory comes into service (marked bad much like failures on standard hard drives... although I get the impression the flash controller may be the thing remembering it's bad... wasn't clear on this now I have something else to ask him)
2. Flash memory will fail... it can only be written to so many times before it will no longer be able to be written to... and the number of times is definitely not as high as a standard hard drive
So it's likely that you can extend the life of a flash device by writing to it less often.
And, not from my brother-in-law discussions, I personally had a flash drive fail (I was using it as the master copy of documents as I moved data between my work machine and home machine while working toward an online degree). When it failed there was no warning previously. It simply stopped working... wouldn't be read and wouldn't write. I suspect my batch file that performed the backups to it must have written to it too many times (it was a smaller 128MB drive so, considering the above discussion about fail-over memory a smaller drive SHOULD fail faster...)
Hope that helps
Re:In my case (Score:3, Informative)
Re:Was there a point to this article? (Score:5, Informative)
Yes and no. A page or cell failure will result in I/O errors if there are no more spares, and if it occurs during a read cycle, it -should- result in I/O errors for all subsequent reads from that cell or page until it gets rewritten to a new cell or page. If it doesn't work that way, then the device is fundamentally violating the contract between the device and the OS to report all nonrecoverable errors that result in data loss.
Also, while a multi-chip design reduces the probability of a device failing outright, it dramatically increases the probability of a failure. First, using a separate controller significantly increases the probability of failure because instead of having interconnect traces on a slab of silicon that (electromigration notwithstanding) almost never change or fail if they work from the factory, you have solder joints exposed on a circuit board. Solder joints are the most common cause of circuit failure in my experience.
Even ignoring the increased risk of having extra solder joints between the controller and flash parts, the odds of failure are still much worse for multi-chip devices. Remember your RAID MTBF theory. The MTBF of a collection of devices is equal to the MTBF of one device divided by the number of devices. If you have one part, the MTBF on that slab of silicon and associated solder joints might be a year. If you have five parts, the MTBF is now 73 days. That's an extreme example, but sadly, I've seen flash sticks with large numbers of failures in the first month, so that's not nearly as gross an exaggeration as you might think.... And whether one part fails or the whole thing fails, you still lose data.
Also, a controller failure is still likely to cause all flash parts to be inaccessible whether it is integrated into a flash chip or is driving eight discrete flash chips. It's not like you're going to use a separate flash controller per flash part. And I -think- that a device showing zero capacity is probably caused by the flash controller being unable to communicate with the flash parts. If so, then that is much more likely to be caused by a failed connection between the two than by a failed flash controller (unless there are problems with interconnects inside the flash controller chip package failing due to overzealous compliance with ROHS rules).
The original poster also failed to mention the most common failure mode, bar none: poor solder joints or other physical interconnects getting broken by physical force. This is very common among cheap flash drives. I wouldn't expect the same with SSDs, of course---you don't normally carry a SSD in your pocket---but at least in my experience, this one cause of failure is easily an order of magnitude more frequent than any other single cause, and is in all likelihood greater than all the others put together. And that's not even counting actual abuse (washing machines, run over by cars, and so on).
My Lexar JumpDrive Secure flash drive suddenly stopped working, and I talked to my mother, whose entire university class was using that same model of drive. Turns out that between us, we had experienced close to a 50% failure rate on those things within the first month or so, having seen somewhere around 14 or 15 failures. The failure was interesting. Mine failed suddenly, but worked if you tipped the connector at an angle... at least for a couple of seconds once or twice. This told me pretty conclusively that the failure was caused by poor hardware design. As best I can tell, when you carry the drive in your pocket, the cap puts pressure on the USB connector. Over time, this gradually causes solder joint or trace failure (I never cut one open to figure out which) at or near the USB connector.
Since then, I only buy flash devices with mechanisms where the USB connector retracts into a solid housing. Sure, you have an elevated risk of gunk from your pocket getting into the connector because it isn't covered, but at least you don't have the flexing problem. Gunk can be cleaned with a flat toothpick and alcohol. Failed solder joints requires disassembly and SMT soldering skills.... :-)
Re:Burnt out (Score:3, Informative)
Re:In my case (Score:1, Informative)
I blame Macromedia, and maybe YouTube.
Oh, wait, you meant the other "Flash media".
I think you mean Adobe. Macromedia doesn't exist anymore.