How Reliable Are 10TB and 12TB Hard Drives? Backblaze Publishes Q1 2018 Hard Drive Reliability (zdnet.com) 123
Wolfrider writes: Backblaze's hard drive report for the first quarter 2018 makes very interesting reading for anyone who is interested in hard drive performance and reliability. As of March 31, 2018, the company had 100,110 hard drives working for it, made up of 1,922 boot drives and 98,188 data drives, ranging from 3TB WDC WD30EFRX drives all the way up to 10TB and 12TB Seagate ST10000NM0086 and ST12000NM0007 drives, along with 10 Samsung 850 EVO SSDs. [...] The overall Annualized Failure Rate (AFR) for Q1 sat at just 1.2 percent, well below the Q4 2017 AFR of 1.65 percent. Some drives had an AFR of 0 percent (in other words, no drives failed during the period), while the 4TB Seagate ST4000DM000 had the highest AFR of 2.3 percent (out of 30,941 drives the company had in service, 178 failed during the Q1 period).
Largest drive I have = 2 TB (Score:1)
Re:Largest drive I have = 2 TB (Score:4, Informative)
My laptop has 2 x 1Tb drives.
Both are full.
I don't even go mad - I store no video, I don't even OWN any music, my family photos take up about a gig.
But things like virtual machines, programming environments, and even installing, say, 10% of my Steam games fills my hard drives up ludicrously fast.
Compare and contrast to my workplace - where we have about 20-30Gb for each user at maximum and about 10% of that is used. 600+ users, 12Tb of active storage (not counting reserved space, backup, replication, etc.).
It very much depends on what you want to do, but my Steam library could easily hit 300-400Gb on its own. If anyone else used the laptop but me you could likely have each user doing that.
Then things like virtual machines etc. can whack it up enormously.
If you're not a gamer, a video-hoarder, a photographer, a content-creator, a developer, etc. then, sure, you can cope on less space.
Personally, I'm just glad my laptop has two drive bays.
Re: (Score:2)
I *wish* my current laptop had two drive bays. My partially retired 2011 17" MacBook Pro has a 512 GB SSD and a 1 TB spinner inside. Much easier to deal with large video projects than the Thunderbolt hard drive attached to my newer 15 incher. Be nice if Apple actually understood that some projects have to be longer than 30 seconds and displayed on screens larger than an iPad. ..... Sadly wanders off to look at his collection of Firewire stuff......
(My NAS has 32TB of space and it's half full. Video i
Re: (Score:1)
Umm. Has Slashdot deprecated the Return key? Are we going back to EDBIC or something?
Re: (Score:1)
EBCDIC. If you're going for nerd cred, at least get it right.
Re: (Score:2)
Or if you run a large project.
My "small" environment at work produces 40GB of final product for every full build, every day. If we include intermediate products (which make troubleshooting much easier), it's pushing 120GB per build... and we're trying to ramp up to about 20 full builds each day. Quick computation estimates about 48TB per month at that rate.
Now, we don't need to keep old builds around forever... most of the time. We currently have about 15TB of data we can't get rid of for contract reasons,
Re: (Score:1)
Re: (Score:2)
ZFS is not permitted, for reasons mostly boiling down to either "we didn't pay enough money for it" (for anything BSD) or "that's too expensive" (for anything Oracle). Free (as in beer) is a big scary concept according to one of our contract terms, but they still won't approve purchases.
I've been pushing the idea of a BTRFS-based NAS to get most of the same features, but even going to a NAS at all is a big architecture change, enough to require many assurances that "centralized is a good thing" and "this is
Re: (Score:2)
I don't know in what industry you work, but this makes me cringe... in finance or pharma this is just a regulatory risk waiting to turn into huge fines and more important, loss of reputation. Maybe you could point out to your managers that damage to reputation will always be much more expensive than the cost of a decent storage system.
Re: (Score:2)
Well, it's a government contract, passed through several rounds of subcontracting... In the eyes of management, the risk of having to redo documentation and approvals is scarier than the risk of data loss. There's no risk to reputation, because this architecture was properly approved by the customer back when it was built, when the builds were only a few tens of megabytes. Any failure now is not our problem.
Re: (Score:2)
Hmm okay. I understand. It still makes me cringe.
I think due diligence here means covering your ass with a few formal warnings here and there. You know it will all come down crashing at some point, and then everyone will look downward for someone to blame and they end up with the IT people as potential scapegoats.
Re: (Score:2)
Android OS builds run about 300 GB each. And I usually have to track 3 trees. Builds go massively faster if I'm on an SSD, so my most active development goes there. You'd think a Fortune 500 company would buy every developer a massive multi-terabyte SSD, but part of our success is not spending money until we absolutely have to (or so I assume). I managed to get a 512GB drive, and that's how I and most other developers have been functioning for the last 3 years.
The difference in built time on HDD and SSD is
Re: (Score:1)
Indeed. I run virtual machines on my QNAP, backups and volume snapshots that takes up space. Then there's all my home videos, 400GB recorded on my last vacation of boring stuff. :D I have no problem filling the 4x8TB drives and the 4*4TB drives.
The 4TB drives are WD drives that have run for 5 years now, iirc, the scheduled disk scrubbing havent revealed anything yet on those , but the 2TB drives that were replaced with 8TB because they were starting to get recoverable errors and have been with me since 2 QN
Re: (Score:1)
I also don't own much of the music I store. ;-)
Re: (Score:2)
Sure.
But I literally don't have music. Because I don't listen to it.
If I hear it, fine. But I don't purchase, "steal", stream or store it.
I think I "own" a single music track, and that was purchased for something to do with work.
Re: (Score:2)
You have 60TB to back up 20TB of data?
Re: (Score:1)
You have 60TB to back up 20TB of data?
That sounds right to me. I don't know how may TB of storage I have as I haven't bothered to add it all up in a while. But I have three times that amount in backups. I run one mirrored backup daily to a set of drives, another that runs monthly, I think that one's differential, and a third that runs two weeks after the monthly backup to external drives that I swap out to an offsite location. Actually I guess since I keep a set of external drives on site and one off site, I actually have 4X the drive capacity
Re: (Score:2)
That's roughly where I was at for a number of years, but when my wife got pregnant last year we figured it was time to assemble some local storage on our network for keeping original copies of our photos and videos, storing our Time Machine backups, and hosting our Plex and (old, not-yet-stripped-of-DRM) iTunes content, so I picked up 4 x 5TB drives and slapped them in a directly-connected enclosure with a RAID 5 configuration. It's FAR more than we currently need, but Black Friday served up some steep disc
Re: Largest drive I have = 2 TB (Score:2)
Re: (Score:1)
While I'm impatiently waiting until the 16TB drives are released so the 12TB prices will come down to reasonable levels.
Re: (Score:2)
I have a 6TB of RAID storage and it's nearly full. Almost all of that is ripping all Blu-Rays, DVDs and CDs to central storage.
So, by the sound of it... (Score:2)
I recall their SCSI drives being the shit...
Re: So, by the sound of it... (Score:3, Informative)
Re: So, by the sound of it... (Score:3)
Re: (Score:3)
Yeah the HGST drives are looking very good. The seagate drives (the first 2, might work out to the better investment though; due to being 2.5 to 3 times the size).
(ie If you have 1000 seagate 12TB drives (12000TB), and need to replace 10 per year (1% failure rate), that might still work out better than having 3000 HGST 4TB drives (the same 12000TB), where you need to replace 0.5% year because that's 15 of them.
I'm guessing that 10 12TB costs probably costs around twice as much as a 15 of the 4TB drives, but
Re: (Score:2)
what the fuck are you gibbering on about?
Obviously, filesystem volume capacity due to RAID organization sits on top of total underlying physicl drive capacity. But that makes no difference here.
Re: (Score:2)
HGST drives have been the most reliable ones for at least 3 years now. I only buy HGST drives and across all my computers, not a single one has failed, not even a bad block.
Re: (Score:2)
Not really.
Rows with less than 10K drive count should be ignored because they tend to have a higher error margin. This leaves Seagate as highest failure rate with 2.9% for their 4TB drives.
Re: So, by the sound of it... (Score:1)
But that is unfair as none of the stats for WD would be included - doh! Either include and compare or filter and ignore, you can't have it both ways?
Re: (Score:2)
Rule #3 of analytics: data is politically neutral.
So all WD drives would be excluded, so what? We won't include them in the comparison either.
Re: (Score:2)
Rule #2 of analytics: your subjective choices and interpretation are wildly biased.
The correct approach is to compare drives or manufacturers based on statistical testing while making a minimum of arbitrary choices.
Re: (Score:2)
Yes, if you have the right data for it. If you don't, you include the subset you have enough data for and don't talk about the drives/manufacturers you don't have enough data for.
Re: (Score:2)
Not really. Rows with less than 10K drive count should be ignored because they tend to have a higher error margin. This leaves Seagate as highest failure rate with 2.9% for their 4TB drives.
Why "not really"? They list the error margins and the failure rate for WD 6 TB drives remains higher than Seagate 6 TB even taking into account the reported confidence interval. Also, the confidence intervals for about 1,000 drives are very similar to those of about 10,000 drives. You don't need 10k drives to get an accurate estimate.
Re: (Score:2)
I assume you are talking about confidence intervals. The problem with them is that they are calculated solely based on sample size. They're not weighted against the possibility of having a bad batch, for example. Improperly stored or transported or built drives could severely skew the data.
But even if we reduce the lower range bound to 1000 drives, the classification remains unchanged.
Re: (Score:2)
Well, there is a reason why they have fewer Western Digital drives...
Re: (Score:2)
I think that reason is that while the WDs might be more reliable, their extra cost doesn't usually make it worth while for the scale they are working at.
Re: (Score:2)
Yes, price. Backblaze buys the cheapest HDDs they can find.
victory: MBA (Score:2)
No.
Efficient frontier [wikipedia.org]
Shannon's theorem: as you approach the Shannon coding limit, the cost of failure becomes linear.
The primary term in the linear model is cost_of_drive / (mean_working_life * drive_capacity). In metric, the unit comes out to Big Macs/B-s, but we'll use USD/TB-year.
There is also a power consumption term, and a performance term. The first of these is significant to Backblaze, whereas the second does little to differentiate the quali
Re: So, by the sound of it... (Score:1)
But if you look at lifetime stats the two Western Digital drives have the highest failure rates.
https://www.backblaze.com/blog/wp-content/uploads/2018/05/blog-Q1-2018-lifetime-drive-stats.png
Re: (Score:3)
Anecdotal evidence:
I've been at my current workplace for 4 years. In that time EVERY SINGLE Seagate-branded drive has failed - dozens and dozens of them. In RAID arrays, in servers (15K SAS etc.), in NAS, in desktops, in clients.
Replacing with WD as we go - and repurposing old WDs they already had to serve in the Seagate's places - I've had precisely one WD failure. And we think that took a whack because it hit the floor.
Seagate drives honestly are shocking in their build quality.
Re: (Score:2)
FYI Samsung only makes SSDs at this point. They sold off their HDD business to Seagate at the end of 2011: https://www.seagate.com/about-... [seagate.com]
Re: (Score:2)
I'm discussing THOUSANDS of individual purchases, from all kinds of manufacturers, dozens of different resellers, at all kinds of levels (everything from servers to storage arrays, desktops, hard drive purchases) bought over nearly a decade.
Re: (Score:1)
HGST (nee IBM) haven't been that common, but have been very reliable. I bought a fair number during the DeskStar days and was very happy with them. I still have a few HGST humming happily away.
You may have had the HGST Deskstars, but it's obvious you didn't ever use any of the IBM Deathstars. I had hundreds of the 60GB ones fail from the click of death. 100% failure rate on those damn things. I have 15K RPM SCSI Seagates that still work fine, along with an array of Micropolis Tomahawk SCSI drives. Not that I spin those up very often. They sound like a jet engine starting.
I've said it before, but hard drive manufacturers have been pretty cyclical over the years as to which was good and which wa
Re: (Score:2)
Re: (Score:2)
Are you being paid by WDC? Because the stats published in TFA don't support your contention that WDC is the king.
Re: (Score:3)
I don't care about sales, eps and growth.
I care about hard drive reliability.
Re: (Score:2)
In general, yes.
I'm not really surprised though, given how much cost-cutting goes on - the real reason is they are stupidly cheap - HGST drives are horrendously expensive (but yeah, they have a way lower failure rate) and WD drives are somewhere in the middle. And yes, if retailers can heavily discount Seagate drives, it really means they didn't cost too much to begin with.
So far the worst drives universally
Re: (Score:3)
The chart you want to look at is the last one in the report - lifetime failure rates. Mainly the annualized failure rate and confidence interval high/low columns. Those percentages take into account the number of drives in the sample, and how many days they've been in serv
Re: (Score:3)
couple months back, a separate division of my employer had 3 of a 16 disk group fail within 2 days -- killing a raid-6 group. These were 8TB seagate disks and were ~6 months old.
This is why, as a rule, you shouldn't populate a RAID with drives from the same manufactured batch.
Re: (Score:2)
Very true, old advice, but can not be repeated often enough!
Burst coin (Score:1)
If they're not mining Burst they're missing out on some money.
Re: (Score:2)
If they're not mining Burst they're missing out on some money.
Burstcoin was a cute idea, but no --- mining it's not profitable if you buy 4TB drives for that: you'll lose money on the purchase,
and probably cause a premature failure of your hardware.
The coin would either need much more value, or we'd need a much cheaper storage medium than even tape.
Re: (Score:2)
I'm not talking about us guys, but the guys in the article, who are keeping almost a hundred thousand drives powered up all the time for their tests. It's a positive delta if they get some Burst while at it. While it's a rounding error for you and me, the rounding error might become noticeable at that capacity.
Personally have had good luck with Seagate (Score:2)
Over time I've had pretty good luck with Seagate drives, and if you look at the data it seems some models are more stable than others...
That said it does seem like in recent years HGST has gotten pretty good so I've started to shift to them.
Re: (Score:3)
Re: (Score:1)
I still believe that the best hard drives are Western Digital Black and Gold models. I have some that are well past their prime and should have failed but keep on reliably clicking away. I just re-formatted one and still no bad sectors.
Now that you've said that, watch Murphy's law kick in.
Anecdotally... (Score:1)
The thing is all of the hard drive failures I've ever had have been Western Digital, so I've grown very reluctant to buy products from them. In fact just recently I had to buy a Western Digital drive as it was the only SD reader integrated unit I could find that worked standalone well... the only problem is that transfer speeds when you connect it via USB are at USB2.0 speeds it seems like, making it worthless for anything but a standalone drive.
So even though Black and Gold models are probably very nice,
Re: (Score:2)
Over time I've had pretty good luck with Seagate drives, and if you look at the data it seems some models are more stable than others...
Yeah, I outfitted a RAID array with the infamous ST3000DM001 [backblaze.com] several years ago and had to replace three or four of them during the two-year warranty period (as I recall, one of the warranty replacements itself crapped out fairly quickly). After the warranties ran out, I started replacing failures with WD and HGST and things have stabilized. Had I originally sprung for 4TB Seagates I probably would have been fine in comparison.
Re: (Score:2)
Over time I've had pretty good luck with Seagate drives
I've had drives fail from every manufacturer. I've only had drives fail suddenly without any forewarning from Seagate.
What a great public service (Score:5, Insightful)
Do any other cloud-storage services publish stats like this?
Thank you, Backblaze.
MTBF = LOL (Score:2)
I have a Synology NAS loaded with Western Digital Red drives configured in a Raid 10 array.
First drive failure happened a few days ago with an unimpressive 385 hours on it. Drives normally run around 88f temp wise.
Considering the MTBF is supposed to be around 1,000,000 hours I am less than impressed with its lifespan. Glad I don't have their 10TB models installed as the cost to replace one is triple.
So, take any claims of how reliable it is with a grain of salt because while Unit X may outlast the univers
Re: (Score:2)
First drive failure happened a few days ago with an unimpressive 385 hours on it. Drives normally run around 88f temp wise.
Considering the MTBF is supposed to be around 1,000,000 hours I am less than impressed with its lifespan.
MTBF is a statistic that captures randomised failures and accelerated failures due to end of life. What you described is an infant mortality failure and is completely meaningless to MTBF figures. They are purposely filtered out to not skew the results due to a shoddy production.
Unit Y may keel over tomorrow
Yes that's where the "mean" under mean time between failures comes in. There's no guarantee that it will last 1000000 hours. That said failures that exclude infant mortality follow a standard statistical curve, and statistically it i
Re: (Score:2)
First drive failure happened a few days ago with an unimpressive 385 hours on it.
So... still under warranty right?
Better than the WD Red that failed on me about a year out of warranty.
Warranty Turnaround Times (Score:5, Informative)
I have a wide variety of drives at work, both HDD and SSD. I mostly buy "enterprise" grade drives, and specifically look for models with a 5-year warranty. What I've discovered recently, however, is that here are huge differences in how manufacturers fulfill their warranties. When a drive fails, what I'm looking for is to obtain a replacement as soon as possible. I can live with a degraded RAID array for a few days, perhaps, but not for weeks. With an "Advance RMA", the manufacturer will ship a replacement drive immediately rather than waiting to receive the defective drive. (A credit card is provided to cover their loss if the defective drive is never received).
My most recent experiences can be summarized as follows:
Western Digital HDD - Advance RMA is available
Seagate HDD - Advance RMA is not available
HGST HDD - Advance RMA is not available
Even with Advance RMA, I have to wait for ground shipping. I wish that expedited (air) shipping was also available.
I'm saving a special category of experiences for Intel SSDs - experiences so awful there are in a class by themselves. I've had the misfortune to suffer two failed Intel SSDs. Both happened to be M.2 format SSDs. One was SATA, the other NVMe. Firstly, just getting an RMA started with Intel is painful. Be prepared to disassemble whatever computer is affected, because providing a model number and serial number are not enough. They also require something called an "SA" number that can only be found on a sticker attached to the device. Second, be prepared to wait a LONG time. I'm talking weeks to MONTHS to get a replacement. If you need the affected computer back up and running within a reasonable timeframe, you'll need to purchase another SSD in spite of your warranty coverage.
Re: (Score:1)
Strange, in the past I've filed advance replacements from Seagate.
Re: (Score:2)
Re: (Score:2)
I know I'm a bit late to this discussion, but I thought this might help. I ran in to this issue some time ago, and now before I put a new hard drive into a computer, I lay it on the flatbed scanner and scan the top of it (and the bottom, if there's anything to see there).
My experience w/Seagate's engineers (Score:5, Interesting)
Once upon a time in the penultimate decade of the last century, I was chief fixer dude for a manufacturer which had built some custom stuff Seagate used to bulk-test drives in their engineering department. That stuff kept coming back for warranty service but nothing was ever found to be wrong with it, which was a red flag, and the creation of the test setup required about six hours of tech labor so the damn flag was on fire. I got nowhere in my first round of calls to Seagate, but when the stuff came back yet again I was more persistent and finally got to the bottom of it.
Seagate had a guy who was somehow involved with that engineering test system, and every time something went wrong, whether it was an actual system failure or just an unexpected outcome, said guy jerked everything still under warranty out of the system and sent it back to the manufacturers for service. Everything, whether it was potentially related to the troubling observation or not. In driving my way to someone in charge I spoke with folks at Seagate who were incredibly frustrated with the shotgun approach because it kept their test system out of service for far longer than it ever should have been, and eventually they allowed me to reach the shotgun monkey's boss's boss. I explained to him that our warranty terms applied only to product which had failed in normal service, and that on-demand conformance testing was a full pop T&M (time and materials) service for which they would henceforth be charged.
The stuff was not seen again in the time I remained employed by that company and I've happily avoided Seagate ever since.