There's No End In Sight For Data Storage Capacity (computerworld.com) 107
Lucas123 writes: Several key technologies are coming to market in the next three years that will ensure data storage will not only keep up with but exceed demand. Heat-assisted magnetic recording and bit-patterned media promise to increase hard drive capacity initially by 40% and later by 10-fold, or as Seagate's marketing proclaims: 20TB hard drives by 2020. At the same time, resistive RAM technologies, such as Intel/Micron's 3D XPoint, promise storage-class memory that's 1,000 times faster and more resilient than today's NAND flash, but it will be expensive — at first. Meanwhile, NAND flash makers have created roadmaps for 3D NAND technology that will grow to more than 100 layers in the next two to three generations, increasing performance and capacity while ultimately lowering costs to that of hard drives."Very soon flash will be cheaper than rotating media," said Siva Sivaram, executive vice president of memory at SanDisk.
Reliability? (Score:5, Insightful)
I'll generally take reliability over volume. I wish they'd work on that more.
Re: (Score:1)
Re:Reliability? (Score:5, Interesting)
The big purchasers of storage want the opposite. They prefer cheap capacity over reliability. All their data is replicated multiple time so losing a storage device is nothing to them, the data just auto-replicates to other devices.
Re: (Score:3)
Agreed. Since they are required to replicate data to ensure high availability anyway, the major cost is not the quality of the medium, but rather the costs of raw storage. For quality considerations to overcome capacity, they would need a medium that is utterly reliable, which is a very, very hard thing to prove. That means that dropping quality by some low percentage, if it reduces the price of storage appreciably, is an acceptable trade off.
As an individual, we do not expect to buy a lot of storage, an
Re: (Score:2)
For many, if not most individuals, quality would be priceless because we aren't going to necessarily want to create a RAID array of our disks.
Not this individual. My use case is different than yours, I guess. The amount of data I actually generate myself is miniscule. If I need big drives at all, it's to fill them up with media of various kinds. Since almost all of it would be trivial to re-acquire (re-rip from the original media, etc) I'd hardly shed any tears if my one, non-redundant drive went south. Storage, for me, simply isn't "mission critical."
Re: (Score:2)
I'm with you on this one. My home server has around 8TB of data on it and out of that probably 7.5TB is videos/music/games that I can redownload or rerip as necessary if I lose them.
I use a manual system, personally. /home is on a RAID1. /mnt/media is a JBOD pool. Everything easily replaced goes in /mnt/media, everything important goes in /home/$user which is also backed up. /mnt/media just gets a file listing taken every night so if it blows up I know what I lost.
Re:Reliability? (Score:5, Insightful)
So, the time required to feed optical discs into a computer and/or browse web pages to get downloads is worthless? Some of us value our time, and don't seem to have enough of it.
Re: (Score:2)
So, the time required to feed optical discs into a computer and/or browse web pages to get downloads is worthless? Some of us value our time, and don't seem to have enough of it.
A valid point, but I have automation already set up to handle things for me. Sonarr handles my TV library, Couch Potato does the same for movies. As long as I don't lose their databases (which are small and easy to back up) I can restore my TV and movie collections basically at the push of a button. Those programs will then go out and light up my internet connection for a few days and the missing parts will (mostly) reappear. My music collection is synced to Google. The majority of the rest is cached S
Re: (Score:2, Interesting)
It doesn't seem like you're counting bit rot in your failures.
Re: (Score:2, Funny)
If your bits are rotting, maybe you should be monitoring your humidity levels.
Re: (Score:2)
Or check for fungus. Dry rot is a pain.
Re: Reliability? (Score:4, Informative)
Don't we all have cron jobs to scrub our zpools?
Re: (Score:2)
Zpool? Is that related to that "illegal" filesystem in Linux? :-)
Re: (Score:2)
Re: (Score:1)
Re: (Score:2)
i have a rack at the end of a hot aisle with a sliding door attached to it. so every time somebody wants to enter the hot aisle, they slide the door open and let it roll back behind them. and that means vibrations. when i replaced rotational hard disks with SSDs, i went from cca 30% disk failure rate within first 18 months to 3%. to me, SSDs are a Godsend.
for anyone who wants to know, the failing disks were SATA WD RE2/3/4. Seagate Constellation ES.2/3 had failure rate of about 5%. None of my SAS disks seem
Re: (Score:2)
Re: (Score:1)
It's partly because they never had a choice, or didn't understand how unreliable typical drives are/were.
Selling a more reliable drive at a bit more cost will not be given much attention UNLESS one first knows how unreliable typical drives are. Most consumers don't know.
Note that I found RAID to often not be helpful, because if the OS gets hosed, which is common with Windows, the RAID won't work right either. Perhaps with enough expertise on configuring one can avoid that, but that's beyond the consumer lev
Re: (Score:1)
So you prefer SSD than? SSD reliability already far outstrips spinning disks, so you should be happy with the move to solid state.
Re: (Score:2)
even if that were true (it isn't), the advantages of SSDs far outweigh their disadvantages. to get the throughput and iops i can reach with SSDs, i wouldn't be able to get with SAS disks for 3x the cost.
Re: (Score:2)
my favourite side effect of ext4 is spinning laptop disks up every couple of seconds. i.e. disk wants to go into standby as APM commands, but jbd2 spins it up almost immediately. this way you end up with load_cycle_count increasing by thousands every day. death to all rotational media!
Re: (Score:2)
So, your single point of data is more important than tests that have actually been run?
http://techreport.com/review/2... [techreport.com]
I would love to see the spinning disk that can handle 2.1 petabytes being written to it. But of course you have run the thousands of tests for years on end needed to identify the unreliability of SSDs vs HDD.
http://www.tomshardware.com/re... [tomshardware.com]
Re: (Score:1)
Re: (Score:2)
If you believe the hype, 3D Xpoint nvram will be more reliable than either NAND or spinning rust.
And (again, if you believe the hype) it will continue to get cheaper and cheaper, while NAND won't.
Re: (Score:2)
I'll generally take reliability over volume. I wish they'd work on that more.
Not just reliability, but archival (write-once) media that are reliable for very long periods of time.
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
If you value reliability, you want ZFS and RAIDZ.
Re: (Score:2)
I use btrfs. It's still not faced the sort of testing that enterprise users would want, but I've put it through some pretty horrific abuses and it's came out almost undamaged, including a really awful hard drive that was randomly and silently dropping writes when under heavy load. Thank you, Seagate.
Re: (Score:2)
have you seen an enterprise setup built in the last 5 years that used raidz instead of mirrors?
Re: (Score:1)
Re: (Score:2)
Redundancy is better than reliability, and low cost-per GB means redundancy is easily affordable. Because no matter how much you spend on a drive, you're never going to hit 100% reliability, which means that you still have to do everything you're going right now in order to protect your data, with multiple levels of backup plus RAID if you need high availability. Cheap per-GB costs means you can build in all the reliability you want into a system yourself, and it's probably going to ultimately be cheaper
Re: (Score:2)
I'll generally take reliability over volume. I wish they'd work on that more.
Why? As someone who has gone through some 30 HDDs over the year who's never lost any data in failures I have to ask: what are you doing wrong?
what is the point of streaming (Score:4, Interesting)
Re: (Score:2)
Re: (Score:2)
No matter how great your local storage is, you can't carry *all the media*
Maybe not, but eventually, you could carry all the media you own.
I already do this for music.
If Moore's law continues to hold, in 20 years I'll be able to do the same for Movies, TV, and Video.
I'll still download new stuff.
If you want to call playing something as it's downloading "streaming" then I suppose I'll do that too.
Re: (Score:1)
What, you don't value being constantly tethered to an unreliable or unportable internet connection while your viewing habits are sold to the highest bidder? You don't like your 1950's communications platform being used as a square-peg round-hole solution to transfer large sections of data? You sir are living in the past, and must secretly be a luddite.
Re: (Score:2)
Only pirates have large storage drices (Score:2)
Or, at least, that will be the effect once all of the legal/available methods of ripping your own media are gone.
why stop there? (Score:4, Funny)
War of the marketing material... (Score:4, Insightful)
HDD company says HDDs are going to continue to be the best price/capacity for a while, and it's going to be awesome, traditional SSD company says that's bollocks and HDDs will be completely obviated in the same timeframe, and Intel/Micron say 'screw nand, we got something better'
It's great to see real advances in technology (unlike the various BS 'revolutions' that are commonplace in pure software), but it would be nice if once in a while a marketing person said something straightforward and honest.
Re: (Score:2)
SSD company says ... HDDs will be completely obviated in the same timeframe
No one said that. They said that the price of NAND based storage will drop to be competitive with HDDs, but I suspect that there's an unspoken assumption there of HDDs staying static... in other words, NAND storage in a few years will be equivalent to the price per GB of HDDs now.
Even if NAND storage reaches price parity with HDDs, there will probably remain markets for both, since they have different performance behavior for the lifetime of the device, sine NAND cells will die over time. On the other
Re: (Score:3)
"Very soon flash will become cheaper than rotating media." In which case you don't buy rotating media anymore. SSD commands a premium because it's got advantages over HDD, so much so that many modern data center SAN racks are going all-SSD despite it costing 4 times as much.
Re: (Score:3)
How long before they stick a couple of embedded CPU's or GPU's along with those SSD's?
Re: (Score:2)
How long before they stick a couple of embedded CPU's or GPU's along with those SSD's?
Hah! Computer on a disk. It will take "upgrading my computer's hard drive" to a whole new level.
Re: (Score:2)
Hard disk drives already have CPU microcontrollers to handle the cache management algorithms. Some companies actually tried putting some arithmetic logic on their VRAM chips so that pixel operations could be done without fetching across the data bus.
Re: (Score:3)
I think they pay 4x because the value is worth more than 4x.
You get a dramatic performance improvement which in itself provides efficiencies in service delivery, reduced power consumption. And the performance benefit itself is an exponential improvement, not just an incremental one.
Re: (Score:2)
This is why I never use the term "value" in economics. I've written full macroeconomic theories and a decent amount of market economics without talking about anything called "value".
Of course, theories of value are so broken they eventually concluded that things have value for magical reasons.
Re: (Score:2)
There will be a place for HDDs, with their price advantage... but already, SSDs are storing stuff more densely than HDDS can. HDDs that use SMR or HAMR will be useful, either as a lower tier for storage, or be used for home NAS servers, just because they are relatively inexpensive, and at the low price point, a 3 TB WD Red HDD for a C-note is a lot cheaper than a 3TB SSD for that use. No, you won't want to run iSCSI from that array, but for stashing files and having some redundancy, it is good enough.
Re: (Score:2)
Well the way I see it the main issue for HDDs is that they don't scale down well. I just looked at the prices now and the cheapest are around 500GB, but you can't get a 250GB drive for half or 125GB drive for a quarter of that but you can get a 128GB SSD for the same. And for a lot of people who are mostly streaming or working in business and not heavy into home video that's plenty. A HDD is still unmatched if you need many TBs of capacity, but hey most people just don't. No, we're not average.
Re: (Score:2)
but it would be nice if once in a while a marketing person said something straightforward and honest.
It is all honest. From the point of view of each company they are putting the R&D into doing exactly what the marketing people are currently saying.
The only real question is who's R&D will bare the fruits first:
1. Will HDD vendors achieve the next boost in storage first.
2. Will SDD vendors get 100 layer 3D NAND working first.
3. Will Intel get their new memory format out first.
No one is being dishonest.
640K (Score:2)
That is, we can produce a lot of data, but what if is it all GIGO?
Re: (Score:2)
I'm guessing they're assuming that the processing power of tomorrow will make short work of the data they're storing today.
If you have all the garbage it's no longer garbage (Score:5, Interesting)
Garbage is actually filled with valuable substances. GIGO is more applicable to older, discrete entry systems which have specific analysis paths.
Consider real garbage in your trash, or in transfer station, or in a landfill. It generally has no value - it's garbage. But at a larger and larger scale it begins to have, statsitically, more valuable material in it. Now consider minerals trapped in the earth. We regularly process millions of metric tons of earth to refine and process into the elements we need.
A truck full of garbage is a smelly mess. An earth full of garbage is a resource which can be mined for nearly anything you need. That's why all the trash we put in Google has greater value as the total amount of garbage they collect increases. It's still garbage...but at that scale it can be refined into money.
Re:640K (Score:4, Insightful)
This is how we can combat Big Brother surveillance (Score:2)
This is, actually, the key to fighting constant monitoring by the NSA and other three-letter agencies, I believe: generate a lot of spurious data, too much for them to store, much less analyze.
If I kept sending email of 10 MB files (I know, that's small nowadays) which were randomly generated and had no meaning , and then erased them once they reached the other side (e.g. maybe a different email account of mine), then that's no skin off my back since I know I shouldn't care. Any monitoring agencies, though
Re: (Score:2)
Hmm... I like this idea.
10 MB file created, encrypted, mailed, sent to /dev/null. /dev/null.
Recipient encrypts it again, mails to next person, sent to
Daisy chain it so more people can participate.
Every third (or so) time it's encrypted, the next one gets deleted and a new 10 MB random file is generated.
Have it go in circles BUT also have it 'cross-fire' so that it's sending data to random participants.
Get more than 1 file moving around.
Make it tit-easy and let people join/participate.
It can all be done over
Re: (Score:3)
LTO 7 WORM tapes seem to be the best thing for this as of now. Not cheap for the drive, but cartridges are not bad, and have a very long archival life.
After that, perhaps Amazon Glacier comes to mind. Cheap to get the data in... costly to get it out.
Re: (Score:2)
So, an LTO tape drive then?
Will it ever get cheap again? (Score:5, Insightful)
Back in 2011 I could get a 1.5TB drive for 45€, now five years later the best I can get is 3TB for 90€. Double the storage for double the price. If I just want to spend 50€ I only get 1TB. It's nice that we now have 6TB and 8TB drives, but they aren't cheap and so far haven't really lowered the price of the smaller drives and given how long this has already taken I am not even sure if HDDs will ever get cheap again before SSDs will take that space.
Re: (Score:2)
Re: (Score:3)
And don't fret about the rate of price drops slowing down since 2009 in the graph. The y-axis on that
Re: (Score:2)
Price per GB has resumed dropping [cbsistatic.com] since the effect of the Thailand flooding and HDD consolidation in 2011-2012.
The most frustrating part isn't so much the price per gigabyte, as that is back to pre-flood levels, but that you only get that good GB/$ rate in the $100 price range, while you used to get that in the $50 range. If you only need 1 or 2TB you are still paying quite a bit more then five years ago, it's only with 3TB-8TB drives that you get a slightly better rate then pre-flood.
Re: (Score:2)
Back in 2011 I could get a 1.5TB drive for 45€, now five years later the best I can get is 3TB for 90€. Double the storage for double the price.
Just as well there was no major natural disaster skewing your statistics in that time frame ....
Good. (Score:2)
"Very soon flash will be cheaper than rotating media," said Siva Sivaram, executive vice president of memory at SanDisk.
I can hardly wait. Just as core replaced drum and ram replaced drum I am excited to get rid of spinning platters. The day I can get a 3TB SSD for around $100 will be a great day.
Of course it might be possible to get a 40TB HD for that price by then...
Re: (Score:2)
Perhaps misplaced ... but I remember the time when we had races for the smallest disk size (as in form factor). Something like 1" disks. The first iPods had super small disks, and basically read one track and stopped the disk then. Fotos of such disks usually where comparing it with a small coin, "a dime" or "one Mark" (that was pre Euro). That was not that long ago.
Steve would say: is that not incredible? No, I think his term was "amazing" ... anyway.
Re: (Score:2)
40TB HDD? It's apt to fail before you read all the data off of it... Ok, I'm only slightly joking there. The IOPS will still be piss poor on spinning media, so it will only get good throughput with large files, current benchmarks on 8TB drives show around a 150MBps read and write, if that holds the same, it will take over 70 hours to transfer all 40TB, in the best case scenarios.
Re: (Score:2)
Ok then an 8TB 2.5" HDD for $90.
Even at today's prices I see HDDs as more of archive storage. You put large amounts of data like media on them. Also if IOPS count then you will just keep adding spindles to an array.
Not likely. (Score:2)
that will ensure data storage will not only keep up with but exceed demand
If that were to happen, the bottom would drop out of the market; they're obviously not going to produce so much that they shoot themselves in the foot.
Will anyone look at all this data? (Score:2)
I was talking with someone that works with storage systems, he mentioned other technologies to replace HD and their looking into massive petabytes (sp?) of storage capacity. I asked with so much data, when does anyone have enough time to look back at it. His reply, "they will not."
OK some data will be examined, I'm thinking from my perspective. I have about three major email accounts. One has 360 messages, I recently trashed 800 but few years ago I archived 2000 messages (but haven't looked at them since)
Yawn (Score:1)
Data storage capacities are going up. News at 11.
Seagate claimed 60TB by 2016 (Score:4, Interesting)
This post on Slashdot reminds me of previous post on Slashdot [slashdot.org] from 2012 about 60TB 3.5 inch and 10TB 2.5 inch drives by 2016. Again it was based on Seagate's 2006 patent on heat assisted magnetic recording. I think it is more realistic at his point to expect LTO tape to provide 220TB storage [slashdot.org] than Seagate making good on it's dream of HAMR drives. This latest claim should be an indication of how much trouble Seagate is having with their heat-assisted technology. They are essentially giving themselves a 4 year extension to achieve only a third of the results that they previously claimed.
Re: (Score:3)
It goes back even farther than that!
Here's an article from 2010 reporting Seagate promising 100TB HAMR hard drives:
http://www.myce.com/news/seaga... [myce.com]
Here's an article from 2006 reporting Seagate promising HAMR hard drives in "a few years":
http://webcache.googleusercont... [googleusercontent.com]
Also I'm puzzled by the claims about hundred layer 3D NAND chips. I can see how a hundred-layer chip would increase density and therefore could reduce access latency, but I don't see how it could significantly reduce cost-per-bit. Sure, th
Unlimited storage (Score:1)