Follow Slashdot stories on Twitter

 


Forgot your password?
Close
typodupeerror
×
Data Storage Hardware

eBay Deploys 100TB of SSDs, Cuts Rackspace By Half 197

Posted by Soulskill from the go-big-or-go-home dept.
Lucas123 writes "eBay's QA division was facing mounting performance issues related to its exponential growth of virtual servers, so instead of purchasing more 15k rpm Fibre Channel drives, the company began migrating over to a pure SSD environment. eBay said half of its 4,000 VMs are now attached to SSDs. The changeout has improved the time it takes the online site to deploy a VM from 45 minutes to 5 minutes and had a tremendous impact on its rack space requirements. 'One rack [of SSD storage] is equal to eight or nine racks of something else,' said Michael Craft, eBay's manager of QA Systems Administration."
This discussion has been archived. No new comments can be posted.

eBay Deploys 100TB of SSDs, Cuts Rackspace By Half More

eBay Deploys 100TB of SSDs, Cuts Rackspace By Half

Comments Filter:

  • depends if you are IO bound or need storage (Score:3)

    by smash ( 1351 ) on Wednesday August 03, 2011 @02:28AM (#36969364) Homepage Journal
    For sites like ebay i have no doubt this makes sense. For the average small business I suspect they are far less IO bound and need storage...

    • Might even pay for itself by the years end
      Barring any major catastrophes - expect to see may companies with server farms to go this route soon

    • How many small businesses have more than 1TB of data, in total? Unless you're in a business that does a lot of image or video work, you're probably generating less than 1GB per month. One drive - hard disk or solid state - will probably have a large enough storage capacity for a typical small business. The bigger problem is redundancy and backups, not total storage capacity.

      • If one drive has enough storage capacity for all your data, then one drive has enough capacity for backups too.. I've just taken to using HDDs for backup now. No expensive tape drive necessary, and hard drives must easily be cheaper per GB by now too..

    • Re: (Score:3, Insightful)

      by account_deleted ( 4530225 )
      Comment removed based on user account deletion
      • I've seen plenty of traditional HDDs fail without warning. I'd be wary of looking for signs of impending failure on any kind of drive.

        Remember, RAID is used to protect against drive failure. Dual Raptors in RAID0 is just asking for trouble - you're doubling the chance of failure. They'd be much better off with dual SSDs in RAID1 - much better performance and power consumption and better protection against sudden failure.

        I'm worried about your customers that "don't have time" to perform backups. What do

        • Re: (Score:2)

          by vlm ( 69642 )

          I'm worried about your customers that "don't have time" to perform backups. What do they do when their laptop is stolen? Maybe their work and thus their data doesn't have much value.

          Or they do everything, including storage, online. I'll publicly admit I don't backup my work laptop... The only reason I have it is to SSH, and I'm not backing up gigs of stuff when all I basically need is "putty", which I already have on a flash drive, and is widely available on the internet.

          Customs wants to search my laptop, OK, its basically a vanilla install with putty. And a lot of empty space.

          • Actually, I tend to make sure that any work I do is either replicated elsewhere (checked into SVN, stored on DropBox, cloud storage etc) or is easy to re-create. HairyFeet's customer however, lost a load of work when their drive broke without warning.

            I tend to think of hard drives as temporary storage - if the data is important, have it in multiple physical places.

        • In a workstation scenario, couldn't you RAID1 an SSD and a fast HD in case the SSD dies like hairyfeet's gamer customers?

          Would the SSD be slowed down by the HD trying to keep up or would the RAID controller's Cache prevent that?

          • Write performnace would be limited by the slower mechanical drive since all writes must go to both drives.

            Read performance would depend entirely on how well the RAID distributed reads between the drives.

          • Generally, RAID1 needs the disk sizes to be the same and usually the writes have to be committed to both drives, so you'd most likely end up with a RAID that is limited to the size of the SSD and limited to the speed of the HDD.

            Rather than using RAID, you could use the SSD as a single drive and have differentials/backups copied across to the HDD every couple of minutes or so - that'd provide some protection against disk failure without overly penalising the speed.

        • Re: (Score:2)

          by Guspaz ( 556486 )

          I'd rather a drive fail suddenly without warning if the alternative is "slowly and silently corrupting data until somebody notices it because it tripped a SMART threshold". Because I've had that happen, and all sorts of other nasty gradual failures.

          Any enterprise setup is going to have redundancy built-in. If a drive up and fails suddenly and without warning, you swap it out, boom, done. If you've got hot or cold standby drives already in the array, icing on the cake. But if a drive starts a slow march towa

          • I'd agree with that. I prefer systems that fail fast rather than fail slow - they don't leave you wondering if things are working ok.

      • Re: (Score:2)

        by DAldredge ( 2353 )
        OCZ SSD drives?

  • Uhh.. cost? (Score:3)

    by Dahamma ( 304068 ) on Wednesday August 03, 2011 @02:41AM (#36969434)

    Of course everyone would love to replace all of their storage with SSD if price was no object.

    The closest they come to mentioning cost is:

    Though SSD is typically a magnitude of order more expensive than hard disk drive storage, Craft said the Nimbus arrays were "on par" with the cost of his previous storage, which a Nimbus spokesman said was from NetApp and HP 3PAR. (Craft declined to identify the vendors).

    So, cost of new SSDs was similar to whatever HDDs they bought years ago? Yeah, that's kind of how it goes...

    Nimbus prices its product on a per-terabyte basis - it charges $10,000 per usable terabyte

    $10,000 per terabyte. Ok, then. Sure, it's faster, if you are willing and able to pay 10x the cost of *current* HDD-based systems...

    • Re:Uhh.. cost? (Score:5, Informative)

      by Anonymous Coward on Wednesday August 03, 2011 @03:23AM (#36969598)

      Depends on your workload. (Disclosure: I work in storage for a living.)

      Sometimes, what you need is raw, bulk storage. There are two serious contenders in this space: tape, and disk. You use tape if you have a lot of data you need to store, but not much that you need to access regularly: less power, and it scales to near infinite levels of storage (at the cost of very slow access for a given piece of data.) Or you use disk if you need to access most of it reasonably regularly. SSDs are not, and never will be, a contender in this space - you're paying through the nose on a per GB basis.

      On the other hand, sometimes what you need is IOs per second. Database administrators are very familiar with this - you need a bit of data from over here, and a bit of data from over there, and maybe a little bit more from somewhere in the middle, and you need it five minutes ago. Traditionally, you got this performance by building a large array across many spindles, and giving up half, three quarters, or even more of your disk space, in return for that nice, fast section of disk on the outside of the platter. Lots of running hard drives, drawing lots of power, generating lots of heat, and costing a lot of money for storage space that isn't even half used - because if you throw something else on that empty space, you completely ruin the throughput of your major production database.

      In that latter space, SSD is king. Sure, it's more expensive on a dollars per GB basis, but hey, guess what - GB isn't the important metric here. You figure out which bit of data is being hammered, and you move it across to the SSD. Rather like profiling an application: pick the function that takes 90% of the time in the software, optimise the wazoo out of it, and you get a significant improvement (rather than picking something at random and optimising it to billy-oh, and getting not much return for your investment.)

      So yeah - SSDs aren't going to compete in raw capacity any time soon. But in random I/O performance, they make a hell of a lot of sense. In some respects, yes, they most definitely are cheaper than traditional platters of spinning rust - see the aforementioned massive RAID set across dozens of spindles.

      • VMs are in the IO category for sure (Score:5, Informative)

        by Sycraft-fu ( 314770 ) on Wednesday August 03, 2011 @06:00AM (#36970246)

        You discover modern hardware does virtualization real well. You get a good host software, like vSphere or something on new hardware and you have extremely near native speeds. The CPUs handle almost everything just like it was running as the host OS, and sharing the CPU resources works great. Memory is likewise real good, in fact VMs only use what they need at the time so they can have a higher memory limit collectively than the system RAM and share, so long as they don't all try to use it all at once.

        You really do have a situation where you can divide down a system pretty evenly and lose nothing. Let's say you had an app that used 2 cores to the max all the time and 3GB of RAM. You'd find that it would run more or less just as well on VM server with 4 cores and 8GB of RAM, half assigned to each of two VMs, as it would on two 2 core 4GB RAM boxes. ...Right up until you talk storage, then everything falls down. You have two VMs heavily access one regular magnetic drive at the same time and performance doesn't drop in half, it goes way below that. The drive is not good at random access and that is what it gets with two VMs hitting it at the same time, even if their individual operations are rather sequential.

        It is a bottleneck that can really keep things from scaling like the other hardware can handle.

        At work I use VMs to maintain images for instructional labs (since they all have different, custom requirements). When I'm doing install work on multiple labs, I always do it sequentially. I have plenty of CPU, a hyper-threaded 4 core i7, plenty of RAM, 8GB, there's no reason I can't load up multiples. However since they all live on the same magnetic disk, it is slower to do the installs in parallel than sequential.

        If I had an SSD, I'd fire up probably 3-4 at once and have them all do installs at the same time, as it would be faster.

        • For running on HD's I think there should be one OS with one picture of the filesystem. So that caching, prioritisation and request reordering can all occur in one place, with as much information about the immediate future as possible. Each VM should then import a section of that filesystem. But if you're talking about databases, they tend to replace most of the filesystem level caching anyway, so you'd still be better off with SSD's and / or dedicated storage.

      • Re: (Score:2)

        by bareman ( 60518 )

        If I had a mod point, I'd mod this response upwards.

        The parent to it was considering only capacity. And from a capacity standpoint, no SSD doesn't make any sense and is about 10x the price. But for performance SSD is king and yes, cheaper than the spinning rust.

      • You figure out which bit of data is being hammered, and you move it across to the SSD. Rather like profiling an application: pick the function that takes 90% of the time in the software, optimise the wazoo out of it, and you get a significant improvement (rather than picking something at random and optimising it to billy-oh, and getting not much return for your investment.)

        Or you do what eBay is apparently doing and say, screw it, we're doing 5 blades, and throw all of your storage on SSD,

      • SSDs are not, and never will be, a contender in this space - you're paying through the nose on a per GB basis.

        This is one of those predictions guaranteed to fail. Eventually it will be cheaper to produce SSDs than disks with a bunch of moving parts. How eventually? Too eventually for my tastes... but someday.

    • Re: (Score:2)

      by smash ( 1351 )

      Again, depends if you need bulk storage, or fast i/o. If, to get the IO throughput required, you need to purchase far more hard drive spindles than you otherwise would need for the capacity required, then the total amount of storage you get being less with SSD may not be an issue (so long as it is "enough"). e.g. (fake numbers), if you need say 100k iops and 1tb of space, this could perhaps be done with 10 magnetic disks, or 2 SSDs. the additional space provided by the magnetic disk is of no use if your

    • $10,000 per terabyte. Ok, then. Sure, it's faster, if you are willing and able to pay 10x the cost of *current* HDD-based systems...

      There will always be applications where the high dollar [latest and greatest] solution will provide *vastly* better performance than the current 'standard'.

      If you're replacing 8 or 9 TB of 15K drives with 1 TB of SSDs, then the accounting becomes a bit more manageable.
      And raw price:performance doesn't always tell the whole story, not when paying more can save in other ways.

    • 1 TB can be had for about $1,675 these days. Take a look at OCZ SSDs on new egg. The price is falling nicely, though I doubt it'll catch spinners in the next 10 years for cheapness.

      • What are the failure rates? That seems rather relevant when youre dealing with massive RAID arrays.

      • I'm not so sure, it all depends on moore's law and how well we can keep up with it. Essentially, SSDs should go the way processors have - doubling in capacity every 18months or so.
        For the sake of argument, let's say that a 120Gb SSD is $200, the same $200 in 18months time should, theoretically, get you 240Gb. In 3 years time, it should be 480Gb, 4.5 years would be nearly 1TB and in 6 years, 2TB SSDs shouldn't be that unthinkable, for the same $200 you'd spend today. It was only couple of years ago that a 2

    • Agree. So in other words, if we read between the lines, if they had replaced their older spinning disk with new spinning disk, they would have reduced their rack space by 20 times.

    • Re: (Score:2)

      by Guspaz ( 556486 )

      $10,000 per terabyte. Ok, then. Sure, it's faster, if you are willing and able to pay 10x the cost of *current* HDD-based systems...

      I guess you missed the part where they said that the Nimbus pricing of $10,000 was "on par" with the HDD-based storage arrays they had from NetApp and HP before? Fibre-channel HDD pricing is in the same ballparks as enterprise SSDs.

  • Compact (Score:3)

    by drmofe ( 523606 ) on Wednesday August 03, 2011 @02:48AM (#36969454)
    So the entire eBay VM operation could fit into 6 racks? 200 physical servers @ 1RU each = 5 racks 10x 10TB 2U SSDs = half a rack 5x 2U switches = quarter rack

    • Re: (Score:2)

      by bberens ( 965711 )
      Ebay has 4 DNS entries visible from here, so I would safely multiply that number by at least 4.

  • Zero details (Score:4, Informative)

    by billcopc ( 196330 ) <vrillco@yahoo.com> on Wednesday August 03, 2011 @02:49AM (#36969458) Homepage

    TFA reads like a thinly-veiled promo for Nimbus Data Systems, which I can only guess are pushing a Linux-based SAN appliance full of SSDs. Big whoop.

    What I would love to know is: Why does eBay need 4000 VMs ?

  • I got the impression ebay just terminated a hosting arrangement with Rackspace (the company) -- bringing it inhouse, and cutting Rackspace's revenues in half. :)

  • has improved the time it takes to deploy a VM from 45 minutes to 5 minutes

    uh, any logical explanation for this? SSDs are snappier and the peak I/O can be faster compared to spindle drives - but not by factor 9, or?

    • Re: (Score:2)

      by rtb61 ( 674572 )

      At a guess comparing unformatted hard disk drives to SSD, sneaky.

    • Re: (Score:2)

      by vadim_t ( 324782 )

      Yes, in the real world, data isn't laid out on the disk in the exact order it's going to be read. Especially when filesystem structures are involved.

      Seeks are very expensive. I measured a desktop drive barely doing 300 KB/s when doing random reads. That's of course not very realistic either, but the real world performance is going to be somewhere in between that and the ideal contiguous read speed. Having a disk capable of 100MB/s managing only 5MB/s is quite possible.

  • Reliability? (Score:3)

    by Vectormatic ( 1759674 ) on Wednesday August 03, 2011 @03:35AM (#36969660)

    I read a blog-post a while back stating that SSDs fail a lot more then you would expect. Somewhere around a year of heavy use seems to take most of the life out of a consumer grade ssd. Now i wonder how putting SSDs into Raid 5 (or 6, or whatever) will behave. If a certain model of SSD croaks around X write ops, then i think the nature of Raid will mean that your entire array of SSDs will go bad pretty closely together. It must suck to have two more drives go belly up while rebuilding your array after the first drive failure.

    Perhaps it would make sense to stagger SSDs in different phases of their lifetime to keep simultanious failures at bay, use some burned in drives and some fresh ones.

    • That's okay. I found a blog written by someone from the future, so I sent him a request to fetch some SSD durability data from 15 years into the future.

      I expect a reply any minute now.

    • It really depends on the write cycles, the problem there is that you have about 3000 write cycles on a modern consumer grade ssd, combine that with the fact that most people buy really small ones to save some bucks and they start to use it heavily by swapping on them etc... and you have a situation where you very likely can reach those 3000s within a year or two.
      If you use them in a sane manner and have a decent size so that wear levelling can do its magic you should not hit that limit within the lifespan o

      • i fully realize that consumer grade SSD != enterprise SSD, but a tighter grouping of failures is still very likely due to the way SSDs work. I for one am curious to see what effect this will have on RAID strategies in the future. As for usage, people using their SSD for swap are either stupid, or like the speedup that much that they are willing to eat the cost of an SSD over a years time, but in enterprise land those drives can see very heavy use as well, especially when logging/databases are involved.

        Also,

        • Well they are just replacing their VM servers, the databases are possibly elsewhere on the network so the writes to the SSD in that scenario should only occur when they update a VM. (Just guessing though).
          Still, I take your point of a series of SSDs used for the same purpose are more likely to fail around the same time than ye olde HDDs.

    • Re: (Score:2)

      by vlm ( 69642 )

      Perhaps it would make sense to stagger SSDs in different phases of their lifetime to keep simultanious failures at bay, use some burned in drives and some fresh ones.

      Trust me, as I guy who's run raid arrays of spinning rust for well over a decade, you REALLY need to do that with old fashioned drives too.

      Worst experience in the world is having a RAID-1 with two consecutive serial number drives and both bearings let go at the same time.

  • 'One rack [of SSD storage] is equal to eight or nine racks of something else,' said Michael Craft, eBay's manager of QA Systems Administration."

    Is he talking about performance or price. I can imagine that a single rack of enterprise SSD's could easily cost the same as 9 racks of anything else.

    • Re: (Score:2)

      by bberens ( 965711 )
      You're measuring storage capacity. Ebay is IO bound rather than storage bound. So they measure in reads/($*second) where $ is a function that incorporates comparative failure rates, electricity costs, and probably a few other bits that don't come to mind at the moment.

  • Return On Investment (Score:4, Insightful)

    by ritcereal ( 1399801 ) on Wednesday August 03, 2011 @03:47AM (#36969722)
    While most people instantly gravitate towards the upfront cost and performance of going solid state, I would make one important point. Reducing your data center space by 9 racks is significant in terms of power, cooling and that is all on top of the purchase price and support contracts. Regardless if ebay owns their own data center or colocates, the cost per square foot in a data center and the continued operation of such a large storage system is more then likely to provide a higher return on investment. eBay isn't in the business of looking cool and hip, they're in the business of selling stuff as cheaply as possible and I'm certain their CIO cares only about the bottom line.

    • And don't forget that the SSD solution uses less electricity. That means you're not only saving on the electricity, but you're also spending less money removing the heat associated with using electricity. Being able to rebuild a server in five minutes as opposed to forty five minutes probably means they can keep more backup servers slack and turn them on only in case of emergency. Since SSD use less electricity on standby, they probably can keep the SSD idling. And if it's so much smaller, you can pack mor

  • Finally we are getting a chance of seeing real reliability stats of the SSD!

    That if eBay would be kind enough to publish the data couple of years later.

  • I hope my hosting company will change to such SSDs, and I will not have to wake up those rotating disks early in the morning.
  • I call total BS on this post for a few reasons
    1. 100 TB is about 250 SSD drives. Ebay runs a few hundred peta of storage over several datacenters throughout the US [dbms2.com], not including all other countries.
    2. We run several racks at Telx. Our cost per month in NYC including electric is only about 3k per cabinet. Does Ebay really care about 3 cabinets?
    3. Over half the systems now access the shared storage which contains the drives? Yes, if I map a drive to a particular SAN I guess I am now accessing the data. Does that me
    • SSD life is limited with number of write operations. you cant use them like normal disks in the business ebay is using.

      but read operations are unlimited. so, if you are going to just read files from a hard disk, ssd makes the perfect candidate. in random reads, they are approx 40 times faster than best hdd at the minimum.

      so, you just put 250 ssd disks, put your VM images on it, and, as the article says, it boosts your vm deployment time to other systems from 45 minutes to 5 minutes - there is nothing
  • even the most touted and expensive 'enterprise ssd' can die out on you unexpectedly.

  • They sweet-spot in enterprise storage is doing deduplication with SSD with both direct attached and networked storage, plus 15k 2.5" and 7.2K 3.5" disk for the rest. Deduplication saves a lot space and with SSD it works like cache, especially in scaled out environments.

Slashdot Top Deals

//GO.SYSIN DD *, DOODAH, DOODAH

Close