Catch up on stories from the past week (and beyond) at the Slashdot story archive

 


Forgot your password?
Close
typodupeerror
×

Generator Delays May Slow Data Center Projects 257

Posted by ScuttleMonkey from the so-that's-why-mmos-can't-expand-to-meet-needs dept.
miller60 writes "The data center building boom is causing backlogs for new generator orders, with some companies reporting delivery delays of up to a year for new 2,000kw units, which are the current standard for mission-critical facilities. Generator availability is 'the No. 1 thing that will drive your construction schedules,' according to Equinix, which is building centers in three major markets. 'This will be a big issue for the next wave of data center builds,' says another industry executive. Used generators and smaller units tend to be more available than the 2 megawatt units, but companies targeting the enterprise sector may be wary of relaying on used units or smaller generators than those powering competing facilities."
This discussion has been archived. No new comments can be posted.

Generator Delays May Slow Data Center Projects More

Generator Delays May Slow Data Center Projects

Comments Filter:

  • 2 MEGAwatts?!?! (Score:5, Funny)

    by AKAImBatman ( 238306 ) * <akaimbatman AT gmail DOT com> on Wednesday October 25, 2006 @01:02PM (#16580584) Homepage Journal
    "Generator lead time for a nice 2 megawatt diesel engine is now up to a year for one generator," Josh Snowhorn of Terremark said in a panel at the NANOG conference earlier this year.

    HOLY MOLY, that's a lot of power! If you had 250 watt power supplies, all running at maximum, you'd be able to power 8,000 power supplies simultaneously! You could run a small town on a generator that large! Or to put it another way, you could use it to power a Diesel Locomotive capable of pulling dozens of fully loaded cars.

    Wow, just wow. That's just an incredible amount of power to be putting in a datacenter. Is it even possible for these centers to run off the grid?

    "So we can build all the raised floor we want, and then sit around and wait six months for a generator."

    The part that I don't undestand is this: Why do they need just one generator? If you're having difficulties obtaining a 2Mw unit, wouldn't it make sense to get two smaller units? You'd waste a bit of extra space, but you'd have redundancy that a single genearator couldn't offer.

    • Re:2 MEGAwatts?!?! (Score:4, Insightful)

      by CastrTroy ( 595695 ) on Wednesday October 25, 2006 @01:04PM (#16580648)
      Why don't they just go find an old diesel locomotive and convert that into a generator? Or get a really big diesel truck. A diesel generator isn't much more than a diesel engine. I see opportunities here for new manufacturers.

      • Re: (Score:2, Insightful)

        by MadEE ( 784327 )
        A deisel train engine is a generator the traction motors on the train are just that motors that are driven by the generator. It's actually a really cool design for a vehicle and allows for breaking using resistive loads across the traction motors (aka dynamic breaking)

      • Re:2 MEGAwatts?!?! (Score:5, Informative)

        by Volante3192 ( 953645 ) on Wednesday October 25, 2006 @01:12PM (#16580782)
        Well, converting an existing engine gives you the same, if not more, concerns as getting a used generator. As TFA mentioned, there's no problem finding used 2MW generators, but it's the stigma of using used equipment period.

        Add to that the changes and parts necessary to change engines geared to creating propulsion to engines geared to creating electricity.

        I doubt any company who considers their data center a key component of thier infrastructure to risk their backup solution on an untested refurbished generator.

        • Re: (Score:3, Informative)

          by account_deleted ( 4530225 )
          Comment removed based on user account deletion
        • Add to that the changes and parts necessary to change engines geared to creating propulsion to engines geared to creating electricity.
          Indulging in some Railroad Pedantry here, but locomotives use a diesel driven generator to drive electric motors at the drive wheels. You could, with a little jury-rigging, conceivably plop a diesel locomotive down next to your building and wire it up to your electrical main switch gear and just start it up...

          • Re:2 MEGAwatts?!?! (Score:5, Funny)

            by johneee ( 626549 ) on Wednesday October 25, 2006 @02:14PM (#16581924)
            Same with Hovercraft. Of course, the supply of hovercraft is quite a bit more limited than locomotives, but that's beside the point.

            Plus, you'd have to deal with all the eels.

          • Re: (Score:2, Interesting)

            by thatnerdguy ( 551590 )
            At least one city here in Quebec did just that during the Ice Storm of 1998. They wheeled a locomotive into the center of the city and used it to power important buildings.

        • Re: (Score:2)

          by 1u3hr ( 530656 )
          I doubt any company who considers their data center a key component of thier infrastructure to risk their backup solution on an untested refurbished generator.

          It's not in the front office. Who cares if it's not shiny and new? And a refurbished generator most certainly would not be "untested". Whether new or refurbished, you'd get a service contract.

      • Re: (Score:2)

        by Intron ( 870560 )
        What you want is an old diesel-electric loco like the Alco S-2 [railfan.net] which is basically a 1000 HP diesel generator and some electric motors that you won't need. Imagine having one of these sitting outside your data center battery room. Unlimited nerd points.
        • Do you know what the actual output of the generators used in modern diesel-electric locomotives is?

          I wonder what the voltage/frequency is. I assume that the motors are all AC, but if the generators are designed to produce low-frequency AC, then you're going to have issues using that in a standard datacenter, unless you are one of those rare places that does DC distribution (where you'd just run it thought a rectifier and the incoming frequency would be irrelevant, provided your rectifier could cope). The co

      • Re: (Score:2, Insightful)

        by drrck ( 959788 )
        You are correct that a GenSet is not that much more than a generator paired with a diesel engine, however the entire engine industry as a whole is somewhat supplier constrained.
        If a new company were to come and attempt to meet the demand present they would be fighting for the same parts and resources that other more established companies are.
        • So what you're saying is someone needs to start-up a company making fly-wheels and induction coils.
          -nB
      • Comment removed based on user account deletion
      • The big-boy generators (say, 20 megawatts) use gas-turbines for efficiency. You should see the size of units they install on cruise ships.
        • Actually it was in fashion a few years ago when tere were electricity shortages to install smallish gas turbine gensets to run your house or business independently. I remember reading about a McDonalds in Chicago that had done such a thing.

          As "the infrastructure" continues to fall apart, and energy prices are bound to spike again, and turbine technology becomes cheap and ubuquitous, I can see diesel turbines replacing IC as the power source of choice, even down to the hundreds of KW range.

      • Re: (Score:3, Informative)

        by Politburo ( 640618 )
        You'd have to look into the air pollution regulations. Depending on your location, there are fairly strict guidelines for generator engines. Staring in March, in NJ your generator cannot emit more than 1.5 g/bhp-hr of NOx no matter when you installed it (0.90 g/bhp-hr for new units). The current EPA standard (1997) for new train engines ranges from 5.5-14.0 g/bhp-hr (there is a stricter standard in the works). The current EPA standard (2004) for heavy-duty diesel vehicles is 2.4 g/bhp-hr. An old train or tr
      • they do. My old boss told me about the place they used to have an old diesel engine from a locomotive to use as a backup generator. They'd test it every couple of months... and it'd shake the building :-)

    • Re: (Score:2, Insightful)

      by SevenHands ( 984677 )
      "Wow, just wow. That's just an incredible amount of power to be putting in a datacenter. Is it even possible for these centers to run off the grid?" The generators have to supply enough on demand power to satisfy peak requirements. Examples could be getting the elevator(s) stuck between floors to where they're supposed to go. Air conditioning is another large power drain and in a datacentre, there are huge quantities of heat contributing components to deal with. Even backup lighting utilizes a surprisin

      • The generators have to supply enough on demand power to satisfy peak requirements. Examples could be getting the elevator(s) stuck between floors to where they're supposed to go. Air conditioning is another large power drain and in a datacentre, there are huge quantities of heat contributing components to deal with. Even backup lighting utilizes a surprising amount of power.

        So what you're saying is, the generator has to be able to power a small skyscraper in an emergency?

        • Small skyscraper (Score:3, Interesting)

          by Kadin2048 ( 468275 )
          I would bet that a datacenter probably uses as much electricity as a small skyscraper. Because of the high equipment densities, a 1-story datacenter filled with racks probably has HVAC requirements that are like a multi-story office building filled with cubes. IT doesn't have the same lighting requirements, but that's not nearly the draw that heating and cooling are.

          Actually, I bet that in many situations, if you just pulled the plug on a 'center, very bad things might happen to the equipment, aside from th

      • I've often wondered why they don't liquid cool these, or use ducted air cooling, then recover a portion of the waste energy. Any time you have a temperature difference you can get some returns. Considering these datacenters generate so much thermal energy, you'd think they would attempt to recover some rather than spend *more* energy to move it outside.

        This is why certain engine testing facilities use their dynos to power portions of their facility - instead of burning fuel and paying for electricity, they

        • There are units that can do that, sort of. They don't convert it into electrical energy, but you can use the extra heat to keep your building warm. There's a ski resort near me that runs entirely off of the grid, and uses one of those.
        • Duct the hot air from the data center out into the office buildings in the winter and duct cold outside in. In the summer though there's no free lunch.

    • Re: (Score:2)

      by Vihai ( 668734 )
      Why do they need just one generator? If you're having difficulties obtaining a 2Mw unit, wouldn't it make sense to get two smaller units?
      No, you would need at least 3, 1 MW units, otherwise you would double your probability of failure.

    • Re:2 MEGAwatts?!?! (Score:5, Informative)

      by jeffmeden ( 135043 ) on Wednesday October 25, 2006 @01:22PM (#16580990) Homepage Journal
      On your comment about data center size: Consolidation is the buzzword of the decade. 'Big boy' data centers start at 2MW and get BIGGER. It's not uncommon for a main site facility to be closer to 4MW. These are usually fed from the grid at the 14kV level, often from dual substations. Yes it's a lot of CPUs and supporting equipment, but thats the way we like em!

      Your question on two generators vs. one: Redundancy does NOT come from two half sized units doing a single job, it comes from two FULL sized units each doing half the job. Having two half sized generator units means losing one will cause failure. With a 10,000HR MTBF per unit (rough number used for demonstration) you get a system MTBF of 5,000HR, NOT what I would call redundant! You would need at least 3/2 redundancy (3 generators doing the work of 2) to have a reasonably reliable replacement for a single large unit.

      • Re: (Score:2)

        by Barny ( 103770 )
        As to this, i would have thought it a good idea to have a pair of shiny new generators (each capable of full load) as well as a little more cash for a used one of same size:

        A. you know its in working condition RIGHT NOW (would be nice to try and start up your pair of new generators to find them DOA)

        B. if there is a fault, a problem-over-time fault, with the new generators, you don't want them all dieing in the same week, haveing an old POS there could just save the company if/when excrement hits the air exc
      • So you could use say 5 500kW generators and as long as only one of them failed you would be fine. I bet you could get 5 or 6 500kW generators much cheaper (faster) than one of these high capacity ones on short supply. You could even use cheaper components with a shorter MTBF since the redundancy of the system can mask failures. I bet you could even reconfigure a data center for higher capacity simply by adding more of these inexpensive generators...

        Of course the real money is in RAIG controller units and
        • Two generators serving the load of one provides a 6x improvement in availability. 3/2 gives you about 3x, 4/3 2x, and 5/4 1.5x availability improvement.

          So, the more generators you add, the harder it becomes to actually have an improved reliability. If you did the 5/4, to get "equal" reliability you would need to do at least 6/4. (Same math works for parallel UPS modules.) The other challenge is that to do the preventative maintenance on a running generator takes about 4 hours of cool-down, plus 6 hours

    • Re:2 MEGAwatts?!?! (Score:4, Funny)

      by megaditto ( 982598 ) on Wednesday October 25, 2006 @01:23PM (#16581000)
      That's just an incredible amount of power to be putting in a datacenter.
      Not if your customers requires sharks with freaking laser beams attached to their heads.

      Why do they need just one generator? [...] wouldn't it make sense to get two smaller units?
      I agree. And just imagine the publicity value of having two smaller 1337 kW genenartors powering up your data center in tandem.

    • Re: (Score:3, Interesting)

      by Zak3056 ( 69287 )
      The part that I don't undestand is this: Why do they need just one generator? If you're having difficulties obtaining a 2Mw unit, wouldn't it make sense to get two smaller units?

      Keep reading the article--further down it says that large data centers (like MS and Google are building) need 20 or 30 2MW generators! My question: if you power requirements are that high, surely it must make sense to build your own powerplants? Multi-year construction time, I guess?

      I know there are several 60-75MW units under const
      • need 20 or 30 2MW generators! [...]I know there are several 60-75MW units under construction in the US now

        Comming soon on eBay : 100MW generators
        starting bid : 10$
        condition : mint
        reason : our startup went belly up.

        When the next speculation bubble burst, there're sure going to be a lot of diesel mastodonts left every were...

      • When you start building 75MW power plants for your data center maybe you need to start pushing for more power efficient computing systems.
    • The part that I don't undestand is this: Why do they need just one generator? If you're having difficulties obtaining a 2Mw unit, wouldn't it make sense to get two smaller units? You'd waste a bit of extra space, but you'd have redundancy that a single genearator couldn't offer.

      I used to work at a company who had a lot of equipment at an MCI enterprise-grade datacenter. I forget the exact terminology but they apparently have different classes of datacenters, and this was at the top of the line. It was phy

    • The part that I don't undestand is this: Why do they need just one generator? If you're having difficulties obtaining a 2Mw unit, wouldn't it make sense to get two smaller units? You'd waste a bit of extra space, but you'd have redundancy that a single genearator couldn't offer.

      Remember the old aeronautical engineering maxim of "twin engine planes have twice the rate of engine trouble as single engine planes". Point being, that sort of "redundancy" is only a net gain if you can actually "fly" with one of y

      • Re: (Score:2)

        by hpa ( 7948 )

        Remember the old aeronautical engineering maxim of "twin engine planes have twice the rate of engine trouble as single engine planes". Point being, that sort of "redundancy" is only a net gain if you can actually "fly" with one of your engines dead. If these datacenters require more than 1 megawatt, two 1 megawatt generators is actually a liability. The companion maxim to the above is "It's better to put all your eggs in one basket, so long as you've made sure you've got a REALLY STRONG basket."

        The right

    • Wow, just wow. That's just an incredible amount of power to be putting in a datacenter. Is it even possible for these centers to run off the grid?
      2MW at 33KV is only 60 amps, a 33KV 60A line is hardly going to be a challange to construct.

      so linking them to the grid really shouldn't be a problem.

      you'd be insane to try and run a datacenter off generators all the time. Electricity from deisel generators costs several times what grid electric costs even if you pay consumer prices for grid electricity.

      say 4 kilo
      • I wasn't really worried about the power lines. That's easy enough to handle, as demonstrated by the large transmission lines strung near my home.

        I was more worried about the grid being able to supply the capacity. When you start demanding power in the multi-megawatt range, you enter the territory of small power plants. A LOT of power plants would have difficulties supplying their normal load to residential and commercial customers, plus your industrial needs for power. So either you locate near some rather

    • Re: (Score:2, Informative)

      by Critical Facilities ( 850111 )
      Yes 2 Megawatts. I work running the engineering department for a major financial provider and our campus has two 2 Megawatt generators dedicated to our datacenter (as well as 2 more for the rest of the campus). We actually have most of the infrastructure in place for a third 2 Megawatt generator as we develop the need for power.

      You'd be surprised at how much the average load in KW/sq ft has increased in the last several years for a datacenter. Moore's law is FIRMLY in place and it's getting harder and h
    • All of your points are pretty much spot-on.

      The only major data center I've been in had 6 diesel 12.5kW generators. It also had power feeds from two different grids, water from two different mains, and a water tower that could keep enough chilled water to keep the place running for 2 hours if the chillers all failed. We're talking a MAJOR data center with plenty of redundancy. It did not need 2MW generators. It only needed a total of 75kW.

      There's also a hydroelectric dam located near where I grew up. It coul

      • Hogwash (Score:3, Insightful)

        by Critical Facilities ( 850111 )
        Our datacenter has about 24,000 sq feet of raised floor (not huge by datacenter standards) and we have 2 x 2MegaWatt Generators (as well as redundant utility feeds). To say that 2MW is too much power in this case shows your lack of understanding of what a "MAJOR data center with plenty of redundancy" means.

        Prior to working here, I was an engineer at one of the main datacenters for a big Texas based IT provider (think Ross Perot) and their datacenter had over 100,000 sq feet of raised floor. At last coun

    • Re: (Score:3, Interesting)

      by NerveGas ( 168686 )
      Most places will let each cabinet get two 20-amp circuits, for roughly 5 kilowatts. That means you can only provide power forup to 400 cabinets. But don't forget, they also have to power all of their own equipement, and that you don't want to run a generator at 100% capacity. That means that you could be talking about as few as 200 cabinets from that sort of generator.

      That's still a good number of cabinets in a datacenter, but it's not the unbelievable size that seems to jump out at you when you think of
    • That's only about 10,000 2-U rackable servers, not counting the AC and lighting. It's currently in fashion to install a 20 - 80 KW unit for your new McMansion.

      I've been in lots if data centers large enough to accomodate that many hosts.

      The question might be - Oh, no, are we overbuilding data center capacity AGAIN?

      The other question - these genrators aren't in the basement, are they.

    • Re: (Score:2)

      by AJWM ( 19027 )
      HOLY MOLY, that's a lot of power! If you had 250 watt power supplies, all running at maximum, you'd be able to power 8,000 power supplies simultaneously!

      Well, see, it's 1 megawatt to feed the power supplies of your computers, and another 1 megawatt to run the air conditioning to get rid of all that heat.

  • Easy, just buy... (Score:5, Funny)

    by DigitalRaptor ( 815681 ) on Wednesday October 25, 2006 @01:03PM (#16580602)
    Easy, just buy 400 5kw generators instead of 1 2000kw generator.

    You may have to hire more people to start them, though. And change the oil.

    • Re:Easy, just buy... (Score:5, Informative)

      by Detritus ( 11846 ) on Wednesday October 25, 2006 @01:43PM (#16581366) Homepage
      Then you have to keep them phased together. That can be complicated and error-prone. When you're operating on diesel power and lose a generator, the load gets dumped on the remaining generators, which can cause large frequency errors and brownouts. To put a spare generator online, you have to match frequency and phase with a system that is under severe stress. It can get ugly.
      • You may have missed the sarcasm in my [entire] post, but you are correct. Using a farm of smaller generators has many complications, not the least of which is inefficiency and managability.

    • We have a slashdot idiom for that -

      Imagine a beowulf cluster of these [hallequipment.com]!

  • There is a stopgap measure for this (Score:5, Funny)

    by eln ( 21727 ) * on Wednesday October 25, 2006 @01:04PM (#16580646)
    If you've built out your data center but the generators have not yet arrived, a quick and inexpensive solution is to buy several thousand hamsters and hamster wheels. These will keep your data center humming along in the event of an outage. If PETA starts protesting, sysadmins with larger wheels can be substituted. Most of them will find this activity a welcome break from the inane busywork in their everday lives.

    • I've actually found that I get a better wattage/food pellet ratio with gerbils. Had to put them in as a temporary measure a couple years ago, but they've worked so well since we installed them that we never bothered to upgrade.

      I heard of a guy in L.A. doing the same thing with ferrets, but I for one don't think I could stand the smell. Plus, the little tube-rats tend to escape and crawl under the flooring to nest, and that just raises all kinds of fire hazard issues.

      • Re: (Score:2)

        by eln ( 21727 ) *
        The problem with using gerbils is you always have to keep an eye out for Richard Gere.
    • Someone beat you to it. [otherpower.com]

  • I can understand completely (Score:4, Informative)

    by hurting now ( 967633 ) on Wednesday October 25, 2006 @01:06PM (#16580696) Homepage Journal
    You need to remember, its not just the computers & servers. Its the lights, the heating/AC, all other power needs. We have a generator at my work and it has to power EVERYTHING. 100 desktops 34 thin clients, 19 laptops, 34 servers, the A/C for the data center, the elevator, the emergency lighting, fire detection system, phones... you get the idea. We have a 30 KVA battery backup for the data center in case the generator doesn't kick on the right way.

    • Re: (Score:3, Interesting)

      by scuba_steve_1 ( 849912 )
      I studied this topic for work this year and learned that the current data center rule of thumb is that for every watt that you spend powering a server, you must spend another watt on air conditioning...and it gets worse.

      Now, with servers getting smaller (e.g., blades) and CPUs evolving to multi-core, heat and power usage density is increasing dramatically within the same floor space...and cooling it effectively ranges between difficult and nearly impossible...without ripping everything out and starting over
  • Wouldn't having datacenters switch over to DC power and then just using a massive battery backup system help with this? Or batteries more exspensive than having generators.

    I'd still keep a few generators in backup, but woulnd't you need less of them with DC?

    • Re:DC power? (Score:5, Insightful)

      by Silver Sloth ( 770927 ) on Wednesday October 25, 2006 @01:12PM (#16580790)
      You can keep your generator(s) running indefinitely. Certainly longer than any predictable power outage, but if you're running on batteries you're against the clock. What are you going to do when they start to run down - nip over to the 7-11 and buy all the AAs they've got?

      • Re: (Score:2)

        by kabocox ( 199019 )
        You can keep your generator(s) running indefinitely. Certainly longer than any predictable power outage, but if you're running on batteries you're against the clock. What are you going to do when they start to run down - nip over to the 7-11 and buy all the AAs they've got?

        Um, don't forget fuel for those generators! During an interview at one company a question about the companies on-site generators poped up. The company folks related a humorus tale of having the generator running, but having a very difficu
        • Some of the DC's I have worked with have moved to natural gas generators and some big bottles of gas as well. I have seen one that went as far as a diesel and a NG genset in a Nx2 setup. The nice thing about BG and LP is you can bury the tanks easily.

    • Re: (Score:2)

      by wwest4 ( 183559 ) *
      Yes, you'd need fewer, but the scale they are talking about is still way too cost prohibitive for batteries... it sounds like in this case, there are no mains... just generated power. Even so, you couldn't just use a battery to cover anything but super-brief mains outages, because the extended runtimes would probably require way too much in the way of battery cost, weight, floor space, etc.
    • Wouldn't having datacenters switch over to DC power and then just using a massive battery backup system help with this? Or batteries more exspensive than having generators.

      In large datacenters a battery backup system (UPS) will only last a few minutes, maybe a few hours. Generators can/will run indefinitely as long as clean fuel is available.

      Any datacenter that has a generator will (should) have UPS's in any event. It takes a few seconds/minutes for generators to start up and provide full power when a pow
    • Wouldn't having datacenters switch over to DC power and then just using a massive battery backup system help with this?

      Another comment. The vast majority of equipment in datacenters is designed to run on AC. Most companies simply buy servers from the likes of Dell, IBM, etc. which by default come with AC power supplies. Getting all the customers of a commercial datacenter to switch everything to DC would be cost prohibitive. Then there's the issue of other critical equipment like HVAC for cooling. they

    • Re: (Score:2, Informative)

      by RxScram ( 948658 )
      In a former life, I worked on large (800KVA) uninteruptible power supplies as a field service engineer. Normally, for highly mission critical data centers, as described in the article, the data center is powered by 2 different, independent power grids (if possible.) For example, there would be one feed from power company A, and another feed from power company B. These would be connected by using an automatic transfer switch (ATS) which would immediately switch to the other power feed if the main failed.

    • Wouldn't having datacenters switch over to DC power and then just using a massive battery backup system help with this? Or batteries more exspensive than having generators. I'd still keep a few generators in backup, but woulnd't you need less of them with DC?

      Batteries don't have nearly the power density of diesel fuel. The only advantage to battery backup is that it's instantaneous. Most datacenters DO have a battery system, but its purpose is merely to span the gap between the loss of mains power and bac

  • Suggestion (Score:5, Funny)

    by truthsearch ( 249536 ) on Wednesday October 25, 2006 @01:08PM (#16580724) Homepage Journal
    Take out a $10,000 simolian loan and place a natural gas power plant in a corner. It costs $9,000 simolians and produces "moderate pollution", so residential and commercial zones won't develop well nearby. But your city won't grow quickly without a lot of power.

  • not on my project (Score:4, Informative)

    by superstick58 ( 809423 ) on Wednesday October 25, 2006 @01:09PM (#16580752)
    I didn't see any delays on a project I am working on. We have 4 Gens in the 2MW range plus 3 gas/diesel gens in the 30MW range. Of course, this isn't for a datacenter, but I wonder if the generators would be the same.

    • Re: (Score:3, Informative)

      by AKAImBatman ( 238306 ) *
      We have 4 Gens in the 2MW range plus 3 gas/diesel gens in the 30MW range.

      What are the 4MW units if not diesel? If you're using Gas Turbines, that may explain why you didn't have the same supply problems.

  • Generator Delays (Score:3, Interesting)

    by kilodelta ( 843627 ) on Wednesday October 25, 2006 @01:18PM (#16580910) Homepage
    You would think that companies that make larger generators (I'm talking > 100kW) would understand that demand goes up during hurricane season, and things of that nature.

    We only encountered a one month delay on delivery of our 125kW natural gas fired generator. Our delay was mostly because of hurricane Katrina having struck the gulf coast. We had to pull some serious string but since we wouldn't be moving in until November of 2005, it didn't really impact us.
  • Wrap the entire building in foil. Add a layer of thin dielectric, then another layer of foil.

    Must remember to put rubber mats in all the entryways, of course.

    Bonus benefit: no need for a security system. Just give the phone number of the county coroner to the janitor, so that if someone tries to break in overnight, the bodies will be cleaned up before anyone arrives for work in the morning.

  • Skywest Airlines can tell you... (Score:5, Interesting)

    by DigitalRaptor ( 815681 ) on Wednesday October 25, 2006 @01:35PM (#16581214)
    I was having dinner with an IT guy from Skywest Airlines the other night and he told me in the last big power outage in St. George, Utah where they're based (and I live) their battery backups ran out quickly and there was a hardware problem between their generators and their IT department.

    The end result is that all of their servers and network equipment went out for hours, and they had to cancel a whole lot of flights costing the company well over $1,000,000.

    You can buy a lot of crappy gas generators at the Lowes across the street for $1M. I think I would have sent two guys there and two guys to the gas station to keep the essentials online.

    There is a lot to be said for redundancy. Redundancy is very important. You can't talk about redundancy enough. Seriously, it's better to be twice as redundant than only half as redundant. And three times as redundant is even better than twice!

    • Re: (Score:2)

      by ivan256 ( 17499 )
      The idea of getting a lot of crappy gas generators from Lowes to run in phase gives me a $1million headache.
      • Who said anything about running anything in phase?

        The last time I checked most of their HPUX server run on 110 volts, or ~maybe~ 220.

        If you have 10 mission critical servers, get 10 cheap generators and hook them up on a 1-to-1 basis.

        Very cheap, very effective, very easy.

        WAY cheaper and easier than canceling $1M in flights.

        • Mmmm, okay, they don't have to run in phase. So, assuming that you're talking about the emergency scenario described above (a multi-day blackout, batteries run out), and not talking about stocking them in advance, you run into a few issues:

          1. Lowes tends to run out of generators during blackouts. They're big, and they don't stock that many. Don't count on finding 10 of them during a blackout to run your 10 mission-critical apps.

          2. Assuming you find the generators, even forgetting about phase, ho
          • Here is the story [sltrib.com].

            1. Lowes is right across the street. They don't run out of generators after a 2 or 3 hour outage. Days, maybe; hours, no.

            2. The equipment was successfully running off of power from the UPS's. Each UPS is rackmount and powers 2 to 4 servers. Unplug the UPS from the wall, and plug it into a running generator.

            I've been in the Skywest server room. It's not much bigger than my home office, and no bigger than my master bedroom. We're not talking about a 2,000kw generator. We're talking
    • Gas stations are pretty useless during a power outage. Their pumps are electric.
      • Yes, but very seldom is a power outage so far spread that you can't get to a gas station with power within 10 minutes or so. At least not around here. We have 3 different power companies within 10 miles, and I've never known them to go off at the same time.

        • Well, around *here*, the whole damned Eastern US & Canada went out for couple of a days 3 or 4 years ago.

          We're talkin', no juice for several hundred miles in any direction.

          Luckily, the phone companies still have "talk battery" and I still have corded phones....
          • Well, I do think we'll see more and more of that in the future. The infrastructure is really overwhelmed, and the upgrades and maintenance is slow in coming.

            I've thought a lot about solar or generators and how to get off-grid, or at least not so dependent on the grid.

    • Re: (Score:2)

      by brarrr ( 99867 )
      1M$ worth of generators from lowe's won't work, as I'm fairly certain that they don't sell 3phase models. Running any kind of electronic equipment on a standard 2phase generator is asking for trouble.
      • So running a server off of a battery backup UPS plugged into a generator from Lowes for 2 hours is going to kill the server?

        You're gonna have to sell me on that...

        If you need to keep 10 servers online you buy 10 generators and 10 UPS's, plug each server into a UPS, each UPS into a generator (already running).

        I'm not talking about daisy-chaining the generators in to power the whole building.

  • One of the biggest delays we had building a new data center was in acquiring the transfer switches .... even once the generators and UPS systems were installed, they're useless without the transfer switches. There was about an 8-month lead time on those, and heaven help you if one of them arrives DOA. :)
  • Seems like the general environmental trend is to tear down the dams built in the 1930's (at least in California). Some of these dams also provide hydro-electric power. While most data centers are looking for on site power, you have to wonder if we're shooting ourselves in the foot by reducing the amount of power being generated from the dams to the grid. New natural gas and nuclear power plants are great if they get built but usually the NIMBY's will come out in force against a new plant.
    • Some of these dams also provide hydro-electric power

      I forget where I read it but I recall something about this being one of the key reasons Google was building a huge datacenter complex in Oregon on the Columbia river. Not only does it provide cheap electricity from hydro power dams but the river also provides cooling for the datacenter.
  • I agree with the article. I certainly wouldn't want to relay on small generators. I might rely on them, but I certain'y wouldn't relay on them.

    Slashdot editors: What do they do? Do they really exist? Do people actually get paid for those jobs?
  • Why not think about the problem from a higher level? The reason you need backup is because the grid isn't completely reliable. Why not remove your dependence on the grid instead of trying to paper over it's failures?

    Install a gigantic solar system / battery storage. Or, a natural gas power plant under your control. Or even wind power. Here in California I bet you could even save money in the long run... Eric
    • You can't use solar, because the servers have to run at night. You can't use wind, because they have to run on still days, too. Natural gas power plant? That's what they're doing, it's just (probably) diesel instead of natural gas, although that isn't necessarily the case, many are natural gas these days.
  • So we have another situation where reliance on centralization is stupid.

    Rather than building Googleplexes why aren't the IT mavens hiring P2P guys to build decentralized systems to farm out work to the desktops already sitting around?

  • Actually, a lot of the delays have been caused by the war. It takes time to build these generators, and when the war started a lot of the generators that were ready to ship were taken by the military (eminent domain) and shipped off to the middle east. Some of the new production also gave them priority. Everyone else has to wait...
  • From all the datacenters relocating outside the US? That seems to be the trend in my locale.

  • Clearing up a couple of misconceptions.... (Score:5, Informative)

    by Above ( 100351 ) on Wednesday October 25, 2006 @03:16PM (#16582916)
    I see a lot of posts that 2MW is a lot of power. Perhaps.

    There are many facilities out there with 10-30 2MW generators. I personally worked on a facility that had 18. 12 were for "critical load", that is, things fed via UPS's. This included all of the servers and networking equipment, the NOC, emergency lighting, fire supression systems and soforth. The other 6 were for "emergency load", these were things that did not need UPS protection (they could go down), just not for long. Virtually all of this load was air conditioning. So when the power went out the A/C's went off and came back 30-60 second later on generator, while the servers and all stayed up the same time.

    While big, it's far from the largest facility out there.

    Now, why would you buy one generator? Well, many buildings use bus variations (the whole N+1, or N+2 thing). So you build your data center for all 18 generators (as above), but install 6 and 3, or half the capacity. You now wait for the building to fill to 25-30%, and then start adding more generators, one at a time. However, they now have to be matched to the other generators.

    Now, why are data center generators special? Well, to switch from one AC source to another the two loads must be approximately in phase (there is some tolerance, but it's small). So in order to be able to switch between generators, switch from UPS to generator, and all that other stuff you need additional circuitry to keep the generator just so. While the engine block and generator were the same as say a diesel locomotive, there were some additional sensors, lots of additional computer control, and some additional quick start features.

    Most sites want their backup generators to be "ready to load" in 15-30 seconds. Indeed, several manufacturers make 10 second ready to load units. However that requires things like engine oil and coolant heaters that basically keep the entire motor warm 24x7x365. These are not found on industrial generators or locomotives.

    So, they really are special, high tech tight tolerance units designed to work in a system. When you connect to WoW or your other large game you're probably one one of a thousand servers run by that company, who is one of but hundreds of companies in the larger colo facilities. It's not uncommon to find 50,000 servers under a single roof. So you need lots, and lots, of 2MW generators.

Slashdot Top Deals

Get hold of portable property. -- Charles Dickens, "Great Expectations"

Close