Forgot your password?
typodupeerror
Power The Internet

Data Center Power Failures Mount 100

Posted by timothy
from the send-money-drugs-and-sealed-lead-acid-batteries dept.
1sockchuck writes "It was a bad week to be a piece of electrical equipment inside a major data center. There have been five major incidents in the past week in which generator or UPS failures have caused data center power outages that left customers offline. Generators were apparently the culprit in a Rackspace outage in Dallas and a fire at Fisher Plaza in Seattle (which disrupted e-commerce Friday), while UPS units were cited in brief outages at Equinix data centers in Sydney and Paris on Thursday and a fire at 151 Front Street in Toronto early Sunday. Google App Engine also had a lengthy outage Thursday, but it was attributed to a data store failure."
This discussion has been archived. No new comments can be posted.

Data Center Power Failures Mount

Comments Filter:
  • by BillyMays (1587805) on Monday July 06, 2009 @07:58PM (#28601737)
    I'm guessing that the majority of these were caused by leaks or spilled drinks. If only you guys had listened to me and gotten Zorbeez(tm)[SOAKS UP 10x ITS OWN WEIGHT!] [wikipedia.org].

    -B. Mays
    • Re: (Score:1, Offtopic)

      by Puppet Master (19479)
      Billy, welcome back!!!
    • by Schemat1c (464768)

      I'm guessing that the majority of these were caused by leaks or spilled drinks. If only you guys had listened to me and gotten Zorbeez(tm)[SOAKS UP 10x ITS OWN WEIGHT!] [wikipedia.org].

      Even that wouldn't work. What you have here is your textbook Pepsi Syndrome and only a President in yellow booties can fix it.

  • by StaticEngine (135635) on Monday July 06, 2009 @08:16PM (#28601939) Homepage

    "A blown transformer appears to be the culprit"

    I'd heard the new movie was crude, but I didn't realize how crude it actually was!

  • We had an outage today. Our servers are hosted with Genesis Hosting, which suffered an outage from their ISP; XO Communications in Chicago. Anyone know what happened?
  • Outages (Score:3, Interesting)

    by Solokron (198043) on Monday July 06, 2009 @08:27PM (#28602049)
    Outages happen more than that. We have been in several data centers, ThePlanet and The Fortress both have had major outages in the last two years which has affected business.
    • Re:Outages (Score:5, Interesting)

      by JWSmythe (446288) <jwsmythe@jws[ ]he.com ['myt' in gap]> on Monday July 06, 2009 @09:05PM (#28602421) Homepage Journal

          I've had equipment and/or worked in many datacenters over the last decade or so. I've worked with even more clients who have had equipment in other datacenters.

          I've only experienced 3 power related outages that I can think of.

          One was a brownout in that area, which cooked the contactors that switched between grid power and their own DC room.

          One was an accident, where a contractor accidentally shorted out a subpanel, and took out about a row of cabinets. I was there for that one. I saw the flash out of the corner of my eye, and by the time I turned my head, he was just flying into the row of cabinets.

          One was a mistake in the colo, where there was a mislabeled circuit, so they cut power to 1/3 of one of our racks.

          There have been even more outages related to connectivity problems. With one major provider who was just terrible (and is now out of business), they had a fault about once a week or less. Every time we called, they said "there was a train derailment that cut a section of fiber in [arbitrary state], which effected their whole network." It was funny at first, but annoying when we started questioning them about why there was no news about all these train derailments. We had to make up our own excuses for the customers, because we couldn't keep telling them the BS story the provider gave. We were smart about it though, and at least had decent excuses, and the whole staff knew which BS story to give for a particular day. The sad part was, we had a T3, and that was huge at the time.

          At my last job, they wanted a full post-mortum done on any fault. If a customer across the country suffered bad latency or packet loss, it was our job to find out why and "fix" it. The management wouldn't accept that there are 3rd party providers who handle some of the transit. So, we'd call our provider demanding it to be fixed (which they couldn't do), and then call the broken provider (who hung up since we weren't their customer), and then got reamed by the boss because we couldn't fix it. Delay tactics worked best after a while. If you're "investigating" a problem long enough, and hold the phone up to your ear enough, the problem will likely be fixed by those who really can. We'd still log a ticket with our provider, because the boss would eventually call the provider referencing the ticket number, and find out there was still nothing that could be done.

          There's pretty much guaranteed to be a fault of some sort between two points on the Internet every day. All anyone can really do is make sure it isn't with your own equipment. That's something I always did before calling to complain about anything. It's embarrassing to hear "did you reboot your router?" and that turns out to really be the problem.

          The only real solution to this is, redundancy. Not just in one facility, but across multiple facilities. If you spread things out enough, sure an isolated problem will effect some people, but not everyone. You want a service to be reliable, redundant machines in each datacenter is the only way to go. When I was running the network (and everything technical) at one job, a datacenter outage wasn't a concern, it was just a minor annoyance. I filed a trouble ticket, and told them to call me when it was fixed. We'd demand reimbursement on the outage time, and made them handle the difference on our 95th percentile bandwidth charges at the end of the month. I wasn't going to take a hit on the bill just because they had an outage in a city, and my other cities had to take the traffic during the outage. When your bill is measured in multiple Gb/s, you have a little more say in how they handle the billing. :)

      • by Muad'Dave (255648)

        ... which cooked the contactors that switched between grid power and their own DC room.

        I read that as contractors. I apparently saw 'contractor' in the next sentence and did the switcheroo. I was going to call you callous for using the term 'cooked'.

        FYI, arc flash [wikipedia.org] is not something to be taken lightly (no pun intended). It's dangerous as all get-out in high voltage panels that have a lot of available fault current. A typical 480V, 20kA fault [wikipedia.org] can release the same amount of energy as 1.5 lbs of TNT.

        • by JWSmythe (446288)

          I figured some people would make that mistake. Some others (like you) actually know the difference. Ya, I'd prefer to not have a cooked contractor. They kinda smell, and it tends to make other contractors not want to work with you. :)

          I was very happy to have not been at the site when it happened, but I did have to fly up to check all of our equipment.

          About half our servers wouldn't come back online, either due to power or networking faults. The network switch

  • Be Redundant! (Score:5, Insightful)

    by drewzhrodague (606182) <drew@zhr[ ]gue.net ['oda' in gap]> on Monday July 06, 2009 @08:29PM (#28602063) Homepage Journal
    Anyone seriously oncerned about their web applications, will have redundant sites, and a way to share the load. Few people pay attention to the fact that DNS requires geographically disparate DNS servers *, such that even in the event of a datacenter fire (or nuclear attack), there will still be an answer for your zone. Couple this with a few smaller server farms in separate places, and there won't be any problems. I went to look it up on wikipedia, but didn't find out where it is required for authoritative DNS servers to be in separate geographic regions. Where did I read this, DNS and BIND?
    • Re:Be Redundant! (Score:5, Informative)

      by W3bbo (727049) on Monday July 06, 2009 @08:54PM (#28602333)
      The DNS RFCs advise that zone nameservers should be in separate subnets. Specifically RFC 2182 recomends that secondary DNS services be spread around geographically.
    • Re: (Score:3, Funny)

      by Firehed (942385)

      In the event of a nuclear attack, you probably have more pressing issues to deal with than your server uptime.

      • by rdnetto (955205)
        But then how will you know who is attacking you, and where to go? Not to mention how to best shield yourself from radiation...
    • Re:Be Redundant! (Score:5, Interesting)

      by JWSmythe (446288) <jwsmythe@jws[ ]he.com ['myt' in gap]> on Monday July 06, 2009 @09:15PM (#28602505) Homepage Journal

          Be nice, people don't read the books nor RFC's any more.

          At the biggest operation I ran, I had redundant servers in multiple cities, and DNS servers in each city. If we lost a city, it was never a big deal, other than the others needing to handle the load. With say 3 cities, a one-city outage only accounted for a 16.6% increase in the other two. Each city was set up to handle >100% of the typical peak day traffic, so it was never a big deal. I don't think we ever suffered a two-city simultaneous failure, even though we simulated them by shutting down a city for a few minutes. Testing days were always my favorite. I loved to prove what we could or couldn't do. I peaked out one provider in a city once. We had the capacity as far as the lines went, but they couldn't handle the bandwidth. It was entertaining when they argued, so I dumped the other two cities to the one in question, and they were begging me to stop. "Oh, so there is a fault. Care to fix it?"

          I could quantify anything (and everything) at that place. I could tell you a month or so in advance what the peak bandwidth would be on a given day, and how many of which class of servers we needed to have operating to handle it. I classed servers by CPU and memory, which in turn gave how many users and how much bandwidth each could do. I only wanted our machines to every peak out at 80%, but sometimes it was fun to run them up through 100%. I set the limits a little low, so we could run at say 105% without a failure.

          Such information let us know if we had a server problem, before we knew we did. I'd notice a server was running 10% low, and that really means that it is going to fail. We'd watch for a little while, and it would. :) We'd power it down, and leave it in the datacenter until we had another scheduled site visit.

      • With say 3 cities, a one-city outage only accounted for a 16.6% increase in the other two.

        You're mistaken. Each of the other two cities would see their load increase 50%.

        • by JWSmythe (446288)

          Normal Operations:
          City 1 - 33.3%
          City 2 - 33.3%
          City 3 - 33.3%

          City 1 stops:
          City 1 - 0%
          City 2 - 33.3%
          City 3 - 33.3%

          Other cities take up the slack:
          City 1 - 0%
          City 2 - 33.3% + 16.6% = 49.9% (mol)
          City 3 - 33.3% + 16.6% = 49.9% (mol)

          If you are really bent about the missing 0.2%, you can work it out in fractions instead. :)

          City 1 - 0/3
          City 2 - 1/3 + 1/6 = 3/6 = 1/2
          City 3 - 1/3 + 1/6 = 3/6 = 1/2

          I have a better time adding and rounding decimals in my head. Since nothing on the Internet is actually perfect, 1% off on

          • Yes. I knew how you calculated it. You're just confused about what you're calculating.

            "City 2 - 33.3% + 16.6%" -- note that the load on City 2 just increased 50%. If it were making 33 widgets and had to make another 16 widgets, it's now making 50% more widgets.

            To be more explicit, each city in your analysis is taking up 16.6% extra of the total load, which means that each city is individually seeing a 50% increase in its own load.
            • by JWSmythe (446288)

              Well, ya, 50% more than it had been handling before. I see.

              My concern was the percentage of totals. That was divided up by the available servers, and then each server was calculated to know it could handle. We had generally 3 classes of servers. They could be considered "old", "not so old", and "new". :) We didn't quite class them like that, but it would be a decent evaluation. If "old" could handle 1x, "not so old" could handle x2, and "new" could handle 3x. It wasn't u

    • by mcrbids (148650)

      It's required that you have two name servers when you register a domain name.

      Physical separation is not required. It's just good practice. (I do, in separate cities on different ISP networks) Having separate nameservers in different geo regions is implicit because you have to register at least two for each domain name. I've seen some people game this by having a single nameserver with two IP addresses, which strikes me as the height of stupidity, but it's not happening on my watch.

      • I've seen some people game this by having a single nameserver with two IP addresses, which strikes me as the height of stupidity

        If everything referenced by the DNS records (web and email services or whatever) is hosted on the same machine as the name server, then it isn't particularly stupid. It's just a small operation that has a single point of failure; redundant DNS isn't going to change that.

        • by vlm (69642)

          If everything referenced by the DNS records (web and email services or whatever) is hosted on the same machine as the name server, then it isn't particularly stupid. It's just a small operation that has a single point of failure; redundant DNS isn't going to change that.

          WITH the single exception I know of, that incoming email will bounce with something like "domain not found" if there is no DNS response at all, vs if there is DNS but the MX record servers can't be reached it'll silently retry. Some totally brain-dead MTAs will bounce, but anything remotely usable will transparently retry later and no one will know it happened.

          And it's not so much a "small operation", as a non-relevant risk. People have a certain expectation of how (un-)reliable email is, due to filtering

          • WITH the single exception I know of, that incoming email will bounce with something like "domain not found" if there is no DNS response at all, vs if there is DNS but the MX record servers can't be reached it'll silently retry.

            Common myth but quite untrue (try it for yourself). If there is no response from any DNS server then it will be considered a temporary failure and delivery attempts will continue at intervals in the background just as if the MX target(s) were not responding.

            Only if a server can be re

    • by Guspaz (556486)

      While geographic diversity is certainly an excellent goal, it's not always that simple. My ISP's network core was located in the Peer 1 suite at 151 Front (whose UPS caused the fire). Power was cut to Peer 1's suite, but not the rest of the building (151 Front has independent power/cooling/etc. per-suite to the extent where each tenant is responsible for getting their own solution).

      Redundant power sources could have mitigated the issue had there not been a fire; running two independent circuits to critical

    • Re: (Score:3, Interesting)

      by ls671 (1122017) *

      Best solution for big outfits is to have at least this setup:

      1) One party being the main contractor. This party doesn't do ANY hosting per say but only manages the fail-over strategy, doing the relevant testing once in a while.

      2) A second party being involved in hosting and managing data centers.

      3) A third party, completely independent from party 2, a competitor of 2 is preferable, which also does hosting and manages data centers.

      It is the same principle when you bring redundant internet connectivity to a b

      • by vlm (69642)

        1) Have the fiber from one provider come into the building from, say, the north side of the building.

        2) Have a competitor, unrelated business wise, that doesn't use the same upstream providers bring his fibers in from the South side of the building.

        3) Discover that both fiber runs connect to the same L.E.C. vault 100 feet away and then run parallel the whole way back to the same central office, and/or they both are carried on the same SONET ring just connected to different ADMs (which would at least give you ADM redundancy).

        Seriously though, step 3) is get a copy of the DLR / CLR of the local loop, and have someone analyze them. Of course how the circuit is designed is not necessarily how it is actually routed, which is even funnier.

        Everyone in the t

        • by jra (5600)

          President Jimmy Carter.

          Nuclear attack evac test.

          Lots of embarassed people.

          It is *epically* difficult to get and keep true physical diversity.

    • It wasn't me!!
    • by sjames (1099)

      On the other hand, if your servers are all down, is there a lot of point to people knowing their IP address?

      One secondary DNS per location is just fine. If there is just one datacenter, then use 2 DNS servers there.

      It is recommended, but not required. The server police won't take your hardware away if you don't do it.

  • by Neanderthal Ninny (1153369) on Monday July 06, 2009 @08:42PM (#28602201)

    My wild guess is they are deferring preventative maintenance on these data centers so we are seeing these major outages now. Fire suppression, UPS, transfer switches, generators, distribution panels, transformers, network gear, server, storage devices and other gear will fail if you don't maintain them properly. As loads increase, the equipment will fail earlier and my guess the people have pushed the limit of this equipment beyond they the lifespan of load rating.

    • Power Fail Often (Score:3, Interesting)

      by blantonl (784786)

      Frankly, if data centers are going to proclaim their redundancy, they should test by power failing the entire data center once every two weeks at a minimum. A data center that goes down twice in a month would get ahead of any issue pretty fast. Lessons learned from the staff and the management are very valuable.

      The marketing messaging:

      "We power fail our data center every two weeks to ensure our backups work..."

      Sound scary? Just think about the data center that has never been through this process. at tha

      • by aaarrrgggh (9205)

        Semi-monthly pull-the-plug tests would reduce reliability. Monthly load tests on generator and a battery monitoring system ensure electrical reliability quite effectively. Only the most inadequate facilities fail to do this.

        The larger problems come from improper change control, a lack of scripting, or an abnormal failure mode. Lack of testing and maintenance is a real problem, and in data centers it is far too often that it is caused by the IT team not understanding the risks of inaction. If you have an ac

      • Frankly, if data centers are going to proclaim their redundancy, they should test by power failing the entire data center once every two weeks at a minimum.

        I disagree. If you perform a full load test of your facility every 2 weeks (or heaven forbid more frequently) you will be buying LOTS and LOTS of UPS batteries. Not to mention putting additoinal wear and tear on your generators, transfer switches, UPS Modules, Control Cabinets, etc.

        You are correct, data centers should do "pull the plug" tests, but not as frequently as you suggest, otherwise they'll effectively be reducing their availability by introducing more risk to the equation.

  • Are they over drawing the power out the units? poor battery that blow up? Not having the right wire gauge? Not cooling the power buses and switches?

  • by Anonymous Coward on Monday July 06, 2009 @09:06PM (#28602441)

    Surprise surprise...there's a downside to consolidation. Hey morons, the internet was invented as a means to ensure redundant communications paths given nuclear warfare. The old central switch (physical switching) was seen as too cumbersome and vulnerable. Now that we have wonderfully redundant communications, and have done away with most of the downsides of physically distributed systems, morons are building logically centralized systems.

    NEWSFLASH - Redundant communications and physical virtualization do very little for you if you build a logical mainframe.

    Truly distributed systems must be physically AND logically DISTRIBUTED with redundant comms paths in order to gain the full benefits of decentralization. (e.g. Distributed isn't distributed if all your authentication is done at one site or all your traffic must pass through .)

    • by Narcocide (102829)

      Someone mod this coward up.

    • Re: (Score:3, Insightful)

      by ls671 (1122017) *

      >> "morons are building logically centralized systems"

      I have worked with such a moron doing architecture on a big government project ;-)) unbelievable...

      His argument was that "The government likes centralized systems" ;-))

    • 1) The internet wasn't redundant, ARPANET was redundant. The internet hasn't been able to withstand a nuclear attack since it was put online.

      Putting all your eggs in one basket is nothing new under the sun. You ever see Ma Bell's idea of a "redundant" circuit? Two wires in the same condiut. But at least Ma Bell was doing it out of thriftiness and laziness, not ignorance and superstition.

    • Re: (Score:3, Insightful)

      by BBCWatcher (900486)

      No, I think you have it exactly backwards, or at least you're missing an important nuance. It's really, really expensive to duplicate everything across two (or more) data centers. And it's full scope increase in IT costs: most or all cost categories increase. We're talking more than double the costs, in round numbers. Beyond the cost, it's very hard technically to recover hundreds or thousands of servers simultaneously or even near-simultaneously, because you are typically trying to recover not hundreds or

  • by asackett (161377) on Monday July 06, 2009 @09:16PM (#28602519) Homepage

    ... saying that it's time to reconsider cost cutting measures. In 15 years in the field I never saw a well designed and well maintained critical power system drop its load. I saw many poorly designed and/or poorly maintained systems drop loads, even catching fire in the process. One such fire in a poorly designed and poorly maintained system took the entire building with it, data center and all. The fire suppression system in that one was never upgraded to meet the needs of the "repurposed space" which was originally a light industrial/office space.

  • by Anonymous Coward

    See story of Qld Health datacentre disaster on ZDnet recently:
    http://www.zdnet.com.au/news/hardware/soa/Horror-story-Qld-Health-datacentre-disaster/0,130061702,339297206,00.htm

  • Even worse... (Score:5, Informative)

    by Anonymous Coward on Monday July 06, 2009 @09:28PM (#28602583)

    I'm one of the guys that services the security system in Fisher Plaza. The damn sprinklers killed half my panels near the scene. Turns out they use gas suppression methods in the data centers, not so much in the utility closets. And the city of Seattle REQUIRES sprinklers throughout the building, even right over the precious, precious servers. In defense of the staff there however, they do not keep them all charged 24/7. Other then that, I have no more info, as they're pretty locked down.

    • Stupid city of Seattle, prioritizing fire safety over property. The nerve! At least the staff knows better than fully trained fire professionals and judged that the system was unnecessary.
    • We had the largest data center in Seattle and believe me we did NOT have sprinklers in our data center. Saying that the city required them sounds like a cop-out to me. Our disaster recovery plan was pretty solid with off-site recovery several thousands of miles away within minutes. Unfortunately, we did not have a disaster recovery plan for being seized by the federal government and sold to a competitor.
  • by jd2112 (1535857)
    I work for a company that makes high-end datacenter power systems, this should be good for business once the trade rags the CxOs read report on the millions and millions of lost business.

    Or at least it will keep the sales staff busy writing up quotes that will be rejected for being too expensive (although much less than the cost of a prolonged outage...)
  • by Dirtside (91468) on Monday July 06, 2009 @10:05PM (#28602833) Journal
    ...what is the normal (historical) rate of data center power failures, and how does the recent spate compare? Five in a week sounds severe, but what's the normal worldwide average? I can imagine that with thousands of data centers around the globe, there's likely a serious failure occurring somewhere in the world once every couple of days.
  • This is why you should look into company's with geographical diversity such as Ubiquity (http://www.ubiquityservers.com) or various other companies in the data center market.
    • by symbolset (646467)

      An individual system will always eventually fail. A single part, a server, a power system, a communication system. We have an architecture of related systems that can survive the isolation of one group or large swaths of systems with no impact on the persistence or availability of data.

      We call it life.

  • by Velox_SwiftFox (57902) on Monday July 06, 2009 @10:32PM (#28603057)

    "Major" data center or not, the one your company employing you at the time is using is the important one.
    In my experiences, data center backups fail about a third the time power is interupted somewhere.

    Servers in an Oakland California center were the victim of the loss of one of three power phases, while the monitoring that would have switched over to the diesel generators was looking at the power level of other phases. UPS systems ran out of power. An extra level of redundancy in the form of rack mount UPSes allowed servers to shut down properly despite the data center's loss of routing.

    Data center #2 was the victim of a simple power outage and immediate failure of the main data center UPS system. According to a security guard I talked to, "it exploded". The diesel backup never had a chance to start.

    Then the doubly-sourced Power Distribution Unit supplying a rack at a third ISP failed in a way that turned off both sources supplying the servers.

    Hint: Add an extra level of UPS redundancy and safe shutdown software daemons, at least. Multiple data centers if you need more nines.

  • Rackspace in Dallas (Score:5, Informative)

    by Thundersnatch (671481) on Monday July 06, 2009 @11:01PM (#28603323) Journal

    We're a Rackspace customer in their DFW datacenter. This is the third power-related outage they've had in the last two years at that supposedly world-class facility.

    The first wasn't really their fault: truck driver with health condition runs into their transformers. Generators kick in, but chillers don't re-start quickly enough. Temps skyrocket in minutes, emergency shutdowns. Maybe the transformes should have had some $50 concrete pylons surrounding them?

    The second outage was the result of a botched generator upgrade.

    This latest outage was the result of a botched UPS maintenance.

    None of the outages was long enough to trigger our failover policy to our DR site, but our customers definitely noticed.

    While their messaging has been very open and honest about the problems, and the SLA credits have been immediate, we pay them nearly $20K per month. Nedless to say, we are shopping, and looking into a "multiple cheap colos" architecture instead of "Tier-1 managed hosting". Nothing beats geographic redundancy.

    • by zonky (1153039) on Monday July 06, 2009 @11:40PM (#28603659)
      That isn't quite right, re: their 2007 outage.

      It wasn't a power issue as such, but the way their chillers reponded to two quick power fluctuations in succession:

      This is what they said:

      Without notifying us, the utility providers cut power, and at that exact moment we were 15 minutes into cycling up the data centerâ(TM)s chillers. Our back up generators kicked in instantaneously, but the transfer to backup power triggered the chillers to stop cycling and then to begin cycling back up againâ"a process that would take on average 30 minutes. Those additional 30 minutes without chillers meant temperatures would rise to levels that could irreparably damage customersâ(TM) servers and devices. We made the decision to gradually pull servers offline before that would happen. And I know we made the right decision, even if it was a hard one to make.
    • Have you considered Atlantic.Net as your collocation provider? We have a Data Center in Central Florida and our pricing is competitive. Give me an email at joshw atlantic.net for a quote. Josh Wieder Atlantic.Net
    • We're a Rackspace customer in their DFW datacenter. This is the third power-related outage they've had in the last two years at that supposedly world-class facility.

      The first wasn't really their fault: truck driver with health condition runs into their transformers. Generators kick in, but chillers don't re-start quickly enough. Temps skyrocket in minutes, emergency shutdowns. Maybe the transformes should have had some $50 concrete pylons surrounding them?

      The second outage was the result of a botched generator upgrade.

      This latest outage was the result of a botched UPS maintenance.

      None of the outages was long enough to trigger our failover policy to our DR site, but our customers definitely noticed.

      While their messaging has been very open and honest about the problems, and the SLA credits have been immediate, we pay them nearly $20K per month. Nedless to say, we are shopping, and looking into a "multiple cheap colos" architecture instead of "Tier-1 managed hosting". Nothing beats geographic redundancy.

      Thundersnatch - I'm sorry to hear you've had the same types of troubles over the past few years at Rackspace. I can't blame you for feeling burned by it, you're paying a good amount to have many 9's of uptime because whatever you're running online is clearly critical to your business. It sounds like this is definitely something we can help you out with. The company I work for (INetU) focuses on working with businesses who run critical operations online. Do you have some time for a call? My contact info

  • by bgd73 (1300953)
    my pc. it is 3400mhz and all the data center host sites of my interest. for the first time in 21020 hours of perfect runtime the ups saved it in a series of northeast thunderstorms coniciding with the outages around the globe. the permeating physics were too much for the world, reverberating to an unintentional master of perfect float, hardly busy, waiting to send a message. Believe it or not.
  • It was a terrible movie, not the kind I like to watch anyway, but for some reason I felt compelled to view the damned thing twice in two days.

    The Big Bad Threat in the film was all about something called a Fire Sale, ("It All Has To Go"), where the population's fear level is spiked up into a panic by a group of bad guys deliberately crashing the national infrastructure by way of hacking all the most important computer systems. --All to create a giant distraction so that the stock market could be plundered

  • Sunspots, Anyone? (Score:3, Interesting)

    by Craig Milo Rogers (6076) on Tuesday July 07, 2009 @05:24AM (#28605331) Homepage

    All these data centers failed at roughly the same time as the sunspots returned [space.com], but that's just a coincidence, right?

  • I'm almost thinking of taking UPS out of the loop here. They cause nearly all the downtime we have. It would be better to just let the machines power off rather than allowing the UPSs to CAUSE the machines to be taken offline. At least if the UPS isn't in circuit, the machines power back up again when the power comes back, but if there's a fault with the UPS or it's batteries, then the machines stay offline until the batteries have been replaced.

    Why the hell the idiots that design UPSs seem to think it's a

    • by C_Kode (102755)

      The UPS isn't the issue and taking it out of the loop would be absolutely dumb. The issue is proper maintaining, testing, most of all the design of HOW it's installed. (scheme if you will)

      We had a power failure (UPS failure) at our backup facility (Peer1 in lower Manhattan) The problem was, they had a UPS, but if that UPS failed, there was nothing behind it. What happen was the UPS failed and the passthough controller burnt up during the failover and took the entire wing down. They have replaced the ol

    • I'm almost thinking of taking UPS out of the loop here.

      That would be suicide. If you think that all of the pieces of gear in a data center could go down and then come back up with no problem, I may have a bridge that you can buy for a very reasonable price.

      Why the hell the idiots that design UPSs seem to think it's a good idea to prevent them turning on if they sense a problem with the batteries is beyond me. Why not let the machines power back up but just make a loud beeping noise until the batteries are fixed. Don't they realise that most of the time the UPS will only properly test the batteries when there's an actual power cut?

      Oh where to start. First of all, enterprise level UPS Systems (not the little "shoe box" APC unit under your desk) do not shut down on battery issues. At worst, during a catastrophic failure, they will trip to bypass. If properly arranged in a 2N or 2N+1 configuration, your Critical Load will migrate

    • by asackett (161377)

      Ain't it amazin' that all those UPS engineers whose workdays are nothing but thinking about the design of systems all arrive at the same conclusion and refuse to start their machines if the battery is failed? It's just incredible that these guys with EE degrees come out of college smart enough to do right things and somehow get it so wrong. Oh wait... I'm being sarcastic, which isn't very nice. I'd never suggest that the idiot isn't them, might instead be you, because that would be bad for my karma.

    • by jabelli (1144769)

      That's because you keep buying the cheap "Back-UPS" home computer crap to run servers on. I'm using a 15-year-old Smart-UPS 600. I've replaced the batteries once. When the original set wore out, it didn't refuse to power up, it complained about the batteries with das blinkenlights and the warning beeper until they were replaced.

Murphy's Law, that brash proletarian restatement of Godel's Theorem. -- Thomas Pynchon, "Gravity's Rainbow"

Working...