Slashdot is powered by your submissions, so send in your scoop

 


Forgot your password?
Close
typodupeerror
×
Australia Google Power

When the Power Goes Out At Google 135

Posted by Soulskill from the larry-and-sergey-ghost-stories dept.
1sockchuck writes "What happens when the power goes out in one of Google's mighty data centers? The company has issued an incident report on a Feb. 24 outage for Google App Engine, which went offline when an entire data center lost power. The post-mortem outlines what went wrong and why, lessons learned and steps taken, which include additional training and documentation for staff and new datastore configurations for App Engine. Google is earning strong reviews for its openness, which is being hailed as an excellent model for industry outage reports. At the other end of the spectrum is Australian host Datacom, where executives are denying that a Melbourne data center experienced water damage during weekend flooding, forcing tech media to document the outage via photos, user stories and emails from the NOC."
This discussion has been archived. No new comments can be posted.

When the Power Goes Out At Google More

When the Power Goes Out At Google

Comments Filter:

  • aren't there any people in the data center to tell them that yes there has been a power outage, so and so machines are affected, etc? sounds like all they have is remote monitoring and if something happens than someone has to drive to the location to see what's wrong

    • Re: (Score:2, Troll)

      by dch24 ( 904899 )
      What I want to know is, what caused the outage?

      The post on the google-appengine group details all the things they did wrong and are going to fix, after the power went out. Fine, I have to plan for outages too. But what caused the unplanned outage?

      • Re:what about having people onsite? (Score:4, Insightful)

        by nedlohs ( 1335013 ) on Monday March 08, 2010 @01:12PM (#31402160)

        Who cares?

        Power failures are expected, what you can do is have plans for when they occur - batteries, generators, service migration to other sites, etc, etc. Those plans (and the execution of them) are what they had problems with.

        • Power failures are expected, what you can do is have plans for when they occur - batteries, generators, service migration to other sites, etc, etc

          Too small scale, too complex, too much human intervention and too unreliable. Minimum of 2 datacenters on opposite sides of the world and you only send half the traffic to each. When the first vanishes the second picks up the traffic. The exact mechanism depends on the level of service you want to provide.

           

        • Re: (Score:2)

          by afidel ( 530433 )
          No, the question is why did the end users *see* the power outage? I would guess Google's insistence on using cheap motherboards with local battery and non-redundant PSU's bit them in the butt here. In a properly designed and maintained datacenter the loss of main power and a single generator won't take out a single server or piece of networking gear, but Google has gone with the RAED (redundant array of expensive datacenters) model instead of the traditional dual PSU, dual PDU, dual UPS, dual generator with

          • Re:what about having people onsite? (Score:4, Insightful)

            by vakuona ( 788200 ) on Monday March 08, 2010 @07:37PM (#31407618)
            Cheap doesn't mean not properly designed! Google doesn't do redundancy on a micro scale. For them it's pointless. In fact, from what I know, Google knows their hardware will fail, so they have written their software to handle hardware failures gracefully. When something like this happens, they write a report, and get someone about to work out a fix so that the outage doesn't recur.

            • Re: (Score:2)

              by afidel ( 530433 )
              Yes, but from the description given in the report Google Apps Engine works much more closely to a traditional application stack with manual datacenter failover and asynchronous data replication (with slower sync data replication available real soon now). It's not the stateless application that typifies Googles larger datacenter experience and calls for a much more traditional HA setup to give better than 3-4 9's of uptime.

      • Re: (Score:3, Insightful)

        by hedwards ( 940851 )
        My parents once lost power for several hours because a crow got fried in one of the transformers down the street. People around here lose power from time to time when a tree falls on a line. Unplanned power outages are going to happen. Even though line reliability is probably higher now than at any time in the past, it still happens and companies like Google that rely upon it being always there should have plans.

        This isn't just about keeping the people that use Google services informed, this is an admiss

        • Re: (Score:1, Flamebait)

          by dave562 ( 969951 )

          This just goes to show that Google is as "incompetent" as anyone else. There was a discussion on here the other day and a poster asked why Microsoft, with all of their resources, hasn't come up with a secure OS yet. It was suggested that the know how to create such an OS is out there, and it would just take money and will on Microsoft's part. This seems like the Google equivalent.

          Google is trying to push Apps as a replacement for Exchange and Office. They are trying to push it as a replacement for hosti

    • Useless for large scale problems (Score:5, Interesting)

      by mcrbids ( 148650 ) on Monday March 08, 2010 @01:12PM (#31402172) Journal

      Of COURSE there are people onsite. Most likely they have anywhere from a dozen to a hundred people onsite. But what's that going to do for you in the case of a large-scale problem?

      The otherwise top rated 365 Main [365main.com] facility in San Francisco went down a few years ago. They had all the shizz, multipoint redundant power, multiple data feeds, earthquake-resistant building, the works. Yet, their equipment wasn't well equipped to handle what actually took them down - a recurring brown-out. It confused their equipment, which failed to "see" the situation as one requiring emergency power, causing the whole building to go dark.

      So there you are, with perhaps 25 staff a 4-story building with tens of thousands of servers, the power is out, nobody can figure out why, and the phone lines are so loaded it's worthless. Even when the power comes back on, it's not like you are going to get "hot hands" in anything less than a week!

      Hey, even with all the best planning, disasters like this DO happen! I had to spend 2 wracking days driving to S.F. (several hours drive) to witness a disaster zone. HUNDREDS of techs just like myself carefully nursing their servers back to health, running disk checks, talking in tense tones on cell phones, etc.

      But what pissed me off (and why I don't host with them anymore) was the overly terse statement that was obviously carefully reviewed to make it damned hard to sue them. Was I ever going to sue them? Probably not, maybe just ask for a break on that month's hosting or something. I mean, I just want the damned stuff to work, and I appreciate that even in the best of situations, things *can* go wrong.

      So now I host with Herakles data center [slashdot.org] which is just as nice as the S.F. facility, except that it's closer, and it's even noticably cheaper. Redundant power, redundant network feeds, just like 365 main. (Better: they had redundancy all the way into my cage, 365 Main just had redundancy to the cage's main power feed)

      And, after a year or two of hosting with Herakles, they had a "brown-out" situation, where one of their main Cisco routers went partially dark, working well enough that their redundant router didn't kick in right away, leaving some routes up and others down while they tried to figure out what was going on.

      When all was said and done, they simply sent out a statement of "Here's what happened, it violates some of your TOS agreements, and here's a claim form". It was so nice, and so open, that out of sheer goodwill, I didn't bother to fill out a claim form, and can't praise them highly enough!

      • Re:Useless for large scale problems (Score:5, Insightful)

        by Critical Facilities ( 850111 ) * on Monday March 08, 2010 @02:24PM (#31403100)

        The otherwise top rated 365 Main [365main.com] facility in San Francisco went down a few years ago. They had all the shizz, multipoint redundant power, multiple data feeds, earthquake-resistant building, the works. Yet, their equipment wasn't well equipped to handle what actually took them down - a recurring brown-out. It confused their equipment, which failed to "see" the situation as one requiring emergency power, causing the whole building to go dark.

        I think you made the right decision in changing providers. I remember that story about the 365 outage, and while I am too lazy to look up the details again, I recall it being as you're telling it. To that end, I'd simply say that they most certainly did have the proper equipment to handle the brown out, but obviously not the proper management. If you're having regular (if intermittent) power problems (brown outs, phase imbalances, voltage harmonic anomolies, spikes, etc), just roll to generator, that's what they're there for.

        I'm sick of people making the assumption that the operators of the facility were just at the mercy of a power quality issue because they have redundant power feeds and automatic transfer switches. Yes, in a perfect world, all the PLCs will function as designed, and the critical load will stay online by itself. However, it takes some foresight and some common sense sometimes to make a decision to mitigate where necessary. I direct all my guys to pre-emptively transfer to our generators if there are frequent irregularities on both of our power feeds (i.e. during a violent thunderstorm, simultaneous utility problems, etc).

        In other words, I'm agreeing with you that the service you received was unacceptable. Along with that (and in rebuttal to the parent post), I'm saying that it's not enough to talk about how they came back from the dead, but why they got there in the first place.

        • Re: (Score:2)

          by Ocker3 ( 1232550 )
          And yet another lesson in customer service, whether tech related or not. Own up to the problem early, apologise, explain what happened, how you fixed it, and how you're going to prevent it from happening again. Any half-way intelligent business customer knows that shite happens, no backup plan is fail proof, what you Really want besides five 9s is a hosting company who's going to be up front and honest. Information is power, sharing information increases that power, it doesn't reduce it, so having your cus

      • But what pissed me off (and why I don't host with them anymore) was the overly terse statement that was obviously carefully reviewed to make it damned hard to sue them. Was I ever going to sue them? Probably not, maybe just ask for a break on that month's hosting or something.

        You wouldn't but come on, you know how we Americans are. We sue when we can't play Halo for a few days [gamespot.com].

        Chances aren't bad that someone was looking for a lawsuit, heading it off at the pass had a chance to prevent some stupid lawsuits which would waste time and only benefit lawyers, possibly requiring some invasive, poorly thought-out court-ordered hinderance which would have slowed the recovery.

      • Of COURSE there are people onsite. Most likely they have anywhere from a dozen to a hundred people onsite.

        and as long as you're quiet and don't try to damage the control systems, you can move about freely and they'll generally ignore you [wordpress.com]

  • My lifestream was interrupted and I didn't even notice! (see http://tech.slashdot.org/story/10/03/08/0024205/Time-To-Take-the-Internet-Seriously [slashdot.org] for reference)

  • Read the comments (Score:5, Insightful)

    by RaigetheFury ( 1000827 ) on Monday March 08, 2010 @12:22PM (#31401574)

    I pity EvilMuppet. Guy is a tool. There are contractual agreements that are in place to prevent pictures, aka the "rules" but when the data center blatantly LIES they are breaking the trust and violating the agreement. Case Law exists where contracts can be violated when one accuses the other of violating said contract.

    That's what happened. The data center was lying about what happened to avoid responsibility for the equipment it was being paid to host. Pictures were taken and are being used to prove the company did violate the trust of the contract.

    You can argue the semantics and legality of it but if this goes to court the pictures will be admissible and the data center will lose.

    • Comment removed based on user account deletion

    • Looking over the contract we have with Datacom, you'd be hard pressed to have the Managing Director's statements be material in affecting a contract violation. Given that the photos were taken well before any statement was made to the public by a Datacom representative takes at least some of the basis away from your argument of trust.

      As for evidence, colleagues of mine have damaged equipment and I have remote monitoring, MRTG graphs and other means of validating facts. How do you think that a particular pub

  • title should read "Google App Engine NOT a Cloud" (Score:1, Funny)

    by Anonymous Coward

    Obviously if the power goes out, and the service goes offline, then it WASN'T a cloud. If it's a cloud, it can't go down. If it goes down, it wasn't a cloud.

    What's there to get?

    • Re: (Score:1, Insightful)

      by Anonymous Coward
      Even a cloud isn't effective if all the nodes go down, it's not magic.

      • Re: (Score:1, Funny)

        by Anonymous Coward

        Whoosh.

    • Re: (Score:2)

      by Davorama ( 11731 )

      Sounds more like fog to me.

    • Obviously if the power goes out, and the service goes offline, then it WASN'T a cloud. If it's a cloud, it can't go down. If it goes down, it wasn't a cloud.

      The cloud got too big and it rained.

  • They had a perfect contingency plan for this case (Score:5, Funny)

    by juanjux ( 125739 ) on Monday March 08, 2010 @12:28PM (#31401636) Homepage

    ...but it was stored on Google Docs.

  • Significantly higher latency? (Score:3, Interesting)

    by nacturation ( 646836 ) * <nacturation AT gmail DOT com> on Monday March 08, 2010 @12:33PM (#31401700) Journal

    A new option for higher availability using synchronous replication for reads and writes, at the cost of significantly higher latency

    Anyone know some numbers around what "significantly higher latency" means? The current performance [google.com] looks to be about 200ms on average. Assuming this higher availability model doesn't commit a DB transaction until it's written to two separate datacenters, is this around 300 - 400ms for each put to the datastore?

    • Anyone know some numbers around what "significantly higher latency" means?

      I suspect not, since the feature hasn't been implemented yet.

  • App Engine down again? (Score:3, Insightful)

    by bjourne ( 1034822 ) on Monday March 08, 2010 @12:40PM (#31401768) Homepage Journal

    App Engine must be Googles absolutely most poorly run project. It has been suffering from outages almost weekly (the status page [google.com] doesn't tell the whole truth unfortunately), unexplainable performance degradations, data corruption (!!!), stale indexes and random weirdness for as long as it has been run. I am one of those who tried for a really long time to make it work, but had to give up despite it being Google and despite all the really cool technology in it. I pity the fool who pays money for that.

    The engineers who work with it are really helpful and approachable both on mailing lists and irc, and the documentation is excellent. But it doesn't help when the infrastructure around it is so flaky.

  • ISO9001 (Score:1, Insightful)

    by Anonymous Coward

    This should be standard practice... It's like the good bits of ISO9001 with a bit more openness. When done right, ISO9001 is a good model to follow.

  • the worst nightmare of data center peeps (Score:4, Interesting)

    by filesiteguy ( 695431 ) <perfectreign@gmail.com> on Monday March 08, 2010 @12:44PM (#31401820)
    i don't run a data center, but manage systems that rely on the data center 18 hrs/day 6 days/week. we pass upwards of $300m through my systems. I've yet to get a satisfactory answer as to exactly what would happen if - say - a water line breaks and floods all the electrical (including the dual redundant UPS systems) in the data center.

    • Re: (Score:3, Informative)

      by SmilingBoy ( 686281 )
      First, your servers will shutdown ungracefully, and then, they will be destroyed with little chance of recovery. You will then have to rebuild your systems, and restore the data from the offsite backup. This will of course take time. If this is too much off a risk, you should run a alternate datacentre mirroring your primary databases that can go live within minutes.
    • switch to the alternate DC - I worked for BT and the set up an alternate DC across town for Telecom Gold just in case the thames flooded

      • Re: (Score:2)

        by afidel ( 530433 )
        *across town*!? Hmm, here in the states best practice (and legal requirements for certain industries) requires significantly more distance than that between DC's. Ours is just inside of reasonable driving range (6 hours) but is on a different power grid, different core services from our Tier-1 ISP, etc.

        • Across town could be 20 miles away in London. On the other side of the Thames is very likely to have it's power and data coming from completely independent systems, even a different power station and over a different part of the national grid.

          Since BT was historically the only telecoms provider, even now they are plenty big enough to easily be in a position to have multiple independent data feeds, and if they all fail, nothing else in the capital is working anyway, so a DC's survival would be a minor issue

    • Well, I’m no expert, but it’s not very hard to get a building water tight, now is it?

      • floods (Score:2, Insightful)

        by zogger ( 617870 )

        Did you ever actually see a big flood? Freaking awesome power, like a fleet of bulldozers. Smashes stuff, rips houses off foundations, knocks huge trees over, will tumble multiple ton boulders ahead of it, etc. Just depends on how big the flood is. We had one late last year here, six inches of rain in a couple of hours, just tore stuff up all over. The "building" that can withstand a flood of significant size exists, it is called a submarine. Most buildings of the normal kind just aren't designed to deal wi

        • Re: (Score:1)

          by Ant P. ( 974313 )

          The structure that can withstand a flood has existed for a lot longer than submersible warships - it's called a "hill". If you don't have one conveniently nearby to use you can even build an artificial one.

          • Re: (Score:1)

            by zogger ( 617870 )

            A hill isn't a building. He was talking about water proofing a building. Under normal conditions, sure, buildings are pretty good to keep you from the weather, but in big floods, most will suffer leakage or outright destruction. That's why you always see people trying to save their homes or businesses with sand bags. It just isn't that common for buildings to be built bad flood tough. Some probably exist, but not too many. And yep, a good building on top of the biggest hill around would be the safest. I was

          • Re: (Score:3, Informative)

            by DragonWriter ( 970822 )

            The structure that can withstand a flood has existed for a lot longer than submersible warships - it's called a "hill". If you don't have one conveniently nearby to use you can even build an artificial one.

            An "artificial hill" intended to protect an area from floods is usually called a "levee", and while certainly extremely useful for their intended purpose, they aren't exactly an ironclad guarantee. So having contingency plans for the case where they fail isn't a bad idea.

            • The structure that can withstand a flood has existed for a lot longer than submersible warships - it's called a "hill". If you don't have one conveniently nearby to use you can even build an artificial one.

              An "artificial hill" intended to protect an area from floods is usually called a "levee", and while certainly extremely useful for their intended purpose, they aren't exactly an ironclad guarantee. So having contingency plans for the case where they fail isn't a bad idea.

              Buildings that are built _on top_ of a hill (even an artificial one), don't have quite the same set of severe problems with flooding that occurs in low-lying areas. ;)

    • Re: (Score:2)

      by Eil ( 82413 )

      I've yet to get a satisfactory answer as to exactly what would happen if - say - a water line breaks and floods all the electrical (including the dual redundant UPS systems) in the data center.

      Simple: the power equipment gets an unscheduled watering and your servers go down.

      If you want to minimize the impact that a disaster can wreak on your servers in a datacenter, then you need to have your entire setup running and synchronously replicated in another datacenter.

      • Funny you mention that. I've been trying to get two solutions going. (Remember, I have zero actual power over server budgets other than recommendations.)

        I have setup all servers under my responsiblity in VM's (using VirtualBox) and am ready to deploy on a minimum of servers with only databases available. (I have roughly 3 TB of data and about 22 TB of images.)

        I've been patiently standing by, waiting for a data center agreement to be formalized, whereby we'll have a hot-site setup in a center about twenty mi
  • OMFG! There's swinging at an outside pitch and there's try to hit one that was thrown in the fuckin' stands!!

  • Generators plus UPS FTMFW (Score:2, Insightful)

    by Anonymous Coward

    Epic fail.

    Any data center worth it's weight in dirt, must have UPS devices sufficient to power all servers plus all network and infrastructure equipment, as well as the HVAC systems too, for a minimum of at least 2 full hours on batteries, in case the backup generators have difficulty in getting started up and online.

    Any data center without both adequate battery-UPS systems plus diesel (or natural gas or propane powered) generators is a rinky-dink, mickey-mouse amateur operation.

    • Yeah, seriously. I worked for a mid-size company that had a very modest server farm (it was a retail-related business), and even we had everything switch to diesel at the instant the grid might go down. Since our switches were POE, and our phone were VOIP, and our computers were laptops, it was like there was no power outage at all. We'd be on the phone with one of our stores and just say 'oh, the power went out, well, back to your issue...'

      It's hard to believe that freakin' Google wouldn't be at that lev
    • quite thers a comment somwhere else about how 356 main was highly regarded lol - if everything insn't running of the batteries 24/7 it aint a real datacentre.

    • Re:Generators plus UPS FTMFW (Score:4, Insightful)

      by Tynin ( 634655 ) on Monday March 08, 2010 @01:46PM (#31402656)
      You are so cute. I know very little about UPS systems, but when I was working in a datacenter that housed 5000 servers we had a two story room that was twice the size of most houses (~2000 sq ft) with rows and rows of batteries. I was told that in the event of a power outage, we had 22 minutes of battery power before everything went out. The idea of having enough for 2 hours would have been one an interesting setup considering how monstrously large this one already was. Besides, I'm unsure why you'd ever need more than that 22min since that is plenty of time for our on site staff to gracefully power down any of our major servers if the backup generator failed to kick in.

    • Re: (Score:3, Funny)

      by Darth_brooks ( 180756 )

      Yeah, and when the guys at the Jesus Christ of Datcenters that you describe have to do something like, say, switch from generator to utility power manually, and the document that details that process is 18 months old and refers to electrical panels that don't exist anymore, you get what you had here. A failure of fail-over procedures. If the lowliest help desk / operator can't at least understand the documentation you've written, then you've failed.

      The only equipment failure listed is a "power failure." Gra

    • Any data center worth it's weight in dirt, must have UPS devices sufficient to power all servers plus all network and infrastructure equipment, as well as the HVAC systems too, for a minimum of at least 2 full hours on batteries, in case the backup generators have difficulty in getting started up and online.

      Google's setup appears to rely on the fact that they have redundant data centers, so failover to another data center addresses this problem. The problem here, as identified in their post-mortem, is that

  • When the Power Goes Out At Google... (Score:3, Informative)

    by binaryseraph ( 955557 ) on Monday March 08, 2010 @01:04PM (#31402066)
    ...a fairy dies.

    • ...a fairy dies.

      I suspect that this will result in a large overpopulation of fairies. Since Google would be to blame for this, perhaps they should begin some sort of fairy mitigation program?

  • try hiring some staff with telco experiance instead of kids with a perfect GPA scores from stanford and design the fraking thing better !
  • I think it would do them good, considering the recent downtime with Assassin's Creed 2. Has anyone seen any info on that outage?

  • Lessons Like (Score:3, Funny)

    by Greyfox ( 87712 ) on Monday March 08, 2010 @02:26PM (#31403138) Homepage Journal
    Don't have all your shit in one data center, maybe? I'd have thought that one would be pretty fundamental. Of course, knowing Google they're going to decide that what they really need is power generation right on site, then they'll just pop off and invent nuclear fusion before lunch.
  • We decided to move three of our divisions into one facility, those included to business facing units and the I.T. division.

    I was charged with laying out the design for data, telecom and electrical for the project. Also had engineering of our little NOC.

    Nice setup - redundant power in the I.T. division, nice big APC UPS for the entire room, had it's own 480V power drop, dual HVAC units, a natural gas fired generator. It's nice to have the money to do this.

    Since we were a state agency we had to use s

    • Re:Back around 2005... (Score:4, Insightful)

      by Richard_at_work ( 517087 ) on Monday March 08, 2010 @05:48PM (#31405900)
      Let me add my own little story, which happened back in the good old days of June 2009.

      The company had spent the past year rearchitecting the entire IT infrastructure, as the complete core application suite for the business was, other than your standard peripheral utilities like Office et al, green screen based, using a proprietary language from the early 1980s that was barely still maintained and wasn't going anywhere fast.

      It was my job to handle the systems infrastructure side of the deal, while another team handled software development and I was way ahead of them - the core business applications were still in the planning stages while the infrastructure to handle and host them was well advanced. The platform we chose was well designed, with onsite redundancy built into the base cost and easily scalable - dare I say it myself, it was a good job. The only thing I had no hand in on the hardware side was the actual building infrastructure, as we had moved to custom built offices about 5 years prior, and there was someone else on the team that handled telecoms and the building. But we had a UPS and a generator, so all seemed well in the world.

      Alongside the new infrastructure came the new business continuity plan. Well, I say 'new' - I can't really say there was an 'old' BCP. Sure, we rented space at a major BC facilities provider, but there had never been any test, and there wasn't even any written documentation as to what to do.

      Here is where I must admit my first failure - the BCP was not treated as an integral, tied-in-like-a-knot part of the infrastructure, it was a separate project running alongside. Sure, the new infrastructure was designed to take a local server failure through redundancy, or even allow ease of moving to an offsite location. That part of it was all in place. My failure was in not ensuring that the offsite location actually existed as the new infrastructure grew.

      However, by the start of 2009, the basic infrastructure needs of the BCP were well known, costed and presented to the company board of directors. And there it sat. Every month I would ask them if it had been signed off, if I could spend the money. Every month I received a negative answer, it just hadn't been discussed at these busy directors meetings.

      And that was my second failure. I had no sponsor in those meetings, there was basically no IT representation (the IT director had resigned after the modernisation was pushed through, he wanted no part in it as he had not been taking the business forward himself). With no sponsor, no one wanted to raise the potential spending of a hundred thousand pounds themselves. And so it sat.

      Then one day in June, we had a routine fan replacement on the UPS. The engineer was signed in, did the replacement under the watchful eye of a senior helpdesk technician, and flipped the UPS back from maintenance bypass to full protected mains. And that was when the first bang happened.

      And all the lights went dark. All the whirring stopped. All the phones stopped ringing. All the people stopped talking.

      It was blissfully quiet for a few precious seconds. And then it was painfully quiet for about another 5. And then all hell broke loose.

      The core business applications did not fair well. The 30 year old architecture essentially had no failsafe for database writes, and as the server had quit in the midst of several thousand writes, we knew we had just lost a significant amount of data.

      Its worth taking several seconds out to explain how the core application language does its job. Firstly, there is no database server, its all C-ISAM datafiles directly read from and written to by each individual application. Locks are handled by each application internally, with OS level locking preventing concurrent writes to the same record in the data file. No database engine, no transaction logging, no roll backs, no error correction, nothing. There was nothing in the language to protect those poor l
      • Yes we also had a failover site for our Central Voter Registration System. Never tested of course because nobody to be the one whose head would roll.

        Apparently it did work though. When we had that DNS fail we were able to see that the hot standby site came up without a hiccup.

        But you make a good point, unless you have someone high up that's going to shepherd your project through, you'd be better be prepared for some ugly times.

        Luckily we had full buy-in on ours. Another thing happened though. I.T.
  • I read the post-mortem and I think they completely missed the mark. Power failed to some machines. They only noticed because "...traffic has problems..." They should have been monitoring the power to detect this situation. They didn't say whether they have the data center power supply on a UPS or not. If it was, it was dying and no one noticed. If they had been monitoring the power they might have avoided the whole mess.

  • Repeat to yourself: "All is well, All is well, All is well" and everything will be exactly like you wish it to be.

    Note originators of response model are not responsible for anyone being taken away to a psychiatric facility because of a belief response model user is psychotic

Slashdot Top Deals

For God's sake, stop researching for a while and begin to think!

Close