When the Power Goes Out At Google 135
1sockchuck writes "What happens when the power goes out in one of Google's mighty data centers? The company has issued an incident report on a Feb. 24 outage for Google App Engine, which went offline when an entire data center lost power. The post-mortem outlines what went wrong and why, lessons learned and steps taken, which include additional training and documentation for staff and new datastore configurations for App Engine. Google is earning strong reviews for its openness, which is being hailed as an excellent model for industry outage reports. At the other end of the spectrum is Australian host Datacom, where executives are denying that a Melbourne data center experienced water damage during weekend flooding, forcing tech media to document the outage via photos, user stories and emails from the NOC."
Re:Don't they have (Score:5, Informative)
Huh? (Score:3, Informative)
When the Power Goes Out At Google... (Score:3, Informative)
Re:the worst nightmare of data center peeps (Score:3, Informative)
Re:floods (Score:3, Informative)
An "artificial hill" intended to protect an area from floods is usually called a "levee", and while certainly extremely useful for their intended purpose, they aren't exactly an ironclad guarantee. So having contingency plans for the case where they fail isn't a bad idea.
Re:Generators plus UPS FTMFW (Score:3, Informative)
You cannot have uptime without power. A mains outage coupled with an unexpected generator failure *will* result in downtime - your decision now is whether you wish your servers to be gracefully shutdown, or just have the rug pulled from under them and hours or days of potential angst as a result. Which is it?
And before you suggest larger UPSes for longer protection, consider why you have both a generator and a UPS in the first place - UPSes cost a lot, they cost a lot to buy, and they cost a lot to maintain, and then they cost a lot to replace after only a few years. A generator in comparison costs a lot less all round.