Multiple Sites Down In SF Power Outage 423
corewtfux writes with word of a major outage apparently centered on 365 Main, a datacenter on the edge of San Francisco's Financial District. Valleywag initially claimed that a drunken person had gotten in and damaged 40 racks, but an update from Technorati's Dave Sifry says the problem is a widespread power outage. Sites affected include Technorati, Netflix (these display nice "We're Dead" pages), Typepad, LiveJournal, Sun.com, and Craigslist (these just time out).
I work in the Financial District (Score:5, Interesting)
Re:No Generators? (Score:2, Interesting)
Redundent power supply? (Score:3, Interesting)
how many data centers? (Score:4, Interesting)
Re:Redundent power supply? (Score:5, Interesting)
They have the HiTec rotary UPSs in all their facilities, which link a generator to a flywheel UPS. It's stupid to not have backup fuel for that type of system; you can only run for 13 seconds before the load crashes.
It is possible that they got a number of small hits and the generators failed to re-start after a few. Good procedures are to stay on generator until utility stabilizes if you have more than one "hit."
Be interesting to find out what happened.
July 24th: RedEnvelope Press Release by 365 Main (Score:4, Interesting)
It was released today....
Re:No Generators? (Score:5, Interesting)
No kidding. years ago in my former job on traffic systems we had a great UPS with a generator on site and the ability keep it fueled up indefinitely. A security contractor came in on the weekend to install something and tried to wire up a new circuit hot. He slipped with a screwdriver and shorted the white phase to the chasis of the breaker panel. I don't think the tip of the driver actually touched ground, but the burn mark is still there to show how close he got.
The resuting current spike blew the 100A fuses (heavy metal strips) both going in to and out of the UPS. With the UPS effectively broken the generator set failed to start and the system gracefully shut down 40 minutes after the incident. Thats not bad. The batteries were only specified to work long enough for the genny to settle at 50Hz.
In the process of blowing the fuses a spike got back into the power supply of one of our DEC Alphas and took out the power supply. The system was redundant at the software level so I didn't notice immediately.
The UPS guy came out and didn't have enough fuses to replace the blown one, but we found that with a bit of brute force and filing attacks some others could be made to fit.
Please type the word in this image: problems
UPS system - it's a Hytec flywheel/diesel combo (Score:4, Interesting)
Data sheet for 365 Main [365main.com]:
The company's San Francisco facility includes two complete back-up systems for electrical power to protect against a power loss. In the unlikely event of a cut to a primary power feed, the state-of-the-art electrical system instantly switches to live back-up generators, avoiding costly downtime for tenants and keeping the data center continuously running.
They use a Hytec Continuous Power System [pageprocessor.nl], which is a motor, generator, flywheel, clutch, and Diesel engine all on the same shaft. They don't use batteries.
With this type of equipment, if for some reason you lose power and the generator doesn't start before the flywheel runs down, you're dead. There's no way to start the thing without external power. Unless you buy the optional Black Start feature [pageprocessor.nl], which has an extra battery pack for starting the Diesel. "Usually the black start facility will not be often needed but it won't hurt to consider installing one. Just imagine if you were unable to start up your UPS system because the mains supply is not available.". Did 365 Main buy that option?
Re:Redundant? (Score:5, Interesting)
Sun.com going down is a good example of someone totally screwing up. They have absolutely NO excuse. The others - maybe they can get away with it and we won't care. If Sun can't keep their own site up, how can I expect them to keep mine up?
Not that uncommon (Score:3, Interesting)
A client of mine had a number of servers in a Sterling, Virginia data center managed by Verio/NTT. It's a good data center and seems to be well-run.
Last September, the data center experienced two complete power failures in the span of three days. To their immense credit, data center management was straight with customers about what had happened. For those who might be interested, their statements about the problem appear here. [dedicatedserver.com]
My point? Make sure you know how to bring your systems back up from a completely cold start, and that you find a way to test this periodically. While we work to ensure that this sort of situation occurs rarely, the fact remains that these sorts of failures DO occur, and they're not as uncommon as the sales and marketing folks would like you to believe.
Phil
Insane level of backup... (Score:5, Interesting)
The only places I've actually seen the insane levels of backup that some would like is in some telco central offices. The one I was associated with the longest had eight-hour-plus battery backup and 8 days of fuel for the diesels. Some of our really remote microwave sites had 24 hour battery and 30 day diesel.
Of course one of those sites failed high up in a mountain range in a mid-winter storm (Tieton, 1978) when the commercial power failed, and the starter battery for the diesel froze. When one of the techs finally got there (after burying his Sno-Cat and walking the last couple miles), he had to chip ice off the steel door to get inside, where he was able to get the diesel started with a little "rewire" of one of the backup battery sets. Oh, his two-way radio also failed during his hike, since it was outside his snowsuit, and the lack of communication caused the company to start two more Sno-Cats and a helicopter in that direction.
The site was out for nearly six hours, IIRC.
Even the BEST designs are subject to failure.
--
Tomas
Re:UPS system - it's a Hytec flywheel/diesel combo (Score:5, Interesting)
And this was in addition to the 48VDC battery backup.
In the entire history of electromechanical switching in the Bell System, no central office was ever down for more than 30 minutes for any reason other than a natural disaster. That record has not been maintained in the computer era.
If you have to build reliable systems, it's worth understanding electromechanical telephone switching. Because the components weren't that reliable, the systems had to be engineered so that the system as a whole was far more reliable than the components. Read up on Number Five Crossbar. [wikipedia.org] The Wikipedia article isn't really enough to understand the architecture, but other references are available.
Re:Insane level of backup... (Score:3, Interesting)
--
Tomas
Re:SAN? Huh? (Score:4, Interesting)
HP has a nice overview of building systems which can failover between widely distributed nodes called Designing Disaster Tolerant High Availability Clusters [hp.com]. It's a bit old, and is focused on ServiceGuard [hp.com], but is still interesting.
Re:Redundent power supply? (Score:1, Interesting)
have to monitor (it always looks fine), it is the deliverable
current over a period of time. And that is something that can
be accurately tested only by loading. That is why a battery
discharge/recharge cycle is common in advanced UPS solutions.
But you are correct about the monthly/weekly tests, which
should tell you something.
Re:Power back but not Craigslist (Score:3, Interesting)
Re:Redundant? (Score:2, Interesting)
Speaking of energy costs associated with heat dissipation, I've alway been curious of a method that could produce energy from wasted heat- as does a solar panel from the sun.
Wrap that supercharged V8 in some energy producing heatwrap, instant hybrid and more horsepower. Run your processor's cooling fan off the energy produced from the excessive heat.
Someone please tell me I'm talking out of my ass, or worse, just gave away the next big idea that could have made me billions.
Millions were paged, and cried out in despair (Score:3, Interesting)
http://tastic.brillig.org/~jwb/dorks.jpg [brillig.org]