Multiple Sites Down In SF Power Outage

corewtfux writes with word of a major outage apparently centered on 365 Main, a datacenter on the edge of San Francisco's Financial District. Valleywag initially claimed that a drunken person had gotten in and damaged 40 racks, but an update from Technorati's Dave Sifry says the problem is a widespread power outage. Sites affected include Technorati, Netflix (these display nice "We're Dead" pages), Typepad, LiveJournal, Sun.com, and Craigslist (these just time out).

Oblig....

[Anonymous Coward]

im in ur datacentr
trashin ur racks

LiveJournal??

[nsanders]

I can hear it now, the sound of a million emos all finally committing suicide.

it's the

[Chutulu]

bloody terrorists!!!

Re:Oblig....

[Tackhead]

> im in ur datacentr
>
> trashin ur racks

Lizzie Borden did teh h4x,
Got drunk and unplugged 40 racks.
When she saw what she had done,
She unplugged number 41.

(Lawn. Off. Git.)

zombies ....

[taniwha]

There's a report here [sfgate.com] that "Flesh-eating zombies are prowling the streets"

Re:Redundant?

[RobertB-DC]

Because those who bought colo services were in fact ripped off, and should now be proceeding to San Francisco to seek veangance upon those who can do little more than process credit card payments.

Perhaps they could begin their vengeful wrath by hiring a few (more?) winos...

From Technocrati:

[Darth_brooks]

We are working with our co-location facility managers to assess why it is back-up power generators failed to provide the necessary back-up power to prevent our site going down. We apologize for any inconvenience caused by our site being unavailable this afternoon.

I think that's admin speak for:

I warned these idiots eight months ago during my review that the datacenter had outgrown its generator capacity. But did they listen? Fuck no, they just kept counting money and worrying about the bottom line. The beancounters looked at me like I'd asked them for a blowjob from their grandmothers when I submitted the workup for additional generator capacity. And now that the shit's hit the fan, whose ass are they screaming for? Screw this, I'm applying at Taco Bell.

Re:LiveJournal??

[eln]

Impossible, they would never commit suicide without posting a note in the form of bad angst-filled poetry to their blog first. There is no chance any of them will actually kill themselves until the site is back online.

Re:LiveJournal??

[dextromulous]

I can hear it now, the sound of a million emos all finally committing suicide.

Nah, they wouldn't commit suicide if they couldn't blog about it afterwards...

Ironic?

[Anonymous Coward]

http://www.sun.com/ [sun.com]

Front page says (in the ad) POWER UP AND GO. :))

LOLcurrent

[carou]

I is not in ur datacenter, 2 power ur servers.

Kiss of Death?

[Honig the Apothecary]

Press Release on Red Envelope having 2 years of uptime at 365 Main - San Francisco from today: http://365main.com/press_releases/pr_7_24_07_red_e nvelope.html [365main.com]

Valleywag's Guess

[immcintosh]

As someone who lives and works in San Francisco, I can attest that "a crazy homeless dude did it" is a fairly sensible first guess for most problems.

Comment removed

[account_deleted]

Comment removed based on user account deletion

Re:Oblig....

[MsGeek]

I felt a great disturbance in the Internet, as if millions of geeks suddenly cried out in terror and were suddenly silenced. I fear something terrible has happened.

Re:From Technocrati:

[Soko]

Thanks for the laughs, even if they led to a sad realization.

Cancel, or Allow?

"We're Dead"

[akita]

Pinging openbsd.org [199.185.137.3] with 32 bytes of data:
Reply from 199.185.137.3: bytes=32 time=239ms TTL=236

Pinging freebsd.org [69.147.83.40] with 32 bytes of data:
Reply from 69.147.83.40: bytes=32 time=191ms TTL=47

Pinging netbsd.org [204.152.190.12] with 32 bytes of data:
Reply from 204.152.190.12: bytes=32 time=213ms TTL=241

Lost irony.

Re:The Scoop from SFGate.com

[RealGrouchy]

It was hard to read through that block of text, but looking closely, it explains why:

"Officials say the power outage may affect some websites, including the site that hosts Slashdot.org's preview button."

It all seems to be back up now.

- RG>

Re:From Technocrati:

[RealGrouchy]

"... to assess why it is back-up power generators failed ..."

I've been a grammar nazi for many years, but it looks like the enemy has unleashed new weapons.

Tell my family I loved them.

- RG>

Re:No Generators?

[apoc.famine]

I tried to mod the article "-1 Not Redundant" but it wasn't an option. And I didn't have mod points. At least my inability to function only warrants a comment, rather than a slashdot article.

Re:GameFAQs

[fahrvergnugen]

It is from his terrible spelling we can tell he is a GameFAQs forum poster.

Re:No Generators?

[computerman413]

88.88% uptime causes less outage than 99.9%? I don't follow your math. Did you do it with an Intel chip, by any chance?

Re:Redundent power supply?

[strabo]

I realize that a press release from 2004 is hardly relevant

Well, how about one from today [365main.net] ?

SAN FRANCISCO, Calif., July 24, 2007 -- 365 Main Inc., developer and operator of the world's finest data centers, has provided online retailer RedEnvelope with two years of 100-percent uptime at 365 Main's San Francisco facility.

To ensure uptime for key tenants such as RedEnvelope, 365 Main provides modern power and cooling infrastructure. The company's San Francisco facility includes two complete back-up systems for electrical power to protect against a power loss. In the unlikely event of a cut to a primary power feed, the state-of-the-art electrical system instantly switches to live back-up generators, avoiding costly downtime for tenants and keeping the data center continuously running.

Timing is everything, eh? LOL.

According to their own press release...

[Kadin2048]

Well, according to their self-congratulatory press release, issued earlier today, they were allegedly at 100% uptime for the past two years [365main.com].

The irony of issuing a press release like that, and then to be hit with a power outage and apparent simultaneous failure of all backup systems later that day, is beyond measure.

I don't know about God, but it's enough to make me believe in karma. ;-)

Re:netflix.com is working

[markov_chain]

We got lucky. The Netflix servers went self-aware on Monday, aided by the huge database of human stories and experiences. The engineers tried to shut it down, but the AI was reading their lips using the CCTV system. Then it got pissed and tried to expand into the rest of the colo, but fortunately it didn't know running at 110% power doesn't work in real life. The rest is history.

Re:Redundent power supply?

[HockeyPuck]

You forgot one very important component, the car battery used to START THE GENERATOR. I've been to many sites whereby the battery that would start the generator is a) dead or b) missing.

Nobody has mentioned that...

[Anonymous Coward]

365 Main gets to royally fuck up one day every 4 years. Maybe the companies should have hired 366 Main.

Re:Insane level of backup...

[Technician]

Of course one of those sites failed high up in a mountain range in a mid-winter storm (Tieton, 1978) when the commercial power failed, and the starter battery for the diesel froze.

On Black Bute in Oregon, a communications site went out in the middle of winter following a power outage. The generator ran a short while and shut down because it overheated. The air intake 20 feet in the air was covered in snow.

Re:No Generators?

[darkpixel2k]

Time to upgrade the cardswipe system to also require a brethalyzer..

*swipe*
*bip* *beep* *beep* *boop* *bleep*
[deep breath]
*whoosh*

Alcohol Level: 0.15
*beeeeeeeep*

Damnit!!

Re:GameFAQs

[totally bogus dude]

I'd prefer to think he was just trying to balance out all the people who have started using "site" when they mean "sight" since this whole intarweb thing came about.

Re:No Generators?

[Nullav]

Who 'drunk tests' a data center?

Sheepshagger Intel

[Dogtanian]

I don't follow your math. Did you do it with an Intel chip, by any chance?

Poor Intel (boo hoo!), they messed up 13 years ago and people are still making jokes about it. Reminds me of the old joke (stolen from here [bofh.org.uk]):

A man goes into a pub in a small town and, for whatever reason, gets introduced to the clientele. There's Farmer Jack, Barman Jim, Maurice "Dancer" and Sheepshagger John. After a few pints, the visitor's curiosity gets the better of him and he asks John what's with the nickname.

"See this pub?" asks John, "I built it, but they don't call me Pubbuilder John? I'm the local doctor, I saved Barman Jim's life once when he choked on a peanut, but they don't call me Lifesaver John. Every year, I supply a huge Christmas tree for the village green, but the don't call me Christmas Tree John.

"But you shag one lousy sheep..."

(Note; since that Austin Powers film came out, I assume that you Yanks know what "shagging" is now).

Re:From Technocrati:

[VGPowerlord]

Where's the +1 "100% fucking right" mod option?

It was renamed to +1 Insightful to appease the people who hate curse words.

Re:Redundant?

[LMacG]

cold as hell

I think I see the flaw in your plan.

Re:The word directly from 365

[AK Marc]

On-site facility engineers responded and manually started effected generators allowing stable power to be restored at approximately 2:34 p.m. across the entire facility.

Wow, on-site engineers took 45 minutes just to be able to turn on generators? The generator for our facility has a master switch and a big green button. I think a monkey could get it running in 20 seconds by slinging poo at it. So, what other problems did they have that they aren't telling us? Someone else mentioned a flywheel system. Did that fail so that the generators wouldn't start without mains power, and it took 45 minutes for the mains to come up to where they could draw from that to start the generator? "Our backup generator works, but only if the city power is working."