How Peer1 Survived Sandy 130
Nerval's Lobster writes "When hurricane Sandy knocked out the electricity in lower Manhattan, data-center operator Peer1 took extreme measures to keep its servers humming, assembling a bucket brigade that carried diesel fuel up several flights of stairs. Ted Smith, senior vice president of operations for Peer1, talks about the decisions made as the floodwaters rose and the main generators went offline, as well as the changes his company has made in the aftermath of the storm. He said, 'When the water got to a point that it had flooded the infrastructure and the basement, we were then operating under the reserves the building had on the roof, and our own storage tanks. Literally, at that point we had to do calculations as to how long we could run. And we believed we had enough diesel fuel—between what is in the building, and in our tanks, to about 9 AM the following day. ... You know the bucket brigade—it’s something I’ve never asked the team to do. If you think about what that was at that time, you’re talking about carrying fuel up 17 flights, in total darkness, throughout a whole evening. We had informed our data center manager that we were shutting down, but he kind of took on it himself to say, ‘Not on my watch.’ And he organized himself, got a temporary solution and then more customers jumped in. And at peak I think we had about 30 people helping.'"
Re:Health and safety? (Score:5, Insightful)
A bucket brigade of Diesel fuel? (Score:4, Insightful)
In total darkness, up 17 flights of stairs, with a flooded basement? Sounds like a recipe for a potentially fatal fire. People's lives are more important than a freaking data center. Sorry, but I don't see this as a heroic story about people trying to keep critical infrastructure running, but as a desperate failure that could easily have turned into a disaster. They never should have gotten to the point where they're continually carrying fuel up stairs. It also sounds like they then decided to pump fuel up a pipe they installed in the stairwell. That doesn't sound terribly safe either, especially when done in a mad rush like I'm sure it was.
Gee.. couldn't have someone planned for this contingency rather than this sort of haphazard, dangerous sounding plan that was thrown together?
Christmas Bonuses For All (Score:5, Insightful)
Re:Health and safety? (Score:5, Insightful)
Re:Christmas Bonuses For All (Score:4, Insightful)
Re:Health and safety? (Score:5, Insightful)
Re:Health and safety? (Score:5, Insightful)
Diesel fuel does not burn the way you think it does.
Re:OSHA must be thrilled (Score:5, Insightful)
OSHA must be thrilled
Getting OSHA / union / bubblewrap parents involved means that those who are capable of helping are not allowed to because of the risk that some idiot gets hurt or damages something.
They have their place and time when things are normal to try and minimize the impact of a disaster, but once that disaster is in full swing, they need to sit down, shut up and let people self-mobilize to get the job done.
In the spring of '97 guys were working heavy equipment for days straight, often by the light of military flares, to build a dike that saved Winnipeg from one of the biggest spring floods in our history [winnipeg.ca] (often "stealing" clay/dirt from nearby farms to get the dike to the heights needed, dragging and dumping scrap cars, buses, anything they could find to shore up the water front side from erosion, etc.). Ignoring the union rules, safety rules, land procurement rules, etc. they got it done in time.
After the flood waters receded, then all the compensating processes kicked in to address the shortcomings.
Was it worth it? (Score:5, Insightful)
Having your employees stay in a emergency stricken zone that is flooded and carrying open canisters of diesel fuel to keep a data center running so that someone in California can share pictures of their cat is really not worth it IMHO.
I am sure some people were probably a little more worried about the lives of their families and themselves rather then some digital data.
I am not going to call someone a hero for this. At some point out there, people using cloud services and online storage are going to have to accept the fact that during emergency situations, their data just isn't accessible, period.
The basic fundamental problem I have about all this and what Sandy has highlighted is that the Internet was designed to be decentralized solely for the purpose of surviving natural or man-made disasters. Why is it then that a data center company creates a single centralized storage site instead of having an auxiliary site somewhere else, even on the other side of the country.
I think this is an epic fail in planning and execution. Anyone using Peer1 shouldn't be happy for putting people's lives in danger when common sense could have had them build in redundancy to their infrastructure allowing people to worry about their families more then your data.
Also, just like with Japan, don't build your backup generators at or below sea level.
another one? (Score:4, Insightful)
How many of these asinine data center advertisements are we going to get? This is at least the 3rd "How such and such data center survived Sandy!" I don't care... it's not news. You told your employees to stand in knee deep water in the middle of tons of electronic equipment and bail water? You're a god damned fool and lucky no-one got killed.
Re:A bucket brigade of Diesel fuel? (Score:4, Insightful)
Diesel burns readily. Doesn't flash. But dragging it up a stairway in minimally closed containers is stupid. Drop 5 gallons in a stairway and you have a real mess even if it didn't burn. It will leak under door frames, it's slippery. Fumes are dangerous. Diesel is really, really hard to clean up after.
This was a bad idea on a number of levels, the fire risk being only one of them.
Re:OSHA must be thrilled (Score:5, Insightful)
Re:Health and safety? (Score:5, Insightful)
In a stairwell, I'd be more worried about diesel's oily slipperiness than its flammability.
Re:OSHA must be thrilled (Score:5, Insightful)
Yesterday, I squandered mod points that were going to expire. Today - I wish I had some. Screw the beauracrats, sometimes you just gotta do what's gotta be done!
Re:Health and safety? (Score:5, Insightful)
The guy should be fired for needless risk to not only the people, but the building in general.
When did being utterly devoid of courage and constantly afraid of every single thing under the sun became a virtue?
"USA, land of the restrained, home of the fearful"?
Re:Health and safety? (Score:4, Insightful)
They're lucky they're not in the middle of a lawsuit
In other words: when lawyers took over.