Forgot your password?
typodupeerror
Hardware

How Peer1 Survived Sandy 130

Posted by Soulskill
from the lots-of-inflatable-rafts dept.
Nerval's Lobster writes "When hurricane Sandy knocked out the electricity in lower Manhattan, data-center operator Peer1 took extreme measures to keep its servers humming, assembling a bucket brigade that carried diesel fuel up several flights of stairs. Ted Smith, senior vice president of operations for Peer1, talks about the decisions made as the floodwaters rose and the main generators went offline, as well as the changes his company has made in the aftermath of the storm. He said, 'When the water got to a point that it had flooded the infrastructure and the basement, we were then operating under the reserves the building had on the roof, and our own storage tanks. Literally, at that point we had to do calculations as to how long we could run. And we believed we had enough diesel fuel—between what is in the building, and in our tanks, to about 9 AM the following day. ... You know the bucket brigade—it’s something I’ve never asked the team to do. If you think about what that was at that time, you’re talking about carrying fuel up 17 flights, in total darkness, throughout a whole evening. We had informed our data center manager that we were shutting down, but he kind of took on it himself to say, ‘Not on my watch.’ And he organized himself, got a temporary solution and then more customers jumped in. And at peak I think we had about 30 people helping.'"
This discussion has been archived. No new comments can be posted.

How Peer1 Survived Sandy

Comments Filter:
  • by crakbone (860662) on Friday December 07, 2012 @12:05PM (#42215849)
    From the story it looks like they specifically did NOT ask the employee to do it. He took it upon himself to find people to do it. Including customers. I would say make sure that guy gets a raise and part of the profits that he kept running.
  • by Vellmont (569020) on Friday December 07, 2012 @12:13PM (#42215977)

    In total darkness, up 17 flights of stairs, with a flooded basement? Sounds like a recipe for a potentially fatal fire. People's lives are more important than a freaking data center. Sorry, but I don't see this as a heroic story about people trying to keep critical infrastructure running, but as a desperate failure that could easily have turned into a disaster. They never should have gotten to the point where they're continually carrying fuel up stairs. It also sounds like they then decided to pump fuel up a pipe they installed in the stairwell. That doesn't sound terribly safe either, especially when done in a mad rush like I'm sure it was.

    Gee.. couldn't have someone planned for this contingency rather than this sort of haphazard, dangerous sounding plan that was thrown together?

  • by Supp0rtLinux (594509) <Supp0rtLinux@yahoo.com> on Friday December 07, 2012 @12:16PM (#42216019)
    All I can say is you damn well better reward all your employees that helped. They kept you up and kept your revenue stream moving. You need to give them some kickass holiday bonuses or you're all major douche bags.
  • by sunking2 (521698) on Friday December 07, 2012 @12:18PM (#42216037)
    Oh please. That is about the most boneheaded thing he could have done. Stairwells full of open buckets of fuel being handed off to people. What could possible happen if a fire occurred? The guy should be fired for needless risk to not only the people, but the building in general. Just so he could keep his stupid uptime hours going after they had already notifed everyone things would be shutting down at any time.
  • by Supp0rtLinux (594509) <Supp0rtLinux@yahoo.com> on Friday December 07, 2012 @12:19PM (#42216063)
    Or if this other comment is true: "From what I hear, based on the StackExchange podcast, and the tweets that went out from SquareSpace and StackExchange during the whole idea is that Peer1 had a complete failure, and it was only due to the hard work of their customers (SE and SquareSpace) that the datacenter was able to remain operational. If your customers have to start carrying buckets of diesel up 17 flights of starirs, you, as a datacenter have failed. Peer1, left to their own devices would have just let the thing shutdown, and apparently head office wasn't aware of how bad things even were." then you better give your customers some free months of service for doing your job for you. Either way, figure out who kept it going and reward them handsomely or you suck!!!
  • by onkelonkel (560274) on Friday December 07, 2012 @12:23PM (#42216115)
    Did you seriously think they were carrying diesel in open buckets? They were almost certainly using proper fuel containers (ie gas cans). Bucket brigade is a figure of speech.
  • by Dishevel (1105119) on Friday December 07, 2012 @12:33PM (#42216271)

    Diesel fuel does not burn the way you think it does.

  • by ehud42 (314607) on Friday December 07, 2012 @12:36PM (#42216323) Homepage

    OSHA must be thrilled

    Getting OSHA / union / bubblewrap parents involved means that those who are capable of helping are not allowed to because of the risk that some idiot gets hurt or damages something.

    They have their place and time when things are normal to try and minimize the impact of a disaster, but once that disaster is in full swing, they need to sit down, shut up and let people self-mobilize to get the job done.

    In the spring of '97 guys were working heavy equipment for days straight, often by the light of military flares, to build a dike that saved Winnipeg from one of the biggest spring floods in our history [winnipeg.ca] (often "stealing" clay/dirt from nearby farms to get the dike to the heights needed, dragging and dumping scrap cars, buses, anything they could find to shore up the water front side from erosion, etc.). Ignoring the union rules, safety rules, land procurement rules, etc. they got it done in time.

    After the flood waters receded, then all the compensating processes kicked in to address the shortcomings.

  • Was it worth it? (Score:5, Insightful)

    by TheSkepticalOptimist (898384) on Friday December 07, 2012 @12:36PM (#42216329)

    Having your employees stay in a emergency stricken zone that is flooded and carrying open canisters of diesel fuel to keep a data center running so that someone in California can share pictures of their cat is really not worth it IMHO.

    I am sure some people were probably a little more worried about the lives of their families and themselves rather then some digital data.

    I am not going to call someone a hero for this. At some point out there, people using cloud services and online storage are going to have to accept the fact that during emergency situations, their data just isn't accessible, period.

    The basic fundamental problem I have about all this and what Sandy has highlighted is that the Internet was designed to be decentralized solely for the purpose of surviving natural or man-made disasters. Why is it then that a data center company creates a single centralized storage site instead of having an auxiliary site somewhere else, even on the other side of the country.

    I think this is an epic fail in planning and execution. Anyone using Peer1 shouldn't be happy for putting people's lives in danger when common sense could have had them build in redundancy to their infrastructure allowing people to worry about their families more then your data.

    Also, just like with Japan, don't build your backup generators at or below sea level.

  • another one? (Score:4, Insightful)

    by Charliemopps (1157495) on Friday December 07, 2012 @12:39PM (#42216359)

    How many of these asinine data center advertisements are we going to get? This is at least the 3rd "How such and such data center survived Sandy!" I don't care... it's not news. You told your employees to stand in knee deep water in the middle of tons of electronic equipment and bail water? You're a god damned fool and lucky no-one got killed.

  • by ColdWetDog (752185) on Friday December 07, 2012 @12:42PM (#42216403) Homepage

    Diesel burns readily. Doesn't flash. But dragging it up a stairway in minimally closed containers is stupid. Drop 5 gallons in a stairway and you have a real mess even if it didn't burn. It will leak under door frames, it's slippery. Fumes are dangerous. Diesel is really, really hard to clean up after.

    This was a bad idea on a number of levels, the fire risk being only one of them.

  • by sunking2 (521698) on Friday December 07, 2012 @12:51PM (#42216511)
    These were web servers, most of which apparently were already shut down. Not a city/town :)
  • by mrchaotica (681592) * on Friday December 07, 2012 @12:55PM (#42216599)

    In a stairwell, I'd be more worried about diesel's oily slipperiness than its flammability.

  • by Runaway1956 (1322357) on Friday December 07, 2012 @12:55PM (#42216605) Homepage Journal

    Yesterday, I squandered mod points that were going to expire. Today - I wish I had some. Screw the beauracrats, sometimes you just gotta do what's gotta be done!

  • by alexgieg (948359) <alexgieg@gmail.com> on Friday December 07, 2012 @01:25PM (#42217027) Homepage

    The guy should be fired for needless risk to not only the people, but the building in general.

    When did being utterly devoid of courage and constantly afraid of every single thing under the sun became a virtue?

    "USA, land of the restrained, home of the fearful"?

  • by alexgieg (948359) <alexgieg@gmail.com> on Friday December 07, 2012 @03:23PM (#42218629) Homepage

    They're lucky they're not in the middle of a lawsuit

    In other words: when lawyers took over.

The best way to avoid responsibility is to say, "I've got responsibilities."

Working...