Slashdot is powered by your submissions, so send in your scoop

 



Forgot your password?
typodupeerror
Hardware

How Peer1 Survived Sandy 130

Posted by Soulskill
from the lots-of-inflatable-rafts dept.
Nerval's Lobster writes "When hurricane Sandy knocked out the electricity in lower Manhattan, data-center operator Peer1 took extreme measures to keep its servers humming, assembling a bucket brigade that carried diesel fuel up several flights of stairs. Ted Smith, senior vice president of operations for Peer1, talks about the decisions made as the floodwaters rose and the main generators went offline, as well as the changes his company has made in the aftermath of the storm. He said, 'When the water got to a point that it had flooded the infrastructure and the basement, we were then operating under the reserves the building had on the roof, and our own storage tanks. Literally, at that point we had to do calculations as to how long we could run. And we believed we had enough diesel fuel—between what is in the building, and in our tanks, to about 9 AM the following day. ... You know the bucket brigade—it’s something I’ve never asked the team to do. If you think about what that was at that time, you’re talking about carrying fuel up 17 flights, in total darkness, throughout a whole evening. We had informed our data center manager that we were shutting down, but he kind of took on it himself to say, ‘Not on my watch.’ And he organized himself, got a temporary solution and then more customers jumped in. And at peak I think we had about 30 people helping.'"
This discussion has been archived. No new comments can be posted.

How Peer1 Survived Sandy

Comments Filter:
  • Surprising (Score:5, Funny)

    by Lieutenant Buddha (1660501) on Friday December 07, 2012 @11:00AM (#42215777)
    I would have thought that they barricaded the doors and windows with wicker baskets and throw pillows. Wait...
  • They didn't survive (Score:5, Informative)

    by CastrTroy (595695) on Friday December 07, 2012 @11:03AM (#42215823) Homepage
    From what I hear, based on the StackExchange podcast, and the tweets that went out from SquareSpace and StackExchange during the whole idea is that Peer1 had a complete failure, and it was only due to the hard work of their customers (SE and SquareSpace) that the datacenter was able to remain operational. If your customers have to start carrying buckets of diesel up 17 flights of starirs, you, as a datacenter have failed. Peer1, left to their own devices would have just let the thing shutdown, and apparently head office wasn't aware of how bad things even were.
    • Re: (Score:3, Informative)

      by Anonymous Coward

      This is exactly what happened. Peer1 did very little itself, and at one point there was the possibility that they would deny access to the customers who were putting in their own time and effort to keep the data center running. Fogcreek maintained a good status blog, if you're interested:
      http://status.fogcreek.com/2012/10/

  • by Anonymous Coward on Friday December 07, 2012 @11:04AM (#42215833)

    http://xkcd.com/705/

  • The myriad of regulations that were probably broken during all this should turn a watchful eye. Luckily no one was hurt and nothing bad happened, but it was just that, luck.
    • OSHA?

      What about their Insurance company.

      "You did what!"

    • by ehud42 (314607) on Friday December 07, 2012 @11:36AM (#42216323) Homepage

      OSHA must be thrilled

      Getting OSHA / union / bubblewrap parents involved means that those who are capable of helping are not allowed to because of the risk that some idiot gets hurt or damages something.

      They have their place and time when things are normal to try and minimize the impact of a disaster, but once that disaster is in full swing, they need to sit down, shut up and let people self-mobilize to get the job done.

      In the spring of '97 guys were working heavy equipment for days straight, often by the light of military flares, to build a dike that saved Winnipeg from one of the biggest spring floods in our history [winnipeg.ca] (often "stealing" clay/dirt from nearby farms to get the dike to the heights needed, dragging and dumping scrap cars, buses, anything they could find to shore up the water front side from erosion, etc.). Ignoring the union rules, safety rules, land procurement rules, etc. they got it done in time.

      After the flood waters receded, then all the compensating processes kicked in to address the shortcomings.

      • by sunking2 (521698) on Friday December 07, 2012 @11:51AM (#42216511)
        These were web servers, most of which apparently were already shut down. Not a city/town :)
        • by ethanms (319039)

          Yeah no kidding... we're not talking about hospital generators... it's freaking web servers.

      • by Runaway1956 (1322357) on Friday December 07, 2012 @11:55AM (#42216605) Homepage Journal

        Yesterday, I squandered mod points that were going to expire. Today - I wish I had some. Screw the beauracrats, sometimes you just gotta do what's gotta be done!

      • Getting OSHA / union / bubblewrap parents involved means that those who are capable of helping are not allowed to because of the risk that some idiot gets hurt or damages something.

        Yes. Because what they were helping was someone's cat's blog stay up, whereas the risk they were taking was someone getting seriously ill from exposure to diesel fuel -- or burning the building down. (Ever dealt with spilled diesel fuel? Nasty, nasty stuff.)

        In the spring of '97 guys were working heavy equipment for days straight.

  • by Vellmont (569020) on Friday December 07, 2012 @11:13AM (#42215977) Homepage

    In total darkness, up 17 flights of stairs, with a flooded basement? Sounds like a recipe for a potentially fatal fire. People's lives are more important than a freaking data center. Sorry, but I don't see this as a heroic story about people trying to keep critical infrastructure running, but as a desperate failure that could easily have turned into a disaster. They never should have gotten to the point where they're continually carrying fuel up stairs. It also sounds like they then decided to pump fuel up a pipe they installed in the stairwell. That doesn't sound terribly safe either, especially when done in a mad rush like I'm sure it was.

    Gee.. couldn't have someone planned for this contingency rather than this sort of haphazard, dangerous sounding plan that was thrown together?

    • by Gordonjcp (186804) on Friday December 07, 2012 @11:17AM (#42216029) Homepage

      You do know that diesel pretty much doesn't burn, right? You actually have to try pretty hard to set a puddle of diesel on fire.

      • by sunking2 (521698)
        Wrong, diesel fuel burns just fine, though it is slightly more difficult than gasoline. What it doesn't do is explode under normal compression.
        • by operagost (62405) on Friday December 07, 2012 @11:26AM (#42216155) Homepage Journal
          It also doesn't vaporize at room temperature like gasoline does. A spark can start a gasoline fire, whereas diesel fuel needs to be atomized. Geeks should know this.
          • It also doesn't vaporize at room temperature like gasoline does. A spark can start a gasoline fire, whereas diesel fuel needs to be atomized. Geeks should know this.

            Those of us who drive diesels, do :)

            • Those of us who have been taught how to repair furnaces know this, also.

              (For those not familiar, home heating oil is the exact same material as diesel fuel, just taxed differently)

              • (For those not familiar, home heating oil is the exact same material as diesel fuel, just taxed differently)

                Did not know that (heating oil isn't very popular where I live)

                Learn something new every day!

              • by LoRdTAW (99712) on Friday December 07, 2012 @02:41PM (#42218869)

                Also called dyed #2 heating oil or dyed #2 diesel. They add a red dye to the fuel which enables its presence to be detected in on-road vehicle tanks. Some rural gas stations and truck stops sell dyed #2 oil (as well as kerosene) out of a pump right next to the other fuel pumps. The heating oil taxes are much lower than road fuel taxes so its very tempting to put heating fuel in your tank which costs nearly half of what you normally pay. But during roadside inspections they will check the tanks for red fuel. God help you if you get caught, high fines and they may impound your vehicle. In Louisiana they charge you $100 per gallon of vehicle fuel tank capacity, even if they only find a trace. Many trucks have a 50-300 gallon capacity, OUCH! They do however allow you to fill tanks of off road vehicles like site trucks, construction/farm equipment as well as the refrigeration systems on reefer trailers. It just cant be in the tank of a vehicle that normally travels on a public road.

                • Interesting. I know a couple guys that run their trucks on recycled transmission fluid (they buy used fluid and filter it, basically). It's bright red like heating oil. Lucky for them I guess, that they don't do checks like that up here.
          • by ColdWetDog (752185) on Friday December 07, 2012 @11:42AM (#42216403) Homepage

            Diesel burns readily. Doesn't flash. But dragging it up a stairway in minimally closed containers is stupid. Drop 5 gallons in a stairway and you have a real mess even if it didn't burn. It will leak under door frames, it's slippery. Fumes are dangerous. Diesel is really, really hard to clean up after.

            This was a bad idea on a number of levels, the fire risk being only one of them.

          • by timeOday (582209)

            a spark can start a gasoline fire, whereas diesel fuel needs to be atomized. Geeks should know this.

            "Atomized" sounds rather exotic. I use diesel fuel rather than whatever is sold as lighter fuel on my charcoal grill routinely, almost weekly, and let me tell you, if you moisten stuff with diesel and set a match to it, it burns. It doesn't "woof!" as much as gasoline, but it certainly does burn.

          • diesel fuel needs to be atomized

            You haven't started a burn pile with diesel, eh?

        • by afidel (530433)

          Not really, you can apply a propane torch to a puddle of diesel and not get it lit, when we used it to light bonfires with wet wood we'd have to use lighter fluid to get the diesel hot enough to vaporize and then it would finally burn.

        • by Bengie (1121981)
          Diesel is so bad at starting on fire, you don't start the diesel on fire, you start something else on fire and let it sit with direct flame contact to the diesel for quite a while. You really need an established fire already to get diesel to start. Unless the building is already on fire or someone is lighting up cloth and dropping it in the stair well, you probably won't get any action.
      • by Vellmont (569020)

        The flash point is 143 degrees Fahrenheit. So you'd need to get a portion of it near something that produces a lot of heat... like say a generator, or a server.

      • You do know that diesel pretty much doesn't burn, right? You actually have to try pretty hard to set a puddle of diesel on fire.

        Back in my mid-teens (mumble years ago) we lived way the hell out in the country and burned most of our garbage. The procedure I followed was this: 1) pack barrel full of garbage, 2) pour two gallons of diesel fuel over the garbage, 3) light a match and drop it on the diesel soaked garbage, 4) jump back before I got singed.

        Diesel fuel vaporizes just fine at room temperatu

        • by Gordonjcp (186804)

          Diesel doesn't start to vapourise until about 40ÂC so I'd hate to think how hot your rooms are. What do you think the glow plugs in diesel engines are for?

          • Here's a free clue for you: Put a bucket of diesel in your room and sniff - guess what you're smelling? That's right - diesel fumes. And they *will* burn in sufficient concentration.

            Glow plugs are for warming and starting a cold engine.

            Idiot.

            • by Gordonjcp (186804)

              Exactly. When the engine is cold, you need the red-hot glow plugs to get the diesel hot enough to burn. Otherwise, the stuff just won't go off.

              Have you ever tried to start a diesel engine with the glow plugs out of action? Doesn't work so well, does it? Eventually - if your battery is good enough - you'll have generated enough heat in the cylinders from compression alone for the diesel to light.

              It is impossible to get diesel vapour to ignite unless it's at a ridiculous concentration, and very hot. Dies

    • by Kagato (116051)

      You can't pump 17 stories from the top, that's physics. It has to be pumped from the bottom. The basement was flooded, the pump was swamped. What other contingency do you think they should have made. These guys were hardly alone in having fuel and pumps in the basement that swamped.

      You could go the route of having fuel and generators on the third floor. That's usually not an option unless you own the building as no land lord wants generators and fuel in the middle of the building.

  • by Anonymous Coward on Friday December 07, 2012 @11:15AM (#42215999)

    ----Original Message-----
    From: PEER 1 Hosting NOC [mailto:@peer1.com]
    Sent: Wednesday, December 05, 2012 4:04 PM
    To: @peer1.com
    Cc: @peer1.com
    Subject: DAILY UPDATE - NYC DATA CENTER - December 5, 2012

    Dear Customer,

    Our facility engineers have identified an electrical explosion located in basement 2 that caused the building to flip to generator. Commercial power is available but the building advised that we stay on generator until it is safe to do so.

    More updates to follow once available.

    --
    Network Operations Center

    PEER 1 Hosting | Ping & People

    1000-555 West Hastings Street
    Vancouver, BC, Canada V6B 4N5

  • by Supp0rtLinux (594509) <Supp0rtLinux@yahoo.com> on Friday December 07, 2012 @11:16AM (#42216019)
    All I can say is you damn well better reward all your employees that helped. They kept you up and kept your revenue stream moving. You need to give them some kickass holiday bonuses or you're all major douche bags.
    • by Supp0rtLinux (594509) <Supp0rtLinux@yahoo.com> on Friday December 07, 2012 @11:19AM (#42216063)
      Or if this other comment is true: "From what I hear, based on the StackExchange podcast, and the tweets that went out from SquareSpace and StackExchange during the whole idea is that Peer1 had a complete failure, and it was only due to the hard work of their customers (SE and SquareSpace) that the datacenter was able to remain operational. If your customers have to start carrying buckets of diesel up 17 flights of starirs, you, as a datacenter have failed. Peer1, left to their own devices would have just let the thing shutdown, and apparently head office wasn't aware of how bad things even were." then you better give your customers some free months of service for doing your job for you. Either way, figure out who kept it going and reward them handsomely or you suck!!!
  • The original post sounds like a snippet from that Corey Doctorow end-of-the-world novel [craphound.com]. Did they have to find parts to fashion a rudimentary lathe along the way? I applaud the efforts of the server team, but as one commenter stated, it sounds like a failure of the company's business continuity/disaster recovery plan. The cost of dealing with employees and customers in a burn ward should overshadow revenue flow.
  • Idiots are +5 Insightful for lambasting hosting companies for not maintaining DR and remote site capabilities throughout Sandy.

    Seriously Peer1's efforts are all one can ask for and I applaud their efforts to stay online during what has to be a worst case scenario for them aside from Pandemic.

  • by frovingslosh (582462) on Friday December 07, 2012 @11:35AM (#42216313)
    I don't find the bucket brigade thing that interesting. And I have little sympathy for a company that chooses to put a data center in a flood plane (and in very expensive real estate at that). What I find interesting is that the data center apparently was able to keep a connection to the rest of the world. I would have expected the power outages and the flooding to disconnect it, even if it could power itself.
    • And I have little sympathy for a company that chooses to put a data center in a flood plane

      That's "floodplain" or "flood plain".

    • Why?

      Afaict fiber cables don't care about being underwater and any sane datacenter will run their network gear off protected power. So as long as the power stays on and the datacenters at the other end of the fibers stay up communication should be maintained.

    • I have some stuff at Peer1, and honestly... the reason they are there is that the main east coast trunk to europe is there. The ping times to Europe from there are limited by the speed of light alone. That, and the trading floors are in close physical proximity.

  • Was it worth it? (Score:5, Insightful)

    by TheSkepticalOptimist (898384) on Friday December 07, 2012 @11:36AM (#42216329)

    Having your employees stay in a emergency stricken zone that is flooded and carrying open canisters of diesel fuel to keep a data center running so that someone in California can share pictures of their cat is really not worth it IMHO.

    I am sure some people were probably a little more worried about the lives of their families and themselves rather then some digital data.

    I am not going to call someone a hero for this. At some point out there, people using cloud services and online storage are going to have to accept the fact that during emergency situations, their data just isn't accessible, period.

    The basic fundamental problem I have about all this and what Sandy has highlighted is that the Internet was designed to be decentralized solely for the purpose of surviving natural or man-made disasters. Why is it then that a data center company creates a single centralized storage site instead of having an auxiliary site somewhere else, even on the other side of the country.

    I think this is an epic fail in planning and execution. Anyone using Peer1 shouldn't be happy for putting people's lives in danger when common sense could have had them build in redundancy to their infrastructure allowing people to worry about their families more then your data.

    Also, just like with Japan, don't build your backup generators at or below sea level.

    • The datacenter operators can provide multiple sites but ultimately it's up to the customers to pay for servers at multiple sites and design their applications to fail over cleanly.

  • another one? (Score:4, Insightful)

    by Charliemopps (1157495) on Friday December 07, 2012 @11:39AM (#42216359)

    How many of these asinine data center advertisements are we going to get? This is at least the 3rd "How such and such data center survived Sandy!" I don't care... it's not news. You told your employees to stand in knee deep water in the middle of tons of electronic equipment and bail water? You're a god damned fool and lucky no-one got killed.

    • by ceoyoyo (59147)

      Agreed. The first one was an obvious Slashvertisement but might have had some interest for a few people as a human interest story. The second one was a dupe. The third one is just annoying.

  • Thus, resulting in the obligatory XKCD [xkcd.com].
  • by ipxodi (156633) on Friday December 07, 2012 @11:50AM (#42216505) Homepage

    Haven't read the details of Peer1's trials and tribulations, but the situation reminds me of the Interdictor blog [livejournal.com], about keeping DirectNIC running during Hurricane Katrina. That was one of the most thrilling blogs I've ever read.

  • by wcrowe (94389) on Friday December 07, 2012 @03:34PM (#42219507)

    ...bucket brigade that carried diesel fuel up several flights of stairs...

    Wow, their servers are diesel powered? Awesome!
     

  • by dgharmon (2564621) on Friday December 07, 2012 @05:24PM (#42220537) Homepage
    When taking the decision to keep the emergency backup fuel pumps in the basement, did no one think of what would happen in the event of flooding.
    • by dkf (304284)

      When taking the decision to keep the emergency backup fuel pumps in the basement, did no one think of what would happen in the event of flooding.

      Probably someone thought about it, but decided that other potential hazards (e.g., a leak in the tank causing fuel to be soaked all through several floors of the building) were more important to deal with. There are many different sorts of risks, and optimal mitigation strategies for one can be very poor for others. You can guess what the likelihood of each particular risk is, but that's definitely guesswork; the whole of New York really wasn't set up with this sort of storm surge in mind.

      • "Probably someone thought about it, but decided that other potential hazards (e.g., a leak in the tank causing fuel to be soaked all through several floors of the building) were more important to deal with"

        Specious logic, the position of the fuel pumps don't contribute to fuel tank leakages ..

        "You can guess what the likelihood of each particular risk is, but that's definitely guesswork;

        No need to guess, certain people are paid a lot of money to analyse the risks,

        "the whole of New York really wa
  • What the SE crew did to keep their site up was amazing. They got no help from their site until way late in the game.

    They counted on Peer1 to handle facilities... and they dropped the ball.

    I have had multiple 72 hours outage due to local power going away. Having to plan to have fuel delivered to the backup gen-set, and having to ration power (I knew the burn rates and what equipment was non-essiental)... blah blah blah.

    The bottom line is that the customers are not in the NOC, so if the organization cares

"Anyone attempting to generate random numbers by deterministic means is, of course, living in a state of sin." -- John Von Neumann

Working...