Forgot your password?
typodupeerror
Hardware

How Peer1 Survived Sandy 130

Posted by Soulskill
from the lots-of-inflatable-rafts dept.
Nerval's Lobster writes "When hurricane Sandy knocked out the electricity in lower Manhattan, data-center operator Peer1 took extreme measures to keep its servers humming, assembling a bucket brigade that carried diesel fuel up several flights of stairs. Ted Smith, senior vice president of operations for Peer1, talks about the decisions made as the floodwaters rose and the main generators went offline, as well as the changes his company has made in the aftermath of the storm. He said, 'When the water got to a point that it had flooded the infrastructure and the basement, we were then operating under the reserves the building had on the roof, and our own storage tanks. Literally, at that point we had to do calculations as to how long we could run. And we believed we had enough diesel fuel—between what is in the building, and in our tanks, to about 9 AM the following day. ... You know the bucket brigade—it’s something I’ve never asked the team to do. If you think about what that was at that time, you’re talking about carrying fuel up 17 flights, in total darkness, throughout a whole evening. We had informed our data center manager that we were shutting down, but he kind of took on it himself to say, ‘Not on my watch.’ And he organized himself, got a temporary solution and then more customers jumped in. And at peak I think we had about 30 people helping.'"
This discussion has been archived. No new comments can be posted.

How Peer1 Survived Sandy

Comments Filter:
  • by Anonymous Coward on Friday December 07, 2012 @11:15AM (#42215999)

    ----Original Message-----
    From: PEER 1 Hosting NOC [mailto:@peer1.com]
    Sent: Wednesday, December 05, 2012 4:04 PM
    To: @peer1.com
    Cc: @peer1.com
    Subject: DAILY UPDATE - NYC DATA CENTER - December 5, 2012

    Dear Customer,

    Our facility engineers have identified an electrical explosion located in basement 2 that caused the building to flip to generator. Commercial power is available but the building advised that we stay on generator until it is safe to do so.

    More updates to follow once available.

    --
    Network Operations Center

    PEER 1 Hosting | Ping & People

    1000-555 West Hastings Street
    Vancouver, BC, Canada V6B 4N5

  • by roninmagus (721889) on Friday December 07, 2012 @11:22AM (#42216101)
    "The guy should be fired" Further proof you cannot please everyone 100% of the time. Boneheaded, yes, dangerous, too. But it worked, didn't it? And now Peak10 gets all kinds of free publicity. The guy deserves recognition.

As the trials of life continue to take their toll, remember that there is always a future in Computer Maintenance. -- National Lampoon, "Deteriorata"

Working...