Forgot your password?
typodupeerror
Hardware

How Peer1 Survived Sandy 130

Posted by Soulskill
from the lots-of-inflatable-rafts dept.
Nerval's Lobster writes "When hurricane Sandy knocked out the electricity in lower Manhattan, data-center operator Peer1 took extreme measures to keep its servers humming, assembling a bucket brigade that carried diesel fuel up several flights of stairs. Ted Smith, senior vice president of operations for Peer1, talks about the decisions made as the floodwaters rose and the main generators went offline, as well as the changes his company has made in the aftermath of the storm. He said, 'When the water got to a point that it had flooded the infrastructure and the basement, we were then operating under the reserves the building had on the roof, and our own storage tanks. Literally, at that point we had to do calculations as to how long we could run. And we believed we had enough diesel fuel—between what is in the building, and in our tanks, to about 9 AM the following day. ... You know the bucket brigade—it’s something I’ve never asked the team to do. If you think about what that was at that time, you’re talking about carrying fuel up 17 flights, in total darkness, throughout a whole evening. We had informed our data center manager that we were shutting down, but he kind of took on it himself to say, ‘Not on my watch.’ And he organized himself, got a temporary solution and then more customers jumped in. And at peak I think we had about 30 people helping.'"
This discussion has been archived. No new comments can be posted.

How Peer1 Survived Sandy

Comments Filter:
  • They didn't survive (Score:5, Informative)

    by CastrTroy (595695) on Friday December 07, 2012 @12:03PM (#42215823) Homepage
    From what I hear, based on the StackExchange podcast, and the tweets that went out from SquareSpace and StackExchange during the whole idea is that Peer1 had a complete failure, and it was only due to the hard work of their customers (SE and SquareSpace) that the datacenter was able to remain operational. If your customers have to start carrying buckets of diesel up 17 flights of starirs, you, as a datacenter have failed. Peer1, left to their own devices would have just let the thing shutdown, and apparently head office wasn't aware of how bad things even were.
  • by Gordonjcp (186804) on Friday December 07, 2012 @12:17PM (#42216029) Homepage

    You do know that diesel pretty much doesn't burn, right? You actually have to try pretty hard to set a puddle of diesel on fire.

  • by Anonymous Coward on Friday December 07, 2012 @12:20PM (#42216071)

    This is exactly what happened. Peer1 did very little itself, and at one point there was the possibility that they would deny access to the customers who were putting in their own time and effort to keep the data center running. Fogcreek maintained a good status blog, if you're interested:
    http://status.fogcreek.com/2012/10/

  • by onkelonkel (560274) on Friday December 07, 2012 @12:26PM (#42216149)
    Oh hell. I just looked at the picture in TFA. They were carrying it in buckets. Buckets with lids, but not fuel cans. You were right, I was wrong.
  • by operagost (62405) on Friday December 07, 2012 @12:26PM (#42216155) Homepage Journal
    It also doesn't vaporize at room temperature like gasoline does. A spark can start a gasoline fire, whereas diesel fuel needs to be atomized. Geeks should know this.
  • by The Moof (859402) on Friday December 07, 2012 @02:56PM (#42218297)

    When did being utterly devoid of courage and constantly afraid of every single thing under the sun became a virtue?

    When it endangers the lives of my customers. Look, I'm all for stories about heroic efforts of uptime, but this company had no disaster plan. They're lucky they're not in the middle of a lawsuit for injury, or worse, wrongful death right now

  • by LoRdTAW (99712) on Friday December 07, 2012 @03:41PM (#42218869)

    Also called dyed #2 heating oil or dyed #2 diesel. They add a red dye to the fuel which enables its presence to be detected in on-road vehicle tanks. Some rural gas stations and truck stops sell dyed #2 oil (as well as kerosene) out of a pump right next to the other fuel pumps. The heating oil taxes are much lower than road fuel taxes so its very tempting to put heating fuel in your tank which costs nearly half of what you normally pay. But during roadside inspections they will check the tanks for red fuel. God help you if you get caught, high fines and they may impound your vehicle. In Louisiana they charge you $100 per gallon of vehicle fuel tank capacity, even if they only find a trace. Many trucks have a 50-300 gallon capacity, OUCH! They do however allow you to fill tanks of off road vehicles like site trucks, construction/farm equipment as well as the refrigeration systems on reefer trailers. It just cant be in the tank of a vehicle that normally travels on a public road.

  • by digitalsolo (1175321) on Friday December 07, 2012 @04:19PM (#42219333) Homepage
    Those look just like the buckets that I get transmission fluid in (from the tractor store). Transmission fluid is a petroleum product, and, in fact, will run a diesel engine just fine. I'd have no issues using those buckets to store/transfer diesel fluid.

Things are not as simple as they seems at first. - Edward Thorp

Working...