How Peer1 Survived Sandy 130
Nerval's Lobster writes "When hurricane Sandy knocked out the electricity in lower Manhattan, data-center operator Peer1 took extreme measures to keep its servers humming, assembling a bucket brigade that carried diesel fuel up several flights of stairs. Ted Smith, senior vice president of operations for Peer1, talks about the decisions made as the floodwaters rose and the main generators went offline, as well as the changes his company has made in the aftermath of the storm. He said, 'When the water got to a point that it had flooded the infrastructure and the basement, we were then operating under the reserves the building had on the roof, and our own storage tanks. Literally, at that point we had to do calculations as to how long we could run. And we believed we had enough diesel fuel—between what is in the building, and in our tanks, to about 9 AM the following day. ... You know the bucket brigade—it’s something I’ve never asked the team to do. If you think about what that was at that time, you’re talking about carrying fuel up 17 flights, in total darkness, throughout a whole evening. We had informed our data center manager that we were shutting down, but he kind of took on it himself to say, ‘Not on my watch.’ And he organized himself, got a temporary solution and then more customers jumped in. And at peak I think we had about 30 people helping.'"
Surprising (Score:5, Funny)
Re: (Score:3)
Re: (Score:2)
I want to use their Python modules:
from Peer1 import ...
Re:Health and safety? (Score:5, Insightful)
Re:Health and safety? (Score:5, Insightful)
Re: (Score:3, Interesting)
Re: (Score:1)
Re: (Score:2)
Re: (Score:1)
If you consider the potential loss of a few days for being down compared to being sued into oblivion if an accident had occurred the answer as to whether it was the correct decision becomes obvious.
I hate this sort of argument. It creates a bogeyman about something that MIGHT happen, no matter how slim the chance, and completely ignores the fact that they did it and the worst case scenario did not happen. Unfortunately, it seems that a great deal of our lives are governed by this sort of thinking.
Re: (Score:2)
Have you ever tried to light a bucket of diesel on fire? It isn't like in the movies.
Re:Health and safety? (Score:5, Insightful)
Re: (Score:3)
Re:Health and safety? (Score:5, Informative)
Re:Health and safety? (Score:4, Informative)
Re: (Score:1)
Re: (Score:2)
Re: (Score:2)
I am sure he would have rathered it too. I bet he has already submited a request :)
That said, they may not be approved containers but, 5 gallon buckets even hold gas. Its what the scrap yard uses when draining cars for crushing. Boy do they get some gas doing that too. Saw a guy drain nearly two "full" buckets (probably about 4 gallons, really silly to fill them to the brim).
I dunno, if there was an "approved container" and a bucket, I would be happy to let someone else use the approved container. Those buc
As long as nobody drops it on the stairs. (Score:2)
I agree that in a flat, well-lit environment they'd probably be okay for gas, and even safer for diesel, especially outdoors where plants and soil would cushion the fall and vapors would dissipate quickly. Falling down an enclosed concrete stairwell though... even if the bucket itself didn't break I seriously doubt the lid would stay on, and then you'd have a real problem on your hands.
Re:Health and safety? (Score:5, Insightful)
Diesel fuel does not burn the way you think it does.
Re: (Score:1)
True, but a spilled 5 gallon bucket of diesel fuel in a dark stairwell with bog-knows-what-else going on sounds really, really, really stupid. That's enough to basically burn the building down not to mention injuring people.
The risk-benefit ratio really doesn't come down on the side of the guy who suggested it.
Re:Health and safety? (Score:5, Insightful)
In a stairwell, I'd be more worried about diesel's oily slipperiness than its flammability.
Re: (Score:2)
Re: (Score:1)
What kind of multistory buildings have you been in that can be taken down or even damaged by 5 gallons of diesel in the stairwells?
These stairwells are made to be fire resistant since they are supposed to be the safe way out in case of a fire, that means they don't burn very easily.
If they spilled the diesel and then somehow managed to fuck up and light it on fire it would burn until the fuel was gone and then they would have to replace parts of the stairwell, thats it.
Re: (Score:2)
most people killed in fires initially sucumb to smoke inhalation. and you're worried about the fucking stairwell materials.....
Re: (Score:1)
True, but a spilled 5 gallon bucket of diesel fuel in a dark stairwell with bog-knows-what-else going on sounds really, really, really stupid. That's enough to basically burn the building down not to mention injuring people.
The risk-benefit ratio really doesn't come down on the side of the guy who suggested it.
You could be more wrong if you tried really hard.
But not much more.
How many people have you heard of that burned to death from a diesel fire?
Is there a reason you have not heard of such a thing?
Let me answer.
It is really hard to get diesel to burn. You have to try.
Re: (Score:2)
Re: (Score:1)
Indeed. It's pretty hard to get a fire started even if you throw a cigarette (or a lighter) right into a puddle of Diesel.
Re: (Score:3)
Everybody likes video!
http://youtu.be/3LeRSPuA5Z4 [youtu.be]
Description: Gasoline is spilled on the ground and lit with a match. Then diesel is spilled on the ground and numerous attempts are made to light it with one then several matches.
Re: (Score:2)
Even more video! A guy tries to light a diesel spill with a blowtorch.
http://youtu.be/xMJQFl4az9I [youtu.be]
Re: (Score:3)
Perhaps you are unconsciously equating diesel fuel to gasoline? If a class alpha fire happened to break out somewhere enroute to the upstairs generator, they could likely have thrown the diesel fuel on the fire to put it out. Of course, that wouldn't be such a great idea for any other class of fire. Still - diesel fuel isn't so flammable as to cause any real hazard.
Re: (Score:2)
If a class alpha fire happened to break out somewhere enroute to the upstairs generator, they could likely have thrown the diesel fuel on the fire to put it out
A class-a fire still puts out a lot of heat. Trying to put it out with diesel will give you a class-b fire to boot, I think.
Flash point of diesel fuel's 144F so it's not exactly something I'd throw on a fire (gasoline's flash point is -45F).
It's not something I'd want to handle around an open flame or anything, but it's pretty safe otherwise.
Re: (Score:3)
Obligatory: http://xkcd.com/705/ [xkcd.com]
Re:Health and safety? (Score:5, Insightful)
The guy should be fired for needless risk to not only the people, but the building in general.
When did being utterly devoid of courage and constantly afraid of every single thing under the sun became a virtue?
"USA, land of the restrained, home of the fearful"?
Re: (Score:2, Informative)
When did being utterly devoid of courage and constantly afraid of every single thing under the sun became a virtue?
When it endangers the lives of my customers. Look, I'm all for stories about heroic efforts of uptime, but this company had no disaster plan. They're lucky they're not in the middle of a lawsuit for injury, or worse, wrongful death right now
Re:Health and safety? (Score:4, Insightful)
They're lucky they're not in the middle of a lawsuit
In other words: when lawyers took over.
Re: (Score:1)
I'm sure the customers participated volunteered willingly. I doubt anyone will sue for spraining an ankle after having volunteered willingly to help in a crisis situation. They clearly all felt they had a lot more riding on this than just their own physical well-being.
Also, in response to some of the above posters; I think the middle of a hurricane, when your basements are flooded, you're highly unlikely to worry about someone accidentally lighting the building on fire.
Re: (Score:3)
When did being utterly devoid of courage and constantly afraid of every single thing under the sun became a virtue?
If it's my company, my money, my neck is on the line...
If I'm working for someone else--who would just as soon lay me off if it became cheap enough to do so--then I do what I'm paid to do. Carrying 5G buckets of diesel up 17 dark flights of steps while the lobby has 4' of water in it is NOT what I'm being paid for.
It has nothing to do with a lack of courage or fear. It has to do with understanding that it's not worth potentially DYING or becoming disabled in a situation where the company will almost certa
Re: (Score:2)
What could possible happen if a fire occurred?
Have you ever tried to light diesel? It's much harder than you seem to think......
Re: (Score:1)
Re: (Score:2)
They didn't survive (Score:5, Informative)
Re: (Score:3, Informative)
This is exactly what happened. Peer1 did very little itself, and at one point there was the possibility that they would deny access to the customers who were putting in their own time and effort to keep the data center running. Fogcreek maintained a good status blog, if you're interested:
http://status.fogcreek.com/2012/10/
Obligatory XKCD (Score:5, Funny)
http://xkcd.com/705/
Re: (Score:2, Troll)
OSHA must be thrilled (Score:2)
Re: (Score:1)
OSHA?
What about their Insurance company.
"You did what!"
Re:OSHA must be thrilled (Score:5, Insightful)
OSHA must be thrilled
Getting OSHA / union / bubblewrap parents involved means that those who are capable of helping are not allowed to because of the risk that some idiot gets hurt or damages something.
They have their place and time when things are normal to try and minimize the impact of a disaster, but once that disaster is in full swing, they need to sit down, shut up and let people self-mobilize to get the job done.
In the spring of '97 guys were working heavy equipment for days straight, often by the light of military flares, to build a dike that saved Winnipeg from one of the biggest spring floods in our history [winnipeg.ca] (often "stealing" clay/dirt from nearby farms to get the dike to the heights needed, dragging and dumping scrap cars, buses, anything they could find to shore up the water front side from erosion, etc.). Ignoring the union rules, safety rules, land procurement rules, etc. they got it done in time.
After the flood waters receded, then all the compensating processes kicked in to address the shortcomings.
Re:OSHA must be thrilled (Score:5, Insightful)
Re: (Score:2)
Yeah no kidding... we're not talking about hospital generators... it's freaking web servers.
Re:OSHA must be thrilled (Score:5, Insightful)
Yesterday, I squandered mod points that were going to expire. Today - I wish I had some. Screw the beauracrats, sometimes you just gotta do what's gotta be done!
Re: (Score:2)
Yes. Because what they were helping was someone's cat's blog stay up, whereas the risk they were taking was someone getting seriously ill from exposure to diesel fuel -- or burning the building down. (Ever dealt with spilled diesel fuel? Nasty, nasty stuff.)
A bucket brigade of Diesel fuel? (Score:4, Insightful)
In total darkness, up 17 flights of stairs, with a flooded basement? Sounds like a recipe for a potentially fatal fire. People's lives are more important than a freaking data center. Sorry, but I don't see this as a heroic story about people trying to keep critical infrastructure running, but as a desperate failure that could easily have turned into a disaster. They never should have gotten to the point where they're continually carrying fuel up stairs. It also sounds like they then decided to pump fuel up a pipe they installed in the stairwell. That doesn't sound terribly safe either, especially when done in a mad rush like I'm sure it was.
Gee.. couldn't have someone planned for this contingency rather than this sort of haphazard, dangerous sounding plan that was thrown together?
Re:A bucket brigade of Diesel fuel? (Score:4, Informative)
You do know that diesel pretty much doesn't burn, right? You actually have to try pretty hard to set a puddle of diesel on fire.
Re: (Score:3)
Re:A bucket brigade of Diesel fuel? (Score:5, Informative)
Re: (Score:3)
It also doesn't vaporize at room temperature like gasoline does. A spark can start a gasoline fire, whereas diesel fuel needs to be atomized. Geeks should know this.
Those of us who drive diesels, do :)
Re: (Score:3)
Those of us who have been taught how to repair furnaces know this, also.
(For those not familiar, home heating oil is the exact same material as diesel fuel, just taxed differently)
Re: (Score:2)
(For those not familiar, home heating oil is the exact same material as diesel fuel, just taxed differently)
Did not know that (heating oil isn't very popular where I live)
Learn something new every day!
Re:A bucket brigade of Diesel fuel? (Score:4, Informative)
Also called dyed #2 heating oil or dyed #2 diesel. They add a red dye to the fuel which enables its presence to be detected in on-road vehicle tanks. Some rural gas stations and truck stops sell dyed #2 oil (as well as kerosene) out of a pump right next to the other fuel pumps. The heating oil taxes are much lower than road fuel taxes so its very tempting to put heating fuel in your tank which costs nearly half of what you normally pay. But during roadside inspections they will check the tanks for red fuel. God help you if you get caught, high fines and they may impound your vehicle. In Louisiana they charge you $100 per gallon of vehicle fuel tank capacity, even if they only find a trace. Many trucks have a 50-300 gallon capacity, OUCH! They do however allow you to fill tanks of off road vehicles like site trucks, construction/farm equipment as well as the refrigeration systems on reefer trailers. It just cant be in the tank of a vehicle that normally travels on a public road.
Re: (Score:2)
Re:A bucket brigade of Diesel fuel? (Score:4, Insightful)
Diesel burns readily. Doesn't flash. But dragging it up a stairway in minimally closed containers is stupid. Drop 5 gallons in a stairway and you have a real mess even if it didn't burn. It will leak under door frames, it's slippery. Fumes are dangerous. Diesel is really, really hard to clean up after.
This was a bad idea on a number of levels, the fire risk being only one of them.
Re: (Score:2)
"Atomized" sounds rather exotic. I use diesel fuel rather than whatever is sold as lighter fuel on my charcoal grill routinely, almost weekly, and let me tell you, if you moisten stuff with diesel and set a match to it, it burns. It doesn't "woof!" as much as gasoline, but it certainly does burn.
Re: (Score:2)
diesel fuel needs to be atomized
You haven't started a burn pile with diesel, eh?
Re: (Score:2)
Not really, you can apply a propane torch to a puddle of diesel and not get it lit, when we used it to light bonfires with wet wood we'd have to use lighter fluid to get the diesel hot enough to vaporize and then it would finally burn.
Re: (Score:3)
Re: (Score:2)
The flash point is 143 degrees Fahrenheit. So you'd need to get a portion of it near something that produces a lot of heat... like say a generator, or a server.
Re: (Score:2)
Back in my mid-teens (mumble years ago) we lived way the hell out in the country and burned most of our garbage. The procedure I followed was this: 1) pack barrel full of garbage, 2) pour two gallons of diesel fuel over the garbage, 3) light a match and drop it on the diesel soaked garbage, 4) jump back before I got singed.
Diesel fuel vaporizes just fine at room temperatu
Re: (Score:2)
Diesel doesn't start to vapourise until about 40ÂC so I'd hate to think how hot your rooms are. What do you think the glow plugs in diesel engines are for?
Re: (Score:2)
Here's a free clue for you: Put a bucket of diesel in your room and sniff - guess what you're smelling? That's right - diesel fumes. And they *will* burn in sufficient concentration.
Glow plugs are for warming and starting a cold engine.
Idiot.
Re: (Score:2)
Exactly. When the engine is cold, you need the red-hot glow plugs to get the diesel hot enough to burn. Otherwise, the stuff just won't go off.
Have you ever tried to start a diesel engine with the glow plugs out of action? Doesn't work so well, does it? Eventually - if your battery is good enough - you'll have generated enough heat in the cylinders from compression alone for the diesel to light.
It is impossible to get diesel vapour to ignite unless it's at a ridiculous concentration, and very hot. Dies
Re: (Score:2)
You can't pump 17 stories from the top, that's physics. It has to be pumped from the bottom. The basement was flooded, the pump was swamped. What other contingency do you think they should have made. These guys were hardly alone in having fuel and pumps in the basement that swamped.
You could go the route of having fuel and generators on the third floor. That's usually not an option unless you own the building as no land lord wants generators and fuel in the middle of the building.
Re: (Score:2)
The solution there is either an in-tank pump or a jet pump [wikipedia.org]. Jet pumps use the venturi effect to push water (or diesel) up the return pipe. I had one in my old house for a well that was 50 feet deep or so. The pump was above ground and there were 2 ABS pipes going down into the well - 1 for the pressurized water that supplied the venturi and one for the returned water. There was a foot valve and venturi at the bottom of the well.
Then there was this....oooops (Score:4, Interesting)
----Original Message-----
From: PEER 1 Hosting NOC [mailto:@peer1.com]
Sent: Wednesday, December 05, 2012 4:04 PM
To: @peer1.com
Cc: @peer1.com
Subject: DAILY UPDATE - NYC DATA CENTER - December 5, 2012
Dear Customer,
Our facility engineers have identified an electrical explosion located in basement 2 that caused the building to flip to generator. Commercial power is available but the building advised that we stay on generator until it is safe to do so.
More updates to follow once available.
--
Network Operations Center
PEER 1 Hosting | Ping & People
1000-555 West Hastings Street
Vancouver, BC, Canada V6B 4N5
Christmas Bonuses For All (Score:5, Insightful)
Re:Christmas Bonuses For All (Score:4, Insightful)
Zombie Apocalypse? (Score:1)
Meanwhile in other threads (Score:2)
Idiots are +5 Insightful for lambasting hosting companies for not maintaining DR and remote site capabilities throughout Sandy.
Seriously Peer1's efforts are all one can ask for and I applaud their efforts to stay online during what has to be a worst case scenario for them aside from Pandemic.
STackExchange had a nice piece on it too... (Score:2)
http://blog.stackoverflow.com/2012/11/se-podcast-36-we-got-hit-by-a-hurricane/ [stackoverflow.com]
http://www.podtrac.com/pts/redirect.mp3/feeds.soundcloud.com/stream/66762703-stack-exchange-stack-exchange-podcast-36.mp3 [podtrac.com]
not what was interesting (Score:3)
Re: (Score:1)
And I have little sympathy for a company that chooses to put a data center in a flood plane
That's "floodplain" or "flood plain".
Re: (Score:2)
Why?
Afaict fiber cables don't care about being underwater and any sane datacenter will run their network gear off protected power. So as long as the power stays on and the datacenters at the other end of the fibers stay up communication should be maintained.
Re: (Score:2)
I have some stuff at Peer1, and honestly... the reason they are there is that the main east coast trunk to europe is there. The ping times to Europe from there are limited by the speed of light alone. That, and the trading floors are in close physical proximity.
Was it worth it? (Score:5, Insightful)
Having your employees stay in a emergency stricken zone that is flooded and carrying open canisters of diesel fuel to keep a data center running so that someone in California can share pictures of their cat is really not worth it IMHO.
I am sure some people were probably a little more worried about the lives of their families and themselves rather then some digital data.
I am not going to call someone a hero for this. At some point out there, people using cloud services and online storage are going to have to accept the fact that during emergency situations, their data just isn't accessible, period.
The basic fundamental problem I have about all this and what Sandy has highlighted is that the Internet was designed to be decentralized solely for the purpose of surviving natural or man-made disasters. Why is it then that a data center company creates a single centralized storage site instead of having an auxiliary site somewhere else, even on the other side of the country.
I think this is an epic fail in planning and execution. Anyone using Peer1 shouldn't be happy for putting people's lives in danger when common sense could have had them build in redundancy to their infrastructure allowing people to worry about their families more then your data.
Also, just like with Japan, don't build your backup generators at or below sea level.
Re: (Score:3)
The datacenter operators can provide multiple sites but ultimately it's up to the customers to pay for servers at multiple sites and design their applications to fail over cleanly.
another one? (Score:4, Insightful)
How many of these asinine data center advertisements are we going to get? This is at least the 3rd "How such and such data center survived Sandy!" I don't care... it's not news. You told your employees to stand in knee deep water in the middle of tons of electronic equipment and bail water? You're a god damned fool and lucky no-one got killed.
Re: (Score:2)
Agreed. The first one was an obvious Slashvertisement but might have had some interest for a few people as a human interest story. The second one was a dupe. The third one is just annoying.
I feel obligated to post this link. (Score:1)
like the Interdictor blog (Score:3)
Haven't read the details of Peer1's trials and tribulations, but the situation reminds me of the Interdictor blog [livejournal.com], about keeping DirectNIC running during Hurricane Katrina. That was one of the most thrilling blogs I've ever read.
Diesel Servers FTW! (Score:3)
...bucket brigade that carried diesel fuel up several flights of stairs...
Wow, their servers are diesel powered? Awesome!
Keeping fuel in the basement? (Score:3)
Re: (Score:2)
When taking the decision to keep the emergency backup fuel pumps in the basement, did no one think of what would happen in the event of flooding.
Probably someone thought about it, but decided that other potential hazards (e.g., a leak in the tank causing fuel to be soaked all through several floors of the building) were more important to deal with. There are many different sorts of risks, and optimal mitigation strategies for one can be very poor for others. You can guess what the likelihood of each particular risk is, but that's definitely guesswork; the whole of New York really wasn't set up with this sort of storm surge in mind.
NYC not prepared for storm surge? (Score:2)
Specious logic, the position of the fuel pumps don't contribute to fuel tank leakages
"You can guess what the likelihood of each particular risk is, but that's definitely guesswork;
No need to guess, certain people are paid a lot of money to analyse the risks,
"the whole of New York really wa
Having Been a Dir. of Ops IRL... (Score:2)
What the SE crew did to keep their site up was amazing. They got no help from their site until way late in the game.
They counted on Peer1 to handle facilities... and they dropped the ball.
I have had multiple 72 hours outage due to local power going away. Having to plan to have fuel delivered to the backup gen-set, and having to ration power (I knew the burn rates and what equipment was non-essiental)... blah blah blah.
The bottom line is that the customers are not in the NOC, so if the organization cares