Datagram Recovers From 'Apocalyptic' Flooding During Sandy 114
1sockchuck writes "During SuperStorm Sandy, few data centers faced a bigger challenge than the Datagram facility in lower Manhattan. The storm surge from Sandy flooded its basement, disabling critical pumps. 'It was apocalyptic,' said CEO Alex Reppen. 'It was like a tidal wave over lower Manhattan.' While companies like CoreSite dealt primarily with the loss of ConEd power, the Datagram team sought to recover operations in an active flood zone. Why was mission-critical equipment in the basement? Because city officials restrict placing fuel tanks on rooftops and upper floors, citing concerns about diesel emerging from the 9-11 attacks."
Smart thinking (Score:5, Insightful)
Everyone knows that flying airplanes into the tops of buildings happens more often than floods in the basement. Gotta keep the priorities straight.
Re: (Score:2)
The concerns were raised by 9/11, but do not necessarily involve airplanes.
In any case, you can place the tanks in the basement and locate the pumps up higher. If the tanks aren't watertight, you'd need to work out a system of separation, or at least float the intake.
Re: (Score:3)
Yes, it is infinitely safer to store a large quantity of fuel in the basement, near the foundation of the building, so if somebody were to figure out a way to explode that fuel it would...
destroy the foundation of the building
You apparently, don't know a lot about demolitions.
Unless you add in a sophisticated (in context of discussion...) apparatus to convert those aforementioned storage tanks into a fuel-air explosive/explosion(F.A.E.), then your fear 'only' results in a wicked fire. Yeah, bad, but not as bad as you paint the picture.
To 'destroy the foundation of the building', you need to target the structural supports with adequate force in the correct direction.
It is much easier to convert rooftop (or similar) fuel-tanks int
Re: (Score:2)
Diesel doesn't explode. The problem with a tank at the top is that the fuel can escape and gravity can help it spread over a much larger area, setting everything flammable ablaze. It can pour down otherwise-fireproof areas, such as elevator shafts and stairwells.
Even if some nightmare scenario had the diesel escaping from it's basement tank, it still wouldn't be able to actually go anywhere and spread it's mischief. The closed nature of a basement is also going to limit combustion.
Re: (Score:1)
Re: (Score:2)
Everyone says this, but I think it is actually the rental truck that makes it explosive...
Re: (Score:2)
And a tank full of diesel becomes vapor how?
Re:Smart thinking (Score:5, Insightful)
Well, if gravity....
Never mind. I doubt you will understand.
The problems with fuel taken away from 9/11 wasn't that planes will fly into the roof. It's that fuel is a liquid and it is subject to gravity which means anything puncturing the tanks will cause it to leak down the building whether it is on fire or not. So imagine a lightning strike happens and years the side of the tank out. It caught fire and is now seeping down the stair well and over the sides of the building and through the crack in the roof that got there by the initial explosion caused by the lightning. This can quickly engulf a building and make escape routes impassable.
Lightning, contractor errors, equipment malfunctions, sabotage, all happen more then floods.
Re:Smart thinking (Score:5, Informative)
I'm sorry but how often does lightning crack the roof of a skyscraper after splitting open a double-walled fuel tank all while missin the lightning rods? That also assumes an exposed tank on the roof. Generators and similar equipment is typically anywhere above the 5th floor. For example the new 4WTC building has it's generators on the ~50th floor.
Equipment malfunction or sabotage could easily have the basement pumps pushing diesel fuel into a huge puddle in the generator room that's on fire. When, excluding 9/11, did generator fuel spill from a roof tank in a skyscraper in the manner you describe?
It's overreaction to a single event. Just like every plastic bag is labeled to remind you not to let infants play with them, poison labels also explicitly state not to eat, and anything with an open flame usually says it's hot.
There are many disadvantages to putting critical infrastructure in the basement as well...as we've seen.
Re: (Score:1)
There are many disadvantages to putting critical infrastructure in the basement as well...as we've seen.
The data center for our school system is located in a basement of a building under an octopus of of a 50 year old sewage plumbing, directly under a restroom with several sinks and toilets. The explanation of why this was done was because at the time the upstairs of the building was just being used for storage, and that the data-center is much of an improvement over the old one?
It turns out that the toilets in the upstairs had been flushed weekly for years to keep the traps in the plumbing full. This was not
Re: (Score:2)
"There are many disadvantages to putting critical infrastructure in the basement as well...as we've seen."
Not if you DO IT RIGHT. Marine designers have known how to seal electrical lines from the ocean for more than ONE HUNDRED YEARS.
There are other alternatives.
Hydraulically-driven pumps built from mostly COTS parts could be powered from well ABOVE flood waters with zero electrical exposure to either fuel or water. Such pumps are a staple of aircraft fuel system design and are highly reliable. Much heavier
Re: (Score:2)
Re: (Score:2)
False. Completely false. Buildings are designed to prevent exactly this problem. Fire doors, fire stops, etc. are all designed, and required by code, to be used in order to prevent fire and smoke from spreading. You're just making this shit up off the top of your head.
Re: (Score:2)
You're assuming the fire doors A. work, and B. are between you and the fire. At least in the case of the WTC collapse, reports suggest that what I described did, in fact, occur, making the upper stairwells impassable.
Whether the fire doors between a basement generator and any stairwells would fail or not (or would get propped open in violation of code because somebody got too hot, or...) is often dependent upon circumstances beyond the designer's control.
Re: (Score:2)
He's not... the number one killer in those sorts of buildings during fires is smoke. You're talking out of your ass because you're assuming he's talking out of his.
Fire code is designed to prevent structure collapse and slow down fire progression in order to give rescue workers time to get to people. Fire emergency PROCEDURES for the workers etc inside the building are designed to slow smoke progression and give firefighers time go get to areas of the building to get people out with oxygen. However those ar
Re: (Score:2)
You may understand gravity but you fundamentally fail at understanding risk. Diesel is not some highly explosive mixture that will give off deadly vapours and ignite at the drop of a dime. If lighting strikes a diesel tank what you end up with is ... a diesel tank. nothing more. It won't catch fire. Throw a match into it it won't catch fire. Electrical faults and it won't catch fire.
Also I disagree wholeheartedly that sabotage and equipment malfunctions to a vessel built to normal storage standards happens
Re: (Score:3)
You may understand gravity but you fundamentally fail at understanding risk. Diesel is not some highly explosive mixture that will give off deadly vapours and ignite at the drop of a dime. If lighting strikes a diesel tank what you end up with is ... a diesel tank. nothing more. It won't catch fire. Throw a match into it it won't catch fire. Electrical faults and it won't catch fire.
Um.. you are assuming to much. The lightning doesn't need to ignite the diesel fuel, it can ignite something else. The lightni
Re: (Score:2)
yes, diesel will ignite with an open flame, it's the same as kerosene
It's possible to get diesel to ignite with a big enough open flame, but you have to get it hot, it's not enough to bring a flame near it. Diesel is not the same as kerosene, which is significantly more volatile. Try putting kerosene in your diesel and see what happens.
Diesel is classed as a hazardous material
So is vegetable oil.
Re: (Score:2)
Diesel is the same as kerosene with impurities for all intents and purposes. Obviously there is a different else they would have the same name. However, I have used on road and off road diesel in kerosene heaters- both the wick and forced air
Re: (Score:2)
Oh, and BTW, Diesel is classed as a hazardous material. I'm not sure what you mean by that comment. In sufficient quantities, you are required to placard the tanks and if transporting, you need a hazardous material endorsement and could need to placard the vehicle. In fact,it only takes like 200 gallons (400 lbs) before some of those hazardous material shipping rules kick in.
Yep for shipping indeed. So are lithium batteries, and all sorts of non flammable things such as bitumen.
I'm talking the American Petroleum Institute, also the IEC and the British standards IP15 don't classify diesel as a hazardous liquid when it comes to explosion and fire risk. I.e. You have a tank full of diesel at room temperature you don't need to ensure that sources of ignition aren't present in the area (FM approved lights, or non-sparking motors etc).
In terms of risk a lightning strike to the buildi
Re: (Score:2)
I was talking a direct strike and the original comment was on the roof, not on a floor inside the building. I've seen lightening rip a 10 inch seam in 1 inch
Re: (Score:2)
Now please don't tell me that a welder or plasma cutter has more power then a lightning bolt.
Only if you fail to understand the difference. A lightning strike is an intensely high voltage burst that instantly dissipates and yet sustains a very low current towards the end. The total power delivered by a lighting strike is actually incredibly low over time. If you take all the energy of a lightning strike and use it to power an arc welder which applies a continuous amount of high current yet much lower voltage to what it's cutting you can probably draw an arc for about 1/3rd of a second or so. How fa
Re: (Score:2)
This has been part of the fire code for ages; it has nothing to do with 9/11. It is also not that hard to have a fuel system that can survive a flooded basement. If you really want to go wild and crazy you can get over the lift limits on suction pumps and still keep all your electrical components on the third floor or higher.
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Hmmm... (Score:5, Funny)
If only there was some sort of technology that allowed us to keep fuel in one place and the generators somewhere else...
Re:Hmmm... (Score:5, Funny)
we could invent a flexible transport pathway for fluids analogous to an electrical wire, perhaps call it a wet-wire. and some kind of electrical pushing device to move fuel uphill through the wet-wire, maybe call it a cycling wetstuff-pushee.
Re: (Score:2)
Put it at the bottom of a Big-Gulp, and yes... yes it will.
Re: (Score:2)
Oversimplifying the problem. Fuel pumped uphill needs to overcome head pressure of where the liquid needs to go requiring significantly more power.
Just the tanks? (Score:5, Interesting)
Well, if the ordnance pertains to fuel tanks specifically, why not put waterproof tanks in the basement, and run sealed lines (including venting) up from there, locating the pumps somewhat higher. Obviously you're limited as to how much higher the pumps can be, but you can draw fuel a fair bit upwards on vacuum (maybe 20 feet?). If you're allowed to send pressurized air down the vent you could put the pumps up higher - I'm not sure what the laws are around that. If there are concerns with pressurized air mixing with fuel, another option might be a tank with a rubber bladder inside where the space between can be pressurized with either air or fluid - that's how they do it in liquid fueled rockets. As long as the tank and lines are waterproof you could keep it in the basement and operate indefinitely - but you'd need to work out all the details (like refueling - if the tank has to operate under pressure then you need to have pressure on the fueling lines as well, and suitable couplings and all that, unless you have more than one tank and can operate on one while fueling another).
All of that entails certain hazards - you'd want well-trained personnel to operate it - you're starting to resemble operations on a jet or spacecraft...
Just the tanks, generators, servers, data ... (Score:2)
Re: (Score:2)
Disaster Recovery does not mean what you think it means.
Re:Just the tanks, generators, servers, data ... (Score:4, Insightful)
Re: (Score:2)
I'm not sure how you are marked insightful. This type of redundancy, (which really puts it OUT of the realm of disaster recovery, and more of a cloud) is only available to the most wealthy and mission critical clients.
Also providing planning and facility services to other companies (which is what they advertise, an important distinction), does not mean one can afford to treat every client they host as mission critical (themselves included). Problems certainly arise when a DR site itself is facing a disa
Re: (Score:2)
Re: (Score:2)
You don't want to pressurize the tank, but you can use long shaft turbine pumps or compressed air displacement pumps in the basement and air compressors at a safe elevation. Not very efficient, but it can be done.
Re: (Score:1)
To avoid the suction height limit, put the fuel pumps in the basement, sealed against water ingress. The fuel pumps could even be inside the tank, submerged in the fuel, as is the case in many cars.
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
You mean like garage forecourt (petrol station) tanks that are invariably underground?
Those are usually a sealed tank with filling and vent pipes above ground. Now, as the filling pipes are capped when not in use, only the vent pipes would be at risk from flooding - and those can be taken up as many storeys as required and vented to the open air (away from windows and aircon intakes obviously).
The pumps could be located above the flood zone feeding generators also above the floodzone.
Re: (Score:1)
Re: (Score:2)
Good ideas. I wonder if you could use a rubber bladder to store the fuel and avoid the need to vent it, and thus the buoyancy problem.
Re: (Score:2)
It would seem to me that a better idea would be to put pumps inside the tanks, similar to the way it's done in cars. The technology for this is well tested and should not require highly trained personnel to operate it. This would avoid all the potential problems you highlight with pressurizing a tank to pushing fuel up to an elevated pump.
Re: (Score:2)
Or you could locate the pumps down in the basement and the motors near the generators, and use an old-fashioned driveshaft to transfer power. You could even run the driveshaft through the fuel line itself, allowing for a hermetically sealed t
Why? (Score:5, Insightful)
If you questioned why and you still placed your companies life in that data center you should be fired for stupidity.
It's the same reason I won't place my companies data at a DC in a crowded downtown area. Sporting events, politcial events, terrorist events.
If you say you don't have a choice then you haven't thought of alternative means. Cloud, managed hosting, or a more weather stable state.
Lower Manhattan is pretty much land filled area and 911 showed how vulnerable the WTC was below ground. They were extremely concenred about the Hudson flooding lower Manhattan.
Again if you placed your companies data at a DC in lower Manhattan you should be fired.
Re:Why? (Score:4, Insightful)
I don't know anything about Datagram, but there are legitimate reasons to have a DC in lower Manhattan... For instance, for latency-limited high-frequency trading operations. I don't know if this is the particular case with Datagram's clients, but the fact that DCs exist in a ludicrously high-rent area means that they probably exist there for a good reason.
Re: (Score:2)
I don't know anything about Datagram, but there are legitimate reasons to have a DC in lower Manhattan... For instance, for latency-limited high-frequency trading operations.
Aren't those servers in Jersey?
http://www.nytimes.com/2011/01/02/business/02speed.html
Re: (Score:3, Interesting)
A /lot/ of servers that are used by trading companies that connect to exchanges are placed as physically close to the exchange as possible. The exchanges themselves have servers and fail over data centers in multiple locations. However the primary locations are surrounded by office towers that are chock full of servers from the trading companies. These servers were likely the majority of the ones that got hosed by Sandy.
I did a bunch of work several years back with a number of exchanges and my budget was al
Re:Why? (Score:4, Informative)
Your information is either wrong or /extremely/ out of date...and you mistake what "the exchange" is these days. The major exchanges (NYSE and NADQ in particular) do not house their matching engines (which is effectively "the exchange") in/on/at their trade floors. They're all located outside of NYC in large datacenters where they colocate servers for HFTs and other customers. It would be impractical in the extreme to run the types of links used by HFT systems between offices in NYC (or anywhere outside of a datacenter.)
They do, of course, have fail-over redundant datacenters.
Also - Matching latency is measured in microseconds, not miliseconds. Taking a single millisecond, much less 100+ms, to match a trade would represent serious delays.
Re: (Score:3)
Just wanted to add: the Dutch production VPRO produced a documentary a while back that deals with this subject matter. From what I recall they actually "tour" the area where all these facilities are located nowadays.
Backlight - Money and Speed: Inside The Black Box [youtube.com]
Re: (Score:2)
My experience is indeed relatively out of date. I was working with these systems between 2005 to 2007. I worked extensively with a very large trading company that had servers worldwide. I later worked for about six months or so down at the CBOT with a different company. I had some intermittent consulting work with secondary companies as well.
As for the delays am not talking about trades, I am talking about services that run on the servers. I didn't work with the trading algorithms, I worked with the servers
Re: (Score:1)
I don't know anything about Datagram, but there are legitimate reasons to have a DC in lower Manhattan... For instance, for latency-limited high-frequency trading operations.
High-frequency trading isn't a legitimate reason for anything.
Re: (Score:2)
Re: (Score:2)
Good theory, except your example is poor. The matching engines for NYSE and Nasdaq are in NJ (with backups elsewhere) and not in lower NYC.
HFT servers colo in those datacenters, they could be in islamabad for all it matters.
There is a lot of business in NYC that doesn't 'make sense' when you look at a cost vs. space perspective though. It's not just the datacenter(s).
Re: (Score:2)
I've worked with such commercial data centers. The access to personnel, equipment, transit, multiple data feeds, and urban electrical power can far outweigh the risks of such urban centers. High reliability offsite facilities can be enormously expensive in both hardware and manpower. A data center where personnel can walk over with installation media and update the BIOS on a full rack of equipment and come back to the office for lunch is a massive savings in engineering time and resources over expensive rem
Speed of light. . . (Score:2)
If your trading needs so little latency that you have a problem with the latency added by the speed of light over a fiber cable to somewhere maybe a couple miles away in NYC on higher ground, of 10-20 miles away on mainland NY, NJ, CT, etc, maybe you're abusing the stock market, and we shouldn't bend over to help you ruin our financial system.
Re: (Score:1)
And Another Thing... (Score:2)
Re: (Score:2)
And a category 1 at that. This is the tyranny of Yankees running the media. If it snows it's `White Doom 2012`, and if it rains it either a `Perferct Storm` or a `Superstorm`. Good gosh, I sat through three hurricanes in a row back in 2004 and it obviously wasn't the end of the world.
Re:And Another Thing... (Score:5, Informative)
That's what I thought at first, having lived through Andrew in Florida -- I was all, "psh, its only a category 1". However, thi sisn't a Yankees media situation. Sandy was significantly more powerful then the category would imply.
For one thing, by the time it hit NYC, it was no longer a hurricane -- it had merged with one or two cold storm systems that were coming in from the other direction. This changed the dynamic of the storm significantly: whereas hurricanes gain their energy from the warm ocean waters, this type of storm gained its energy from the difference between the cold and hot storm systems merging together. Or something. (The precise details are not clear to me: I'm not a meteorologist)
Sandy was also *huge* -- measuring the total energy in the storm, it was bigger then Katrina. Hurricanes can get intense but the brunt of their power is focused. They may have a lot of wind speed, and strictly by that measure Sandy wasn't very impressive... but when you have a cat 1 spread out as far as Sandy was, its pulling in a HUGE amount of water.
It wasn't the wind that was so destructive here: it was the storm surge that the huge storm system brought with it.
More sciency stuff at http://www.washingtonpost.com/blogs/capital-weather-gang/post/sandy-packed-more-total-energy-than-katrina-at-landfall/2012/11/02/baa4e3c4-24f4-11e2-ac85-e669876c6a24_blog.html [washingtonpost.com] (Warning: yankee media)
But, really. Its not just rhetoric of omg the Yanks are finally getting hit that made this seem bad. It really was a very, very, very bad storm and the hurricane classification only makes it seem small.
Re: (Score:2)
Thanks for the informative reply. I stand corrected.
Re: (Score:3)
It was hugely played up by the news. I *personally* was standing in battery park behind the cameras filming the reporters being 'blown around' by the wind at one point. Note: I was standing. Still. It was quite windy but by no means as windy as they flat-out /pretended/ it was.
With that said, the storm surge was unprecedented (at least for the area) and disastrous. The odd part though, since it was water rising up from the ocean instead of rain coming down and flowing somewhere, was that some areas we
Re: (Score:2)
i prefer "weather bomb"
tank and web (Score:1)
Fuel
Some cars and a lot of gas stations use in tank pumps, just put the buster pumps high. Your problems then become leakage in to the tanks and the problem of a tank trying to float when under water.
Backup sites:
Some bussinessed can not justify full sites as backup, some do not need full sites as backup, some would be happy with just a few web pages or files. Using slashdot as an example, "We are closed/down due to ______ check back here for updates" is all that is needed on w
9/11 and Fuel Tanks (Score:5, Informative)
Citing 9/11 is interesting in light of the NIST report: [nist.gov]
Did fuel oil systems in WTC 7 contribute to its collapse?
No. The building had three separate emergency power systems, all of which ran on diesel fuel. The worst-case scenarios associated with fires being fed by ruptured fuel lines-or from fuel stored in day tanks on the lower floors-could not have been sustained long enough, could not have generated sufficient heat to weaken critical interior columns, and/or would have produced large amounts of visible smoke from the lower floors, which were not observed.
As background information, the three systems contained two 12,000 gallon fuel tanks, and two 6,000 gallon tanks beneath the building's loading docks, and a single 6,000 gallon tank on the 1st floor. In addition one system used a 275 gallon tank on the 5th floor, a 275 gallon tank on the 8th floor, and a 50 gallon tank on the 9th floor. Another system used a 275 gallon day tank on the 7th floor.
Several months after the WTC 7 collapse, a contractor recovered an estimated 23,000 gallons of fuel from these tanks. NIST estimated that the unaccounted fuel totaled 1,000 ±1,000 gallons of fuel (in other words, somewhere between 0 and 2,000 gallons, with 1,000 gallons the most likely figure). The fate of the fuel in the day tanks was unknown, so NIST assumed the worst-case scenario, namely that they were full on Sept. 11, 2001. The fate of the fuel of two 6,000 gallon tanks was also unknown. Therefore, NIST also assumed the worst-case scenario for these tanks, namely that all of the fuel would have been available to feed fires either at ground level or on the 5th floor.
Re: (Score:2)
Yea, but that's just 1 building. Shit falling from those towers screwed up half of downtown. There were probably hundreds of leaks, small and large, all over the place that they had to deal with. You can imagine that nearly every building down there had a generator and tank of some sort on the roof. Just because it didnt cause the catastrophic collapse of a very large building, does not mean the city engineers didn't realize they had a potential future problem they needed to address.
Re: (Score:2)
You will never get my business (Score:1)
Moral of story is... (Score:2)
Don't build data centers in a coastal city under sea level. Just like you don't build nuclear power plants in tsunami prone regions where the backup generator is next to the sea. We can all either fear global warming and its effects, or we can adapt to it and start thinking about how to plan for future technological advancement that doesn't leave us victims of bad weather.
There are no natural disasters, only disasters in human arrogance.
Re: (Score:2)
... under maximum possible unimaginable storm+tidal sea level ... there, fixed it for ya.
Put data centers inland at 100 meters (or more) up to protect against even tsunami level damage (that's what's coming next for NYC).
Why don't they use natural gas instead of diesel? (Score:2)
Every time I hear a sales pitch about a data center or hear of someone wanting to add generator power to their building, they talk about diesel powered generators. Usually after they show off their huge generators they talk about how often they run them for test purposes.
When you start to ask about fuel capacity they get kind of hinky and immediately start emphasizing their multi-supplier, prepaid, penalties-for-non-delivery fuel delivery contracts.
Which is all well and good if you have a simple issue, lik
Re: (Score:2)
Storage, for one. What happens when your gas pipeline is disrupted?
Re: (Score:2)
I still blame the city ... just because it is too low and too near the sea. I'd have things inland more. Geographic diversity is still a good thing, too. But lower Manhattan is not a candidate for such things.