Datagram Recovers From 'Apocalyptic' Flooding During Sandy 114
1sockchuck writes "During SuperStorm Sandy, few data centers faced a bigger challenge than the Datagram facility in lower Manhattan. The storm surge from Sandy flooded its basement, disabling critical pumps. 'It was apocalyptic,' said CEO Alex Reppen. 'It was like a tidal wave over lower Manhattan.' While companies like CoreSite dealt primarily with the loss of ConEd power, the Datagram team sought to recover operations in an active flood zone. Why was mission-critical equipment in the basement? Because city officials restrict placing fuel tanks on rooftops and upper floors, citing concerns about diesel emerging from the 9-11 attacks."
Re:Smart thinking (Score:5, Informative)
I'm sorry but how often does lightning crack the roof of a skyscraper after splitting open a double-walled fuel tank all while missin the lightning rods? That also assumes an exposed tank on the roof. Generators and similar equipment is typically anywhere above the 5th floor. For example the new 4WTC building has it's generators on the ~50th floor.
Equipment malfunction or sabotage could easily have the basement pumps pushing diesel fuel into a huge puddle in the generator room that's on fire. When, excluding 9/11, did generator fuel spill from a roof tank in a skyscraper in the manner you describe?
It's overreaction to a single event. Just like every plastic bag is labeled to remind you not to let infants play with them, poison labels also explicitly state not to eat, and anything with an open flame usually says it's hot.
There are many disadvantages to putting critical infrastructure in the basement as well...as we've seen.
Re:And Another Thing... (Score:5, Informative)
That's what I thought at first, having lived through Andrew in Florida -- I was all, "psh, its only a category 1". However, thi sisn't a Yankees media situation. Sandy was significantly more powerful then the category would imply.
For one thing, by the time it hit NYC, it was no longer a hurricane -- it had merged with one or two cold storm systems that were coming in from the other direction. This changed the dynamic of the storm significantly: whereas hurricanes gain their energy from the warm ocean waters, this type of storm gained its energy from the difference between the cold and hot storm systems merging together. Or something. (The precise details are not clear to me: I'm not a meteorologist)
Sandy was also *huge* -- measuring the total energy in the storm, it was bigger then Katrina. Hurricanes can get intense but the brunt of their power is focused. They may have a lot of wind speed, and strictly by that measure Sandy wasn't very impressive... but when you have a cat 1 spread out as far as Sandy was, its pulling in a HUGE amount of water.
It wasn't the wind that was so destructive here: it was the storm surge that the huge storm system brought with it.
More sciency stuff at http://www.washingtonpost.com/blogs/capital-weather-gang/post/sandy-packed-more-total-energy-than-katrina-at-landfall/2012/11/02/baa4e3c4-24f4-11e2-ac85-e669876c6a24_blog.html [washingtonpost.com] (Warning: yankee media)
But, really. Its not just rhetoric of omg the Yanks are finally getting hit that made this seem bad. It really was a very, very, very bad storm and the hurricane classification only makes it seem small.
9/11 and Fuel Tanks (Score:5, Informative)
Citing 9/11 is interesting in light of the NIST report: [nist.gov]
Did fuel oil systems in WTC 7 contribute to its collapse?
No. The building had three separate emergency power systems, all of which ran on diesel fuel. The worst-case scenarios associated with fires being fed by ruptured fuel lines-or from fuel stored in day tanks on the lower floors-could not have been sustained long enough, could not have generated sufficient heat to weaken critical interior columns, and/or would have produced large amounts of visible smoke from the lower floors, which were not observed.
As background information, the three systems contained two 12,000 gallon fuel tanks, and two 6,000 gallon tanks beneath the building's loading docks, and a single 6,000 gallon tank on the 1st floor. In addition one system used a 275 gallon tank on the 5th floor, a 275 gallon tank on the 8th floor, and a 50 gallon tank on the 9th floor. Another system used a 275 gallon day tank on the 7th floor.
Several months after the WTC 7 collapse, a contractor recovered an estimated 23,000 gallons of fuel from these tanks. NIST estimated that the unaccounted fuel totaled 1,000 ±1,000 gallons of fuel (in other words, somewhere between 0 and 2,000 gallons, with 1,000 gallons the most likely figure). The fate of the fuel in the day tanks was unknown, so NIST assumed the worst-case scenario, namely that they were full on Sept. 11, 2001. The fate of the fuel of two 6,000 gallon tanks was also unknown. Therefore, NIST also assumed the worst-case scenario for these tanks, namely that all of the fuel would have been available to feed fires either at ground level or on the 5th floor.
Re:Why? (Score:4, Informative)
Your information is either wrong or /extremely/ out of date...and you mistake what "the exchange" is these days. The major exchanges (NYSE and NADQ in particular) do not house their matching engines (which is effectively "the exchange") in/on/at their trade floors. They're all located outside of NYC in large datacenters where they colocate servers for HFTs and other customers. It would be impractical in the extreme to run the types of links used by HFT systems between offices in NYC (or anywhere outside of a datacenter.)
They do, of course, have fail-over redundant datacenters.
Also - Matching latency is measured in microseconds, not miliseconds. Taking a single millisecond, much less 100+ms, to match a trade would represent serious delays.