Datagram Recovers From 'Apocalyptic' Flooding During Sandy 114
1sockchuck writes "During SuperStorm Sandy, few data centers faced a bigger challenge than the Datagram facility in lower Manhattan. The storm surge from Sandy flooded its basement, disabling critical pumps. 'It was apocalyptic,' said CEO Alex Reppen. 'It was like a tidal wave over lower Manhattan.' While companies like CoreSite dealt primarily with the loss of ConEd power, the Datagram team sought to recover operations in an active flood zone. Why was mission-critical equipment in the basement? Because city officials restrict placing fuel tanks on rooftops and upper floors, citing concerns about diesel emerging from the 9-11 attacks."
Just the tanks? (Score:5, Interesting)
Well, if the ordnance pertains to fuel tanks specifically, why not put waterproof tanks in the basement, and run sealed lines (including venting) up from there, locating the pumps somewhat higher. Obviously you're limited as to how much higher the pumps can be, but you can draw fuel a fair bit upwards on vacuum (maybe 20 feet?). If you're allowed to send pressurized air down the vent you could put the pumps up higher - I'm not sure what the laws are around that. If there are concerns with pressurized air mixing with fuel, another option might be a tank with a rubber bladder inside where the space between can be pressurized with either air or fluid - that's how they do it in liquid fueled rockets. As long as the tank and lines are waterproof you could keep it in the basement and operate indefinitely - but you'd need to work out all the details (like refueling - if the tank has to operate under pressure then you need to have pressure on the fueling lines as well, and suitable couplings and all that, unless you have more than one tank and can operate on one while fueling another).
All of that entails certain hazards - you'd want well-trained personnel to operate it - you're starting to resemble operations on a jet or spacecraft...
Re:Why? (Score:3, Interesting)
A /lot/ of servers that are used by trading companies that connect to exchanges are placed as physically close to the exchange as possible. The exchanges themselves have servers and fail over data centers in multiple locations. However the primary locations are surrounded by office towers that are chock full of servers from the trading companies. These servers were likely the majority of the ones that got hosed by Sandy.
I did a bunch of work several years back with a number of exchanges and my budget was allocated in watts of power consumed instead of dollars. The buildings near the exchanges typically can't handle any more air conditioning and they will use power to ration how many system get installed into racks. Instead of working with rack density, cost or other factors, it was all based on watts. Location really was everything as delays were considered far more expensive than hardware.
They do this for high frequency trading of course, as they felt it was always against their favor when their was a delay on a trade. Delays were measured in the milliseconds and a delay of even a 200 ms was enough to make the IT floor start to get animated. Once delays hit 300 ms the rooms was a screaming fury until the issues were fixed. Floor plans were open without offices to allow for workers to talk without walking. I set up a bunch of monitoring software and automated response systems that would help respond to certain events (for the servers themselves). Pretty neat stuff really, very interesting to work with.