Forgot your password?
typodupeerror
Earth Hardware

New York Data Centers Battle Floods, Utility Outages 186

Posted by Soulskill
from the bunker-in-tight dept.
miller60 writes "At least three data center buildings in lower Manhattan are struggling with power problems amid widespread flooding and utility outages caused by Hurricane Sandy. Flooded basements at two sites took out diesel fuel pumps, leaving them unable to refuel generators on higher levels. One of these was Datagram, which knocked out Buzzfeed and the Gawker network of sites. At 111 8th Avenue, some tenants lost power when Equinix briefly experienced generator problems." The NY Times has a running list of Sandy-related problems, including 5,700 more flight cancellations, 6 million people without power, rising water levels at a nuclear plant, official disaster declarations from President Obama, and a death toll of 38. On the upside, and despite the high water levels, the Nuclear Energy Institute was quick to point out that all 34 nuclear facilities in Sandy's path made it through without problems.
This discussion has been archived. No new comments can be posted.

New York Data Centers Battle Floods, Utility Outages

Comments Filter:
  • by AuMatar (183847) on Tuesday October 30, 2012 @02:14PM (#41821269)

    If we're really lucky, it'll take out all the high frequency traders systems for a few days and we can have an actual market without parasites.

    Nah, who am I kidding. If that actually happened they'd keep Wall Street closed.

    • by Revotron (1115029)
      HFT systems are located as close to the exchange's servers as they physically can be, because all the marketeers think it's bad juju to have a ping time above .01ms.

      Needless to say, if the HFT systems go down, then the market's exchange servers 2 feet away will probably be down, as well.

      I'm afraid this is a battle we just can't win.
      • Re: (Score:3, Informative)

        by jessehager (713802)

        Most of the traders try to get as close to the old Western Union Building at 60 Hudson St. as possible. If not inside it, in a building adjacent to it. That's the central network hub for the financial district.

      • by tlhIngan (30335) <.slashdot. .at. .worf.net.> on Tuesday October 30, 2012 @04:44PM (#41823963)

        HFT systems are located as close to the exchange's servers as they physically can be, because all the marketeers think it's bad juju to have a ping time above .01ms.

        Needless to say, if the HFT systems go down, then the market's exchange servers 2 feet away will probably be down, as well.

        If you're within the datacenter, there's a chance the trading computers are a bit further away from that actually. What happens is that the exchange actually ends up finding the longest cable they need to reach from the trading computer to the farthest rack, then they ensure that every cable from the trading computer any rack is that length. That way all the HFT traders who pay to be physically close in the data center aren't getting any advantage - there's just a huge coil of cable above the rack to ensure every rack gets the same latency. One could argue the coil of cable adds to latency by being an inductor, I suppose...

    • Nah, who am I kidding. If that actually happened they'd keep Wall Street closed.

      Well, now that you mention it...

    • by Laxori666 (748529)

      The stock market has been closed for a few days now.

    • by dintech (998802)
      I work at a major bank and we've lost a data center in New York Plaza. It's flooded and has no power. However, we have backup systems and everything is running out of alternate sites. This is the kind of situation disaster recovery is actually implemented for.
  • by blueforce (192332) <clannagael.gmail@com> on Tuesday October 30, 2012 @02:15PM (#41821287) Homepage Journal
    Why aren't there more datacenters in Kansas, Nebraska, North Dakota, etc.? Surely the threat from Tornados could be mitigated and the electrical infrastructure built out more cheaply than the losses due to coastal disasters, no?
    • by 54mc (897170) <samuelmcraven@ g m a i l.com> on Tuesday October 30, 2012 @02:18PM (#41821319)

      Why aren't there more datacenters in Kansas, Nebraska, North Dakota, etc.?

      If you're paying the premium to host in Manhattan, you're doing so because latency is a big deal to you.

      • by ahem (174666)

        To your point, if you were 1,450 miles away in the middle of Kansas, you'd have a 7.7 millisecond ping time just for speed-of-light latency if you ran redundant fiber from your DC directly to the exchange in Manhattan. Probably add some small amount of time for the network gear on each end of the connection.

        • by Shatrat (855151) on Tuesday October 30, 2012 @02:52PM (#41821795)

          5 microseconds per kilometer tends to be a pretty good approximation, depending on the transport gear.
          Things like FEC, EFEC, dispersion compensation modules (non-bragg grating type), frequent OEO regens can add up and make it worse.

          That would give you a ballpark of 11ms for a 1450 mile circuit.

        • by lennier (44736)

          To your point, if you were 1,450 miles away in the middle of Kansas, you'd have a 7.7 millisecond ping time just for speed-of-light latency if you ran redundant fiber from your DC directly to the exchange in Manhattan.

          Welcome to the Internet as the rest of the world sees it - and why we're less than enthused about "cloud" when all the big servers aren't even on our continent. ;)

      • I'm waiting for people to start asking the questions of "Should we bother rebuilding in New Orle...err...New York, since they're built so close the water, and are a disaster waiting to happen?"

        "What were they thinking building a city between two bodies of water near the ocean so close to sea level"?

        "Maybe we should just move the city to further inland, where its safer and we won't have to go through this over and over again...and keep wasting money"

        I know I've heard that before..hmmmm.

        Seems like the sho

        • AFAIK New York is not below sea level.

          • by GNious (953874)

            New York subway is, at least partially ... I wonder what that is looking like these days...

          • by cayenne8 (626475)

            AFAIK New York is not below sea level.

            No, but some areas are not very much higher than sea level, and combine that with being surrounded by water (Hudson River, etc) and exposed readily to coastal storms on the Atlantic...you're just asking for things like this to happen.

            It isn't like this area has never been hit by a big storm...its just they've been lucky the past few decades, but these type of things cycle.

            • You only consider it a weak storm (in your previous post) because of the failures of our current hurricane classification system.

              Sandy was a category 1 wind event, as proclaimed, but that wind event area was huge, possibly one of the largest hurricane wind events in the Atlantic ocean, which leads too...

              Sandy was at least a category 2 surge event, if not category 3. The direction of the storm, the storms gigantic wind field, and astronomical high tide lead to a surge event far more dangerous then the curren

        • by DerekLyons (302214) <fairwater@gmFREEBSDail.com minus bsd> on Tuesday October 30, 2012 @03:09PM (#41822123) Homepage

          But this was a weak storm, and did about normal damage as happens to places with a storm like this hitting a coastal area.

          I don't know where you live - but here on Planet Earth, nobody rational calls a storm with 100mph winds and an 11 foot storm surge, "weak". Not to mention, this storm was considerably more severe than is "normal" for that area.
           

          Sorry if it sounds like sour grapes...

          No, you don't sound like sour grapes - you sound like an ignorant jackass.
           
          Catch a clue.

          • by cayenne8 (626475)

            I don't know where you live - but here on Planet Earth, nobody rational calls a storm with 100mph winds and an 11 foot storm surge, "weak". Not to mention, this storm was considerably more severe than is "normal" for that area.

            I heard 90MPH gusts...and most of those readings were higher off than ground level....which when you get to the 20th floor of a building..the winds are MUCH higher than on the ground.

            Aside from some high gusts..sustained winds from what I saw last night....were only in the 50-60 mph

            • by bws111 (1216812) on Tuesday October 30, 2012 @04:12PM (#41823417)

              Getting several feet of snow in one day is not all that unusual in Buffalo. What do you think would happen to New Orleans if that happened there? An average cold winter's day where I live has a low temp of about 0F. Think New Orleans could take that for a few weeks at a time? The tidal change in the Bay of Fundy is something like 40ft - think New Orleans could take it? Sure, you get winds over 100MPH in New Orleans, but Mt Washington, NH recorded over 230MPH. Think you could take it?

              Comparing things like wind speed and storm surge and temperature between different regions is a fools game. What really matters is deviation from normal, and this was a very large deviation from normal. Yes, the storm surge was 'only' 13 feet, but the last time it was that high in NY was - unknown. The previous recorded max was in 1830 something, and this beat it. No, 70MPH winds are not that high in absolute terms, but tell that to the trees that couldn't take it (because normally they are only subjected to 50MPH winds).

              In short, get over yourself. The fact that you have experienced similar absolute numbers without devastation does not in any way mean that the same conditions are not devastating elsewhere, or that they shouldn't be devastating. No matter where you live, someone else is living with conditions that you would consider devastating.

              • If that happened to New Orleans, the Buffalonians would be laughing their asses off at the hicks from the sticks who don't know basic life skills like driving on ice. Seen it before. There would also be racist and classist comments.

                A lot of butthurt in this thread - maybe for once the shoe's on the other foot and the laughers are getting a dose of their own medicine.

              • by cayenne8 (626475)
                As someone else pointed out, much of my post...was to illustrate the difference in perception and attitude of those in the US...when a storm like this hits.

                There was a LOT of derision, and a a lot of people saying "we shouldn't rebuild New Orleans"...etc.

                Much of my post was to point out these attitudes in the past...and how much it can hurt when it happens to *you*....in the NE of the US.

                • by Rich0 (548339)

                  Well, I'm all for questioning the rebuilding of New Orleans - it is in a major hurricane zone and substantially below sea level.

                  NYC only rarely gets these problems and is at least above sea level to start, but I do think it would be wiser to not stick critical infrastructure underground even so. It is a really lousy place for the financial hub of the world. There is no reason that the servers can't be up in a mountain somewhere.

                  Oh, and in any case where there needs to be relocation I'm fine with reimburse

            • .I've been through more than my share of storms on the level of intensity that NYC got, and I do know what they are like.

              That means you know New Orleans - and you aptly demonstrate you know fuck all about New York. Monday's storm was well above the norm the New York. That's a stone cold "sure as the sun rises in the East" level fact.

              And...like I mentioned, that area better get used to these storms happening more regularly.

              In a universe where these things run in cycles, you'd have a point. We don'

            • by tlambert (566799)

              As for where I live...I live in the New Orleans area...I've been through more than my share of storms on the level of intensity that NYC got, and I do know what they are like.

              I'm pretty sure the New Orleans area building codes are not up to snuff for a Loma Prieta level earth quake.

              It's similarly unfair to expect the NYC codes to be up to snuff for a Sandy-like storm (although I'd expect them to be a lot closer than they were, given the 20 and 50 year storm history for the area, as you implied, I wouldn't expect them to be quite there still).

              Of course, the codes were not up to snuff in New Orleans for Katrina, but it was a worse disaster than it should have been, given that the

        • by Synerg1y (2169962)

          Hurricanes in NYC are very very rare regardless of time period. A lot of places build to withstand weather like this, there's actual building codes builders have to follow when building on the shore line of an area known to experience this kind of weather, this is why cities in the south east don't come crumbling down at the first gust from a hurricane. New York has a lot of old infrastructure, so it's not one of those. Insurance should cover a lot of the damage, but then they're just going to turn aroun

          • by watice (1347709)
            The infrastructure can withstand most natural disasters (and as you mentioned, time) because most of it is built from stone, concrete, & steel, as opposed to wood. This is fine for winds, hurricanes, even amateur terrorist attempts. However, most of the flooding damage happened not because of the rain downfall which was actually pretty little, but because of the surges happening in the coastal areas. A lot of the infrastructure also have below ground dwellings, cellars, and basements. The building codes
          • The damage is done to the foundations of the building and the electric appliances. I was told the construction sites of WTC got heavily damaged. Of course you can't have salt water in the foundations.

            Do you believe the stories from the nuclear power plants? http://www.nei.org/newsandevents/newsreleases/nuclear-energy-facilities-prove-resilience-during-hurricane-sandy/ [nei.org]

            New York: Indian Point 2—continued operating at 100 percent power Indian Point 3—manual safe shut down from 100 percent power o

        • by couchslug (175151)

          New York is valuable, and has the resources to promptly recover.

          Those who thought putting data centers in BASEMENTS asked for what they got. A broken water main (common enough) could have fucked them just as hard.

          In NOLA, only the container port is necessary, while slums BELOW, repeat BELOW, repeat REQUIRE LEVEE PROTECTION because they are BELOW sea level (NYC is NOT below seal level) are not.

          The __founders__ of NOLA didn't build below sea level, which is why the French Quarter wasn't destroyed by Katrina.

        • Considering a ass whipping GWB took over Katrina, the number of well-heeled liberal Democrats in the Empire State and the proximity to the elections I doubt that there will be any problems with FEMA. Obama is probably happier than a slinky on an escalator that there is something to distract people from the Bengazi debacle. I know as far as hurricanes go Sara is much as far as intensity goes, but she's a big sucker, we've had Gale and near gale [hint.fm] wind in Michigan for a couple days now.

        • "What were they thinking building a city between two bodies of water near the ocean so close to sea level"?

          That it would be a hell of a good seaport?

        • From where I stand (10,000 miles away), it doesn't look like a government vendetta against the south, it appears to me that both sides of politics are falling all over themselves to make sure a repeat of the Katrina response does not happen AGAIN. This temporary bipartisan approach is a good for the USA's political soul, especially so close to an election when you would expect them to be trying to heap balme on the opposition. It's a clear acknowlegement that the people affected by Katrina (such as yourself
        • One could argue they learned their lesson from Katrina....then again one could also argue that last time we have a (R) in office and this time we have a (D). I think both arguments might be at least partially true...
        • Yea, but those pesky EF4 & 5 twisters tend to make a mess of things quite frequently......unless they are underground....
      • by petsounds (593538)

        Makes sense for stock trading, but blogs like Gawker not so much.

    • by maxdread (1769548)

      You might be correct but keep in mind that there are other limitations as well. Cost and need for cooling, talent, price of utilities and infrastructure (not just utilities but potentially network related as well).

      However there has to be something behind the idea of data centers in the midwest, at the very least Iowa is home to a Google data center.

      • by firex726 (1188453)

        As I understand it, it's the availability of fiber. DC needs a fuck ton of fiber, and a big city is more likely to have it already present then some podunk town.

        • by bluefoxlucid (723572) on Tuesday October 30, 2012 @02:35PM (#41821569) Journal
          One bowl of Colon Blow has more fiber.
        • As I understand it, it's the availability of fiber. DC needs a fuck ton of fiber, and a big city is more likely to have it already present then some podunk town.

          Did you realize that fiber connects things? And as a result there is a "fuck-ton" of fiber not just in a big city but anywhere the major fiber trunks go, which is all across the country?

          What did you think all that traffic did when it left the city? Get printed out and put on a Greyhound?

          • by firex726 (1188453)

            Yes, because a single fiber cable in some backwater town with zero redundancy is a great place to build a DC.

            • Yes, because a single fiber cable in some backwater town with zero redundancy

              Why do you think there is zero redundancy?

              Even if you had only one carrier in a town that had a fiber run go through (very unlikely), if the cable gets cut on one side of town the fiber provider can just run all the traffic around the nationwide loop the other way until it gets where it needs to go - and believe me a cut will be fixed VERY quickly.

              A town where "a single fiber" dead ends is obviously not a good location, but there a

    • by OverlordQ (264228) on Tuesday October 30, 2012 @02:21PM (#41821351) Journal

      > Why aren't there more datacenters in Kansas, Nebraska, North Dakota, etc.?

      There are, you just dont hear about them as often because they generally dont have anything newsworthy to report about them.

      • > Why aren't there more datacenters in Kansas, Nebraska, North Dakota, etc.?

        There are, you just dont hear about them as often because they generally dont have anything newsworthy to report about them.

        This.

        We in the midwest stick our data centers underground for a reason; several reasons, actually.

    • by Shatrat (855151)

      The largest datacenter in the world is at 350 E Cermak Road in Chicago.

    • by sunking2 (521698)
      Because nobody wants to live there.
    • by Revotron (1115029) on Tuesday October 30, 2012 @02:30PM (#41821499)
      That's what Chicago is for. Far west enough to avoid most eastern seaboard troubles, far east enough to avoid the earthquakes, but central enough to provide good connectivity and ping times to both coasts.

      There are a few datacenters in Omaha, Nebraska, but they're either lights-out carrier-grade (Level3) or Fortune 500 warm-site backup grade. (CoSentry). They're also ungodly expensive because they're the only players in a 250 mile radius.

      Kansas City, MO has a good selection of datacenters for non-mission-critical systems, but most of the "data fortress" type places are built and run by the business that needs them.

      This is all completely ignoring the issue of latency, though - when you're doing financial transactions there's no better seat in the house than the heart of Wall Street. Every millisecond counts, I've been told.
      • by Rhys (96510)

        I don't know that either Chicago or Kansas City is really far enough from the new madrid fault to be considered safe. Things aren't built to CA earthquake standards around here, even something small(ish) by the time it reaches Chicago is going to do serious amounts of damage. Not to mention there are faults a lot closer that they know next to nothing about (see the higher risk zone west of Aurora at: http://earthquake.usgs.gov/earthquakes/states/illinois/hazards.php) so the margin of error on them could be

    • by wcrowe (94389)

      Maybe. Except the idiots would probably put their data centers in basements, in the downtown areas of cities like Kansas City and Omaha, where the nearby river could inevitably flood them.

      I would want my data center in Denver.

    • by Anonymous Coward on Tuesday October 30, 2012 @02:59PM (#41821921)

      I work for a daatcenter in Kansas. I can tell you first hand that there are a lot of datacenters around here (Kansas City metro). If you aren't doing high speed trading, then you have no reason to not have a datacenter out here. Power is cheaper, space is cheaper, cost of living is cheaper, you don't have to worry about hurricanes or snow, most DCs are tornado proof, and connectivity here is WAY better than people know (and growing faster than any other area in the country). If latency is your primary concern though, then close to the source is your only option.

    • by TWX (665546)

      Why aren't there more datacenters in Kansas, Nebraska, North Dakota, etc.? Surely the threat from Tornados could be mitigated and the electrical infrastructure built out more cheaply than the losses due to coastal disasters, no?

      Probably because, first, more transmission lines are required to establish a backbone to there instead of just putting them where the bulk of the users already are, second, high tech work generally pays well, and thus high tech workers want to live somewhere exciting or with a lot

    • Hahaha! good luck.. New York forgets there is a rest of the state, let alone anything past Philly..

    • Why aren't there more datacenters in Kansas, Nebraska, North Dakota, etc.?

      There are datacenters in the Chicago area, particularly for futures. Between these facilities and New York, the latest latency race is microwave relay towers [wired.com]. Any new stock exchange which decides to pop up and locate themselves in the Midwest has to fight the network effect to become relevant. In the long run (assuming they become popular enough to attract liquidity), they would only end up creating an additional opportunity for

  • by CajunArson (465943) on Tuesday October 30, 2012 @02:24PM (#41821391) Journal

    I was promised a NUKULAR OPOCALIPSE yesterday, and here we are with satellite images that don't even show the entire East Coast as a glowing radioactive wasteland.

    I think that this complete lack of NUKULAR meltdown is 100% absolute irrefutable proof of two key concepts:
    1. Capitalism is an abject failure and we need U.N. control of everything and everyone right now.
    2. NUKULAR power is obviously far too dangerous and should be banned right now before somebody doesn't get killed again JUST LIKE FUKUSHIMA.

    • by AmiMoJo (196126) *

      I'd also like to add that somehow Japan got through the summer peak demand without blackouts, despite only having three active reactors for the entire country. What happened to rolling blackouts and economic ruin that were predicted?

  • Disaster Plan Fail (Score:5, Insightful)

    by OverlordQ (264228) on Tuesday October 30, 2012 @02:25PM (#41821433) Journal

    [...] work hard to assess the situation and our recovery plans.

    How about not putting mission critical equipment susceptible to water damage in the one place all water will go.

    • by mrdogi (82975)

      Information Technology is not important enough in the eyes of so many higher-ups. Until an unrelated complete campus power outage, our primary data center was situated immediately under the Men's restroom, complete with leaky pipes.

      I'll just let that image soak in for a while...

  • It's like the old joke about trailer parks attracting tornadoes. There seems to be some sort of physical law that says that if a data center is going to be set up, it will be in a basement, and in a low-lying area.

  • by Bill Dimm (463823) on Tuesday October 30, 2012 @02:42PM (#41821649) Homepage

    Callcentric apparently had a single datacenter in NYC with no backup power generator [twitter.com]. Lots of discussion here [dslreports.com].

  • that flooding ruins backup generators, pumps, fuel storage. I hope that disaster mitigation plans are reviewed.

    I also hate people who judge negatively from hindsight, but disaster planning is about considering the most probable of the improbable. Flooding looms most threatening and probable of the improbable.

    Perhaps putting all the backup infrastructure on a higher floor makes it harder to maintain, access, and/ or protect from mischief/ terrorism. However plain old flooding seems to be an issue time and time again in disaster scenarios and really needs highest priority in disaster plans.

    • Eleven years ago, flooding from Tropical Storm Allison knocked out generators in the Texas Medical Center in Houston, the largest medical center in the world. Disaster planners should, by now, be well acquainted with the threat of flooding to their backup power. In any case, you are correct that they do not always have the flexibility to install such machinery on floors above ground level. The best that can be done, in many cases, is retrofit the generator rooms with seals and ventilation that can withst

      • And Ike did the same to UTMB. I believe part of the problem is that the people that control the money look at backup systems as something that is already costly, and they don't want to spend any more than they have to on them. They can't not have them if they want to be taken seriously, but they don't want to spend anything they don't have to on systems that, if all goes well, will never be used. Myopic, yes, but that's how a lot of management people think.
  • From my VPS provider (Score:3, Informative)

    by Anonymous Coward on Tuesday October 30, 2012 @05:35PM (#41824643)

    All Linux VPS located in New York City, NY VPS are currently offline. They are located in Internap's LGA6 facility in 111 8th Avenue.

    Please be advised that Internap's LGA11 facility is experiencing significant flooding in the sub-basement of the 75 Broad Street building as a result of Hurricane Sandy. The flooding has submerged and destroyed the site's diesel pumps and is preventing fuel from being pumped to the generators on the mezzanine level.

    Thankfully, our NYC server nodes are not directly located in LGA11's facility, but rather LGA6. The cause of this temporary outage is that LGA6 routes certain parts of the network's backbones through LGA11 which is currently offline as explained above. URPad's downtown NYC facility, located at 111 8th Avenue, is currently experiencing a network-only outage. The datacenter is not located in the storm surge zone, and is not suffering from any flooding. All URPad hardware and assets are safe and remain powered on. Engineers are aware of the network outage and all efforts will be made to restore network connectivity as soon as possible.

    Internap & URPad will continue to work hard to assess the situation and our recovery plans, and will communicate those plans as soon as possible. Thank you for your patience and understanding during this crisis. Please trust that we are doing everything we can to bring your services back online as soon as possible.

    • This reminds me of a story from www.thedailywtf.com New hire is shown the server room: redundant hardened servers, redundant RAID drives, redundant airconditioners, power supplies, phone lines to call people... Comes in a few weeks later, power went out, no airconditioning, servers are toast. why no phone call to the large number of people supposed ot get called ? no redundant power on the phones....... I mean, what kind of a moron has backup diesel in the BASEMENT !!!! ON AN ISLAND !!! Ok, no one thi
  • by cinnamon colbert (732724) on Tuesday October 30, 2012 @06:28PM (#41825211) Journal
    before we can tell all those people along the jersey and CT shores, ya wanna live on the ocean, get your own darn flood insurance, we the taxpayers are tired of picking up the tab so you can have a great view ?

Life would be so much easier if we could just look at the source code. -- Dave Olson

Working...