New York Data Centers Battle Floods, Utility Outages 186
miller60 writes "At least three data center buildings in lower Manhattan are struggling with power problems amid widespread flooding and utility outages caused by Hurricane Sandy. Flooded basements at two sites took out diesel fuel pumps, leaving them unable to refuel generators on higher levels. One of these was Datagram, which knocked out Buzzfeed and the Gawker network of sites. At 111 8th Avenue, some tenants lost power when Equinix briefly experienced generator problems."
The NY Times has a running list of Sandy-related problems, including 5,700 more flight cancellations, 6 million people without power, rising water levels at a nuclear plant, official disaster declarations from President Obama, and a death toll of 38. On the upside, and despite the high water levels, the Nuclear Energy Institute was quick to point out that all 34 nuclear facilities in Sandy's path made it through without problems.
Look at the bright side (Score:5, Interesting)
If we're really lucky, it'll take out all the high frequency traders systems for a few days and we can have an actual market without parasites.
Nah, who am I kidding. If that actually happened they'd keep Wall Street closed.
Re: (Score:3)
Needless to say, if the HFT systems go down, then the market's exchange servers 2 feet away will probably be down, as well.
I'm afraid this is a battle we just can't win.
Re: (Score:3, Informative)
Most of the traders try to get as close to the old Western Union Building at 60 Hudson St. as possible. If not inside it, in a building adjacent to it. That's the central network hub for the financial district.
Re:Look at the bright side (Score:4, Interesting)
If you're within the datacenter, there's a chance the trading computers are a bit further away from that actually. What happens is that the exchange actually ends up finding the longest cable they need to reach from the trading computer to the farthest rack, then they ensure that every cable from the trading computer any rack is that length. That way all the HFT traders who pay to be physically close in the data center aren't getting any advantage - there's just a huge coil of cable above the rack to ensure every rack gets the same latency. One could argue the coil of cable adds to latency by being an inductor, I suppose...
Re: (Score:3)
Nah, who am I kidding. If that actually happened they'd keep Wall Street closed.
Well, now that you mention it...
Re: (Score:2)
The stock market has been closed for a few days now.
Re: (Score:2)
Re: (Score:2)
1. Yeah, how dare they exercise fiduciary responsibility and actually prepare by buying generators?
2. By the way, the pics you linked to clearly show other buildings with power as well.
3. Also in one of those links (the nymag one), the spokesman says they were prepared, they were also very lucky, and oh yes, we know a lot of people aren't as lucky as us so we're going to help out in Battery Park City. Come to location XYZ for charging stations and bottled water.
Where are the mid-American datacenters (Score:5, Interesting)
Re:Where are the mid-American datacenters (Score:5, Insightful)
Why aren't there more datacenters in Kansas, Nebraska, North Dakota, etc.?
If you're paying the premium to host in Manhattan, you're doing so because latency is a big deal to you.
Re: (Score:2)
To your point, if you were 1,450 miles away in the middle of Kansas, you'd have a 7.7 millisecond ping time just for speed-of-light latency if you ran redundant fiber from your DC directly to the exchange in Manhattan. Probably add some small amount of time for the network gear on each end of the connection.
Re:Where are the mid-American datacenters (Score:5, Informative)
5 microseconds per kilometer tends to be a pretty good approximation, depending on the transport gear.
Things like FEC, EFEC, dispersion compensation modules (non-bragg grating type), frequent OEO regens can add up and make it worse.
That would give you a ballpark of 11ms for a 1450 mile circuit.
Re: (Score:3)
To your point, if you were 1,450 miles away in the middle of Kansas, you'd have a 7.7 millisecond ping time just for speed-of-light latency if you ran redundant fiber from your DC directly to the exchange in Manhattan.
Welcome to the Internet as the rest of the world sees it - and why we're less than enthused about "cloud" when all the big servers aren't even on our continent. ;)
I'm waiting for the calls... (Score:2, Interesting)
"What were they thinking building a city between two bodies of water near the ocean so close to sea level"?
"Maybe we should just move the city to further inland, where its safer and we won't have to go through this over and over again...and keep wasting money"
I know I've heard that before..hmmmm.
Seems like the sho
Re: (Score:3)
AFAIK New York is not below sea level.
Re: (Score:2)
New York subway is, at least partially ... I wonder what that is looking like these days...
Re: (Score:2)
No, but some areas are not very much higher than sea level, and combine that with being surrounded by water (Hudson River, etc) and exposed readily to coastal storms on the Atlantic...you're just asking for things like this to happen.
It isn't like this area has never been hit by a big storm...its just they've been lucky the past few decades, but these type of things cycle.
Re: (Score:3)
You only consider it a weak storm (in your previous post) because of the failures of our current hurricane classification system.
Sandy was a category 1 wind event, as proclaimed, but that wind event area was huge, possibly one of the largest hurricane wind events in the Atlantic ocean, which leads too...
Sandy was at least a category 2 surge event, if not category 3. The direction of the storm, the storms gigantic wind field, and astronomical high tide lead to a surge event far more dangerous then the curren
Define irony... (Score:2)
This is a category 5 reductio absurdum event.
Re:I'm waiting for the calls... (Score:4, Insightful)
I don't know where you live - but here on Planet Earth, nobody rational calls a storm with 100mph winds and an 11 foot storm surge, "weak". Not to mention, this storm was considerably more severe than is "normal" for that area.
No, you don't sound like sour grapes - you sound like an ignorant jackass.
Catch a clue.
Re: (Score:2)
I heard 90MPH gusts...and most of those readings were higher off than ground level....which when you get to the 20th floor of a building..the winds are MUCH higher than on the ground.
Aside from some high gusts..sustained winds from what I saw last night....were only in the 50-60 mph
Re:I'm waiting for the calls... (Score:5, Insightful)
Getting several feet of snow in one day is not all that unusual in Buffalo. What do you think would happen to New Orleans if that happened there? An average cold winter's day where I live has a low temp of about 0F. Think New Orleans could take that for a few weeks at a time? The tidal change in the Bay of Fundy is something like 40ft - think New Orleans could take it? Sure, you get winds over 100MPH in New Orleans, but Mt Washington, NH recorded over 230MPH. Think you could take it?
Comparing things like wind speed and storm surge and temperature between different regions is a fools game. What really matters is deviation from normal, and this was a very large deviation from normal. Yes, the storm surge was 'only' 13 feet, but the last time it was that high in NY was - unknown. The previous recorded max was in 1830 something, and this beat it. No, 70MPH winds are not that high in absolute terms, but tell that to the trees that couldn't take it (because normally they are only subjected to 50MPH winds).
In short, get over yourself. The fact that you have experienced similar absolute numbers without devastation does not in any way mean that the same conditions are not devastating elsewhere, or that they shouldn't be devastating. No matter where you live, someone else is living with conditions that you would consider devastating.
Re: (Score:2)
If that happened to New Orleans, the Buffalonians would be laughing their asses off at the hicks from the sticks who don't know basic life skills like driving on ice. Seen it before. There would also be racist and classist comments.
A lot of butthurt in this thread - maybe for once the shoe's on the other foot and the laughers are getting a dose of their own medicine.
Re: (Score:2)
There was a LOT of derision, and a a lot of people saying "we shouldn't rebuild New Orleans"...etc.
Much of my post was to point out these attitudes in the past...and how much it can hurt when it happens to *you*....in the NE of the US.
Re: (Score:2)
Well, I'm all for questioning the rebuilding of New Orleans - it is in a major hurricane zone and substantially below sea level.
NYC only rarely gets these problems and is at least above sea level to start, but I do think it would be wiser to not stick critical infrastructure underground even so. It is a really lousy place for the financial hub of the world. There is no reason that the servers can't be up in a mountain somewhere.
Oh, and in any case where there needs to be relocation I'm fine with reimburse
Re: (Score:2)
That means you know New Orleans - and you aptly demonstrate you know fuck all about New York. Monday's storm was well above the norm the New York. That's a stone cold "sure as the sun rises in the East" level fact.
In a universe where these things run in cycles, you'd have a point. We don'
Re: (Score:2)
As for where I live...I live in the New Orleans area...I've been through more than my share of storms on the level of intensity that NYC got, and I do know what they are like.
I'm pretty sure the New Orleans area building codes are not up to snuff for a Loma Prieta level earth quake.
It's similarly unfair to expect the NYC codes to be up to snuff for a Sandy-like storm (although I'd expect them to be a lot closer than they were, given the 20 and 50 year storm history for the area, as you implied, I wouldn't expect them to be quite there still).
Of course, the codes were not up to snuff in New Orleans for Katrina, but it was a worse disaster than it should have been, given that the
Re: (Score:2)
Hurricanes in NYC are very very rare regardless of time period. A lot of places build to withstand weather like this, there's actual building codes builders have to follow when building on the shore line of an area known to experience this kind of weather, this is why cities in the south east don't come crumbling down at the first gust from a hurricane. New York has a lot of old infrastructure, so it's not one of those. Insurance should cover a lot of the damage, but then they're just going to turn aroun
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Do you believe the stories from the nuclear power plants? http://www.nei.org/newsandevents/newsreleases/nuclear-energy-facilities-prove-resilience-during-hurricane-sandy/ [nei.org]
New York: Indian Point 2—continued operating at 100 percent power Indian Point 3—manual safe shut down from 100 percent power o
Re: (Score:2)
New York is valuable, and has the resources to promptly recover.
Those who thought putting data centers in BASEMENTS asked for what they got. A broken water main (common enough) could have fucked them just as hard.
In NOLA, only the container port is necessary, while slums BELOW, repeat BELOW, repeat REQUIRE LEVEE PROTECTION because they are BELOW sea level (NYC is NOT below seal level) are not.
The __founders__ of NOLA didn't build below sea level, which is why the French Quarter wasn't destroyed by Katrina.
Re: (Score:2)
Considering a ass whipping GWB took over Katrina, the number of well-heeled liberal Democrats in the Empire State and the proximity to the elections I doubt that there will be any problems with FEMA. Obama is probably happier than a slinky on an escalator that there is something to distract people from the Bengazi debacle. I know as far as hurricanes go Sara is much as far as intensity goes, but she's a big sucker, we've had Gale and near gale [hint.fm] wind in Michigan for a couple days now.
Re: (Score:2)
"What were they thinking building a city between two bodies of water near the ocean so close to sea level"?
That it would be a hell of a good seaport?
Re: (Score:2)
Re: (Score:2)
here is the easiest compromise: (Score:2)
http://en.wikipedia.org/wiki/Manhattan,_Kansas [wikipedia.org]
Re: (Score:2)
Re: (Score:2)
Makes sense for stock trading, but blogs like Gawker not so much.
Re: (Score:3)
You might be correct but keep in mind that there are other limitations as well. Cost and need for cooling, talent, price of utilities and infrastructure (not just utilities but potentially network related as well).
However there has to be something behind the idea of data centers in the midwest, at the very least Iowa is home to a Google data center.
Re: (Score:2)
As I understand it, it's the availability of fiber. DC needs a fuck ton of fiber, and a big city is more likely to have it already present then some podunk town.
Re:Where are the mid-American datacenters (Score:5, Funny)
Whee do you think the fiber goes? (Score:2)
As I understand it, it's the availability of fiber. DC needs a fuck ton of fiber, and a big city is more likely to have it already present then some podunk town.
Did you realize that fiber connects things? And as a result there is a "fuck-ton" of fiber not just in a big city but anywhere the major fiber trunks go, which is all across the country?
What did you think all that traffic did when it left the city? Get printed out and put on a Greyhound?
Re: (Score:2)
Yes, because a single fiber cable in some backwater town with zero redundancy is a great place to build a DC.
Lots of redundancy (Score:3)
Yes, because a single fiber cable in some backwater town with zero redundancy
Why do you think there is zero redundancy?
Even if you had only one carrier in a town that had a fiber run go through (very unlikely), if the cable gets cut on one side of town the fiber provider can just run all the traffic around the nationwide loop the other way until it gets where it needs to go - and believe me a cut will be fixed VERY quickly.
A town where "a single fiber" dead ends is obviously not a good location, but there a
Re: (Score:2)
lol.. No, they just like good BBQ pork.
Re:Where are the mid-American datacenters (Score:5, Informative)
> Why aren't there more datacenters in Kansas, Nebraska, North Dakota, etc.?
There are, you just dont hear about them as often because they generally dont have anything newsworthy to report about them.
Re: (Score:2)
> Why aren't there more datacenters in Kansas, Nebraska, North Dakota, etc.?
There are, you just dont hear about them as often because they generally dont have anything newsworthy to report about them.
This.
We in the midwest stick our data centers underground for a reason; several reasons, actually.
Re: (Score:3)
The largest datacenter in the world is at 350 E Cermak Road in Chicago.
Re: (Score:3)
I couldn't say because the network I work on doesn't go that far west, but 350 E Cermak is the center of the Internet, if it has one.
Re: (Score:2)
Re:Where are the mid-American datacenters (Score:4, Informative)
There are a few datacenters in Omaha, Nebraska, but they're either lights-out carrier-grade (Level3) or Fortune 500 warm-site backup grade. (CoSentry). They're also ungodly expensive because they're the only players in a 250 mile radius.
Kansas City, MO has a good selection of datacenters for non-mission-critical systems, but most of the "data fortress" type places are built and run by the business that needs them.
This is all completely ignoring the issue of latency, though - when you're doing financial transactions there's no better seat in the house than the heart of Wall Street. Every millisecond counts, I've been told.
Re: (Score:2)
I don't know that either Chicago or Kansas City is really far enough from the new madrid fault to be considered safe. Things aren't built to CA earthquake standards around here, even something small(ish) by the time it reaches Chicago is going to do serious amounts of damage. Not to mention there are faults a lot closer that they know next to nothing about (see the higher risk zone west of Aurora at: http://earthquake.usgs.gov/earthquakes/states/illinois/hazards.php) so the margin of error on them could be
Re: (Score:2)
Maybe. Except the idiots would probably put their data centers in basements, in the downtown areas of cities like Kansas City and Omaha, where the nearby river could inevitably flood them.
I would want my data center in Denver.
Re:Where are the mid-American datacenters (Score:4, Informative)
I work for a daatcenter in Kansas. I can tell you first hand that there are a lot of datacenters around here (Kansas City metro). If you aren't doing high speed trading, then you have no reason to not have a datacenter out here. Power is cheaper, space is cheaper, cost of living is cheaper, you don't have to worry about hurricanes or snow, most DCs are tornado proof, and connectivity here is WAY better than people know (and growing faster than any other area in the country). If latency is your primary concern though, then close to the source is your only option.
Re: (Score:2)
Probably because, first, more transmission lines are required to establish a backbone to there instead of just putting them where the bulk of the users already are, second, high tech work generally pays well, and thus high tech workers want to live somewhere exciting or with a lot
Re: (Score:2)
Hahaha! good luck.. New York forgets there is a rest of the state, let alone anything past Philly..
Re: (Score:2)
There are datacenters in the Chicago area, particularly for futures. Between these facilities and New York, the latest latency race is microwave relay towers [wired.com]. Any new stock exchange which decides to pop up and locate themselves in the Midwest has to fight the network effect to become relevant. In the long run (assuming they become popular enough to attract liquidity), they would only end up creating an additional opportunity for
Re: (Score:2)
But you have to have someone live in Utah.
Re: (Score:2)
WHERE ARE THE MELTDOWNS!!?!?!?! (Score:4, Funny)
I was promised a NUKULAR OPOCALIPSE yesterday, and here we are with satellite images that don't even show the entire East Coast as a glowing radioactive wasteland.
I think that this complete lack of NUKULAR meltdown is 100% absolute irrefutable proof of two key concepts:
1. Capitalism is an abject failure and we need U.N. control of everything and everyone right now.
2. NUKULAR power is obviously far too dangerous and should be banned right now before somebody doesn't get killed again JUST LIKE FUKUSHIMA.
Re: (Score:2)
I'd also like to add that somehow Japan got through the summer peak demand without blackouts, despite only having three active reactors for the entire country. What happened to rolling blackouts and economic ruin that were predicted?
Disaster Plan Fail (Score:5, Insightful)
How about not putting mission critical equipment susceptible to water damage in the one place all water will go.
Re: (Score:3)
Information Technology is not important enough in the eyes of so many higher-ups. Until an unrelated complete campus power outage, our primary data center was situated immediately under the Men's restroom, complete with leaky pipes.
I'll just let that image soak in for a while...
Re: (Score:2)
Re: (Score:2)
You wouldn't need to put the fuel up there. They originally pump it out of the ground over much deeper distances then most skyscrapers would dream of being tall. Pump it up to the generator, have a containment pad or tank, if a fire breaks out, dump the fuel in the line down to the containment vessel.
Or better yet, don't bother putting either on the roof and just make the room housing it in the basement waterproof. then you can lock down the lower level when there is expected trouble like this and not worry
Re: (Score:2)
It's not impossible either. Look a ships or submarines. You do not need to completely enclose the generators, just keep the below any realistic waterline areas sealed up until above the expected waterline.
Re: (Score:2)
It wasn't the generators that failed, it was the fuel pumps to the generators that failed. The pumps were in the basement. The generators were on ground floor.
Face palm (Score:2)
It's like the old joke about trailer parks attracting tornadoes. There seems to be some sort of physical law that says that if a data center is going to be set up, it will be in a basement, and in a low-lying area.
Callcentric VoIP service also down (Score:3)
Callcentric apparently had a single datacenter in NYC with no backup power generator [twitter.com]. Lots of discussion here [dslreports.com].
Fukushima and Sandy show (Score:4, Insightful)
that flooding ruins backup generators, pumps, fuel storage. I hope that disaster mitigation plans are reviewed.
I also hate people who judge negatively from hindsight, but disaster planning is about considering the most probable of the improbable. Flooding looms most threatening and probable of the improbable.
Perhaps putting all the backup infrastructure on a higher floor makes it harder to maintain, access, and/ or protect from mischief/ terrorism. However plain old flooding seems to be an issue time and time again in disaster scenarios and really needs highest priority in disaster plans.
Re: (Score:3)
Eleven years ago, flooding from Tropical Storm Allison knocked out generators in the Texas Medical Center in Houston, the largest medical center in the world. Disaster planners should, by now, be well acquainted with the threat of flooding to their backup power. In any case, you are correct that they do not always have the flexibility to install such machinery on floors above ground level. The best that can be done, in many cases, is retrofit the generator rooms with seals and ventilation that can withst
Re: (Score:2)
From my VPS provider (Score:3, Informative)
All Linux VPS located in New York City, NY VPS are currently offline. They are located in Internap's LGA6 facility in 111 8th Avenue.
Please be advised that Internap's LGA11 facility is experiencing significant flooding in the sub-basement of the 75 Broad Street building as a result of Hurricane Sandy. The flooding has submerged and destroyed the site's diesel pumps and is preventing fuel from being pumped to the generators on the mezzanine level.
Thankfully, our NYC server nodes are not directly located in LGA11's facility, but rather LGA6. The cause of this temporary outage is that LGA6 routes certain parts of the network's backbones through LGA11 which is currently offline as explained above. URPad's downtown NYC facility, located at 111 8th Avenue, is currently experiencing a network-only outage. The datacenter is not located in the storm surge zone, and is not suffering from any flooding. All URPad hardware and assets are safe and remain powered on. Engineers are aware of the network outage and all efforts will be made to restore network connectivity as soon as possible.
Internap & URPad will continue to work hard to assess the situation and our recovery plans, and will communicate those plans as soon as possible. Thank you for your patience and understanding during this crisis. Please trust that we are doing everything we can to bring your services back online as soon as possible.
Re: (Score:3)
how many days have to go by (Score:3)
Re: (Score:2)
Re: (Score:3)
They were idiots for going out to sea. Even if it was marginally safer for the boat to be at sea than in port for the storm, it was safer for the crew to be in port, or in hotel rooms 100 miles inland. Those people died due to a combination of greedy ownership that valued the ship more than the crew's safety and their own stupidity in not refusing. All this for a reproduction used in movie shoots. A senseless waste.
Re: (Score:3)
Unless you misjudge the size, speed and track of the storm. Now add in a generator that craps out, and cannot pump out the bilge anymore.
When they left port, the projected storm track was much different than what actually happened.
Re:No big deal. It was a cat1 storm (Score:4, Insightful)
The navy has much bigger, more modern ships. There's also national security risks if the navy boats are damaged. No such problems for a movie prop.
And like I said- it may be safer for the boat. It isn't safer for the crew. This wasn't a surprise storm, this had been forecast for a week. Tie up the boat, stay in a hotel (preferably about 100 miles or so inland), and then repair the boat as needed. The health and safety of the crew is far more important than the ship. The owners deserve to be sued into oblivion for even asking. It is NOT acceptable to risk 16 lives to save money on repairs. Hell, take it to a dry dock if you're that scared.
Re: (Score:2)
Re: (Score:3)
1 dead, 1 missing, and 13 came damn close and needed rescue choppers during a hurricane, endangering both themselves and the people who had to save them. Its inexcusable and criminally negligent. It probably doesn't deserve quite the amount of media attention its getting, but it was amazingly stupid.
Re: (Score:2)
They were idiots for going out to sea.
Are you saying every boat that goes out to sea is an idiot? They didn't go out to sea to get away from the storm, the were already out to sea. They were travelling from one side of the country to the other. Apparently they decided to go around the storm, when they had equipment malfunctions.
They didn't jump in their boat, and say "we'll be safer at sea"
Re:No big deal. It was a cat1 storm (Score:5, Insightful)
This is as strong as a case forever to stop paying attention to the media.
Sounds like you already stopped paying attention. Had you even checked the one link in the article, or even read the summary, you'd know it was a catastophy, and New York will not function "perfectly tomorrow like nothing ever happened."
Schools closed. Subways closed 4-5 days. 38 people dead. Market closed. Fire in Queens destroys dozens of homes. Power outages for millions. 7% of the US population in fact without power. Tunnels flooded (subway and car). NYU Tisch hospital evacuated due to flood related generator failures, including premature babies on ventilators. Just a small summary, of just one city.
I'm a former NYer. Have spoken to many friends and family. None expect normal life tomorrow. Some have considerable property damage. None lost a life, thankfully. I live in South Florida, incidentally, and rather well understand how damaging hurricanes can be. Wilma damaged my car, and cancelled my wedding day as the roof caved in on the place we were getting married in that Friday.
This is a rare strong case to NOT stop paying attention to the media.
Re: (Score:3, Interesting)
Everyone depending on mass transportation...a little flooding and the city comes to a stand still?
Hell, we get hit with Cat 1 level storms all the time in the Gulf south area...it is a major PITA, and I do feel for people that got flooded, it is horrible...but it *IS* part of living near the coast of an ocean.
I hope everyone had flood insurance, it is dirt cheap.
But really aside from the expected shoreline damage..this wasn't t
Re:No big deal. It was a cat1 storm (Score:4, Insightful)
"Be better prepared" - than what? The storm surge that flooded lower Manhattan beat the previous record by 2 feet, and that record was set almost 200 years ago. The fact that there were only 38 people killed in the entire region shows just how well prepared they were. Nobody was drowned in the subways, because they stopped the subways before they got flooded. Nobody was stuck in elevators, because they turned off the elevators before the power was shut down.
What is it with all these people saying 'it was not that bad a storm'. It was that bad a storm for the area. It was record flooding. From what I understand, Category 5 hurricanes are 'not that bad a storm' compared to the storms on Jupiter - pretty meaningless comparison, isn't it?
Re: (Score:2)
The bottom line is, while tragic, the damage is pretty much what is expected from a Category One Storm. Could have been a lot worse. There are four more categories of stronger storms to contend with . .
Lots of rain, storm surge, high winds for a while, fire, flooding, no power for several days to months ( YES months. Some places here after IKE had no power for MONTHS ), the usual. At least it didn't spaw
Re: (Score:3)
will be several days to recover, but we have other Data Centers in Mass, NY and Atlanta and Denver which can still cover that half of the country....
-I'm just sayin'
Re: (Score:2)
Re:No big deal. It was a cat1 storm (Score:4, Informative)
No 'might' about it.
Death rate of NYC is around 1,000/week. or 140 per day.
Re: (Score:2)
Re: (Score:2)
38 dead from the storm, not total.
Re:Oblig. XKCD (Score:4, Insightful)
Re: (Score:3)
Re: (Score:3)
Weird, it works just fine for me.
Re:Oblig. XKCD (Score:5, Funny)
http://xkcd.com/404/
Re: (Score:2)
I'm not even white, I'm off-white. We're a new race, we will prevail! - Mitch Hedberg
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Think I might go build a mountain city in Scotland
Just to warn you then, the weather here has been more shite than usual lately. Maybe better than other places in the world but the recent decade has been much worse than the previous three. I can't comment on weather prior to that on account of being too young to remember and ultimately not even existing.