How CoreSite Survived Sandy 60
Nerval's Lobster writes "When Hurricane Sandy hit the East Coast, the combination of high winds, rain, and storm surges wreaked havoc on homes and businesses alike. With a data center on the Avenue of the Americas, CoreSite Realty escaped the worst the storm had to offer. But was it coincidence or careful planning? Slashdot sat down for an interview with Billie Haggard, CoreSite's senior vice president of data centers. He's responsible for the design, construction, maintenance, facilities staffing and uptime, reliability and energy efficiency of CoreSite's data centers. He described what it took to weather the worst weather to hit New York City in decades."
RTFA (Score:4, Interesting)
I found it interesting, especially the part about renting hotel rooms, only to have the staff sleep in cots on-site because the hotels had no electricity, food, or water.
First off, (Score:1)
Re: (Score:2)
First rule : don't build a data center at a location that gets week-long power outages.
Re:First off, (Score:4, Insightful)
First rule : don't build a data center at a location that gets week-long power outages.
Building down in that area is pretty much for one reason only: length of the fiber run to wall street. In a world of nanosecond trading, every mile you are away from Wall Street means that much more of an advantage someone else has over you. Is it extremely risky/expensive to build in these locations? Hell yes. Is it likely financially worth doing so? Hell yes. There are costs to doing business, this is just one you have to factor in and see if the overall risk/reward equation works out (I'm betting it does for these folks)
Re: (Score:2)
Actually, no. The HFT guys are colocated with NYSE over in Jersey where they've been since a bit after 9/11.
Re: (Score:2)
Name that place!
Northeast? Blizzards (and the rare hurricane)
Southeast? Hurricanes
Midwest? Tornadoes and floods
West? Earthquake and forest fires.
Re: (Score:2)
Phoenix [phoenixnap.com] isn't really at risk from any sort of common natural disaster (barring meteorites or something like that). The only real risk is heat, but datacenters have redundant air conditioners for a reason. /not associated with PhoenixNAP //hosts at a facility in Baltimore
Re: (Score:1)
That can happen just about anywhere. You'd have to be down the road from a power plant to have any real assurance of power availability. (something Apple and Facebook, and the state of NC, have done. 'tho NC built their backup facility in Forest City which is a good distance from 3 plants, but on top of the new single turbine coal plant in Cliffside [goo.gl].)
Re: (Score:2)
Excellent! (Score:3)
I'm sure the people in shelters and waiting in line for 6 hours to fill their own generators so they can keep their family warm at night are happy for you.
Re:Excellent! (Score:4, Interesting)
If the people in line had paid in advance like CoreSite did, they wouldn't have run out of fuel, either. Sheesh, these folks have foresight and you bash them because others didn't? And what does being in a shelter because your house was demoloshed have to do with fuel?
Re: (Score:2)
Re:Excellent! (Score:4, Interesting)
I'm not getting your point here. Are you arguing that they should have simply sent all their fuel to shelters and gone down for the entirety of the storm? In what way was it their job to supply the needs of the shelter?
Your argument is similar to parents who say that kids should always finish their food because people are starving in China. One has little to do with the other. I'm betting there's a named logical fallacy here but I can't identify it.
Re: (Score:2)
Couldn't find it? Here ya go:
http://yourlogicalfallacyis.com/appeal-to-emotion [yourlogicalfallacyis.com]
Re: (Score:2)
Re: (Score:2)
Pretty simple: it's the job of government to ensure the safety of its populace. If they needed the extra supplies for emergency use, that's why there's eminent domain and police seizures. But then, they were using diesel fuel anyway so most portable generators simply couldn't have used anything they provided. So, in summary, you have your head up your ass and are just looking for a way to appear superior.
Re:Excellent! (Score:4, Insightful)
It's not about whether they could do it, but how smug he comes across in patting himself on the back for having an excess of fuel while those around do without. Because his internet hosting is by far the most important thing in the world. And shelters would take people who still have a house, yet have no means of heating it.
I was struck by his smugness at having prepaid for fuel, but then a few paragraphs later he pointed out that the carrier that prepaid for a 4 hour generator delivery had their generator confiscated by the police. He was lucky they didn't need to refuel the confiscated generator by confiscating his fuel.
In a big disaster, fuel contracts mean very little - if the government decides that a hospital or police station (or the mayor's mistress's apartment building) needs the fuel more than you do, they will take it.
Re: (Score:1)
In a big disaster, fuel contracts mean very little - if the government decides that a hospital or police station (or the mayor's mistress's apartment building) needs the fuel more than you do, they will take it.
...Which is why they had *2* contracts for diesel delivery...
Re: (Score:2)
In a big disaster, fuel contracts mean very little - if the government decides that a hospital or police station (or the mayor's mistress's apartment building) needs the fuel more than you do, they will take it.
...Which is why they had *2* contracts for diesel delivery...
So the police can confiscate twice the fuel?
In a disaster where fuel is scarce enough to be confiscated by police, I don't think that having 2 contracts is really much better than one - if it was that easy to bring fuel into the city that both of your providers can supply it, then fuel wouldn't be scarce. Which, apparently was the case this time since they got their fuel delivery.
Re: (Score:1)
Re: (Score:2)
Yep, I've got three because I've been in the situation where neither of my of my contracted suppliers could fulfill and everyones business office was closed because it was late on a Friday. We ended up paying a stupid high trip charge to have someone bring fuel from 4 hours away. Since then I've gone to having three companies on contract and even then I realize that any single site is still vulnerable and so we have a warm DR site in a different calamity zone and on a different national grid (but still with
Re: (Score:2)
Still, don't blame this guy (or his company). Blame the local politicians who did not evacuate everywhere they should have and blame the people who ignored the evacuation orders. Blame over-restrictive anti-gouging laws which remove any financial incentive to rush fuel to where it is needed.
Blaming a guy who has regular delivery of diesel when everyone is lining up for gasoline is kind of non-nonsensical, anyway. Diesel would not have helped those people.
Re: (Score:2)
Re: (Score:3)
Generators for data-centers usually run on propane. It's not going to do the locals much good.
I've never seen a propane fueled datacenter class generator. I've seen big natural gas generators, but never propane. They may exist somewhere where the long lifetime of stored propane is a bonus (plus the easy cutover between natural gas and propane). Nearly all of the datacenter generators I've seen are diesel fueled - the generator referenced in the article is diesel fueled.
They likely still have natural gas service without interruption, but their furnaces wont run without electricity. They too could have purchased propane or natural gas generators for their homes for just a couple of hundred dollars.
You're not thinking the problem through if you're suggesting that apartment dwellers should just buy $200 propane powered generators
Re: (Score:1)
Unless you live in an area with no laws, I'm pretty sure there are very well defined rules (and LAWS) prohibiting apartments from having generators, and ESPECIALLY storing combustable fuels -- gasoline, propane, butane, hydrogen, etc. They are pretty serious fire risks. While they're designed not to explode, they still can, and even when they don't, they make a small fire a HUGE fire. Further adding to the no-no, apartments aren't readily equiped for alternate power connections, so you'd have drop cords
Re: (Score:2)
Generators for data-centers usually run on propane. It's not going to do the locals much good. They likely still have natural gas service without interruption, but their furnaces wont run without electricity. They too could have purchased propane or natural gas generators for their homes for just a couple of hundred dollars. But they didn't. Gasoline powered generators for emergencies are about the stupidest thing you can buy. It's expensive and runs out REALLY fast when there's any sort or service interruption.
Not sure about "usually", but my companies' data centers runs on diesel. The Wikipedia article on data centers [wikipedia.org] mentions diesel, not propane.
Re: (Score:1)
Most datacenter generators run on diesel. Onan doesn't even produce a NG or propane genset, Kohler's largest gas genset is 300kW, their largest diesel is 3,250kW. Cat's largest gas is 6,220kW, largest diesel is 17,460kW.
Re: (Score:2)
this was diesel fuel, not gasoline like you put in your car.
Re: (Score:2)
If I put anything but diesel in my car it'd wreck the engine.
If you put dyed, untaxed diesel in your car, you'd be exposing yourself to high fines. It's usually commercial trucks that get caught by this, but farmers get busted from time to time for using non-road fuel in their personal vehicles.
Re: (Score:2)
It is possible, that they should be thankful for the folks keeping this datacenter up. This one might have even been powering Red Cross's (or your favorite disaster recovery organization's) website or their communication systems. You never know how important the datacenter turned out to be.
Re: (Score:2)
Re: (Score:2)
Most small/personal (portable) generators use UNLEADED gasoline (or if you are really smart, you have one hardwired in to your property fed from propane or natural gas) whereas the big commercial ones use diesel, so even if they wanted to "share" it with the masses, it would be mostly useless...
Since small generators tend to not have catalytic converters, they should run just as well on LEADED gasoline (if you can find it).
Re: (Score:2)
Re: (Score:2)
True, but good luck as it's pretty much banned everywhere: http://en.wikipedia.org/wiki/Tetraethyllead [wikipedia.org]
Unless, of course, you find fuel at your local airport, where Leaded 100 Octane fuel is still legally sold. Not sure if it's legal to use it in a generator, but in a disaster you're probably more concerned with living than following the law.
Re:Excellent! (Score:5, Insightful)
"So eight hours in, we already had fuel trucks running. And every 24 hours, we had fuel, even though we didn’t need to." I'm sure the people in shelters and waiting in line for 6 hours to fill their own generators so they can keep their family warm at night are happy for you.
It is not the responsibility of CoreSite to provide fuel for everyone in New York City. Billie Haggard did his job well, and he deserves kudos for that.
Perhaps those people should have heeded the mandatory evacuation warnings and moved further away from the coast--100 miles inland would have made a world of difference to them, come Monday morning. Your house being without power doesn't matter if you're two hours away from it. There's no good reason for putting your children at risk by staying put when a hurricane is coming at you, and you had several days warning.
Re: (Score:3)
The people in shelters had ample time to prepare or prepare to evacuate.
The people with generators had plenty of time to fill their vehicles and fill reserve fuel containers for their gensets. (They had time to buy generators. Fuel cans are cheap enough.)
I always fill my vehicles before major storms, and always have reserve fuel for my small generator. (G.I. fuel cans and STA-BIL fuel additive work very well. Fuel small engines off your reserve or burn it in your vehicle to keep it fresh.)
The best tool in s
Datacenter near me... (Score:2)
Re: (Score:2)
Actually has 2 *weeks* of fuel supply (huge diesel tanks).
So it's probably not in an urban area - fuel codes severely restrict how much fuel you can store, and where you store it. I used to be a tenant in a building that gave up on installing a diesel backup generator since even after exorbitant permit and construction costs, they would have only been able to store about 12 hours of fuel on site. They ended up putting in a natural gas generator instead. Not quite as reliable in a disaster, but unlimited fuel as long as the gas supply is flowing.
Nice photo of the building (Score:4, Funny)
Re: (Score:2)
Looks like an acid trip.
as always it was the storm surge (Score:2)
if you look at the history of NYC area hurricanes Sandy was pretty weak. Category 1 with only 15mph forward velocity. we have had a lot of more powerful hurricanes hit NYC in the past. the 1938 storm was Cat 3 when it struck NYC and had a 70mph forward speed.
what happened was the it made landfall in southern NJ at high tide and a full moon and the winds pushed the water into NYC and land in general. This is what caused all the flooding. on the west side we had 5 foot deep flooding. on the east side there wa
Re: (Score:2)
we're a block from the hudson river and i don't think we have ever had 5 foot deep water in the streets. most of this was laid out almost 10 years ago and it's luck. i can see predicting a foot or two of flooding, but 5 feet in NYC is insane. i checked out the history of past hurricanes and didn't read anything about this kind of flooding here
Police confiscation (Score:2)
Can anyone elaborate on what radio system he was t
Re: (Score:2)
Re: (Score:2)
I was going to guess that it was a trunked radio system like many large law enforcement agencies use. Each site probably has access to a datalink of some sort (possibly IP based, possibly something more specialized, like a satelite link) that can carry traffic between sites, and a central controller at each site relays traffic to individual users. That would probably allow the sort of performance he describes.
Re: (Score:3)
Can anyone elaborate on what radio system he was talking about? I contributed to a DR plan several years ago, but my concerns about cell phone reliability were shot down.
Probably a regional UHF business radio network provider like Fisher Wireless [fisherwireless.com]. I think Fisher is a regional provider that only covers California and AZ, but I know national coverage is available. I think all of the interconnects between their radio sites run over UHF links, so they are completely independent of the cellular and wired network (but I'm not 100% sure about that).
Talk to any business radio provider and they can hook you up. Prices are reasonable, for around $20 - $50/month you can get unlimited
Language Weirdness (Score:2)
Slashdot is traditionally a bad place for language nazism, but the wording in this summary is just plain weird.
With a data center on the Avenue of the Americas, CoreSite Realty escaped the worst the storm had to offer.
When I first read this sentence I thought: a real estate company with a data center? Then I looked at the company website and discovered that they just call themselves "CoreSite"; no "Realty". So allow me to ask: Huh?
But was it coincidence or careful planning?
"Coincidence"? I think you mean "luck".
It sounds like they weren't really impacted (Score:1)
Just the fact that the fuel truck could even get to their facility says that they did not receive any significant weather.