Forgot your password?
typodupeerror
Hardware Technology

How CoreSite Survived Sandy 60

Posted by Soulskill
from the lots-of-emergency-flotation-devices dept.
Nerval's Lobster writes "When Hurricane Sandy hit the East Coast, the combination of high winds, rain, and storm surges wreaked havoc on homes and businesses alike. With a data center on the Avenue of the Americas, CoreSite Realty escaped the worst the storm had to offer. But was it coincidence or careful planning? Slashdot sat down for an interview with Billie Haggard, CoreSite's senior vice president of data centers. He's responsible for the design, construction, maintenance, facilities staffing and uptime, reliability and energy efficiency of CoreSite's data centers. He described what it took to weather the worst weather to hit New York City in decades."
This discussion has been archived. No new comments can be posted.

How CoreSite Survived Sandy

Comments Filter:
  • RTFA (Score:4, Interesting)

    by mcgrew (92797) * on Wednesday November 07, 2012 @03:43PM (#41910969) Homepage Journal

    I found it interesting, especially the part about renting hotel rooms, only to have the staff sleep in cots on-site because the hotels had no electricity, food, or water.

  • It doesn't sound like they had it quite so bad. Yes, they planned for most every contingency but TFA did say they only had a little flooding in the basement. They did have to deal with short power outages (my power has gone out for weeks at a time) but I'd like to think that most larger corporations would have current Continuation of Business plans. These guys did and kudos for that.
    • by oneiros27 (46144)

      First rule : don't build a data center at a location that gets week-long power outages.

      • Re:First off, (Score:4, Insightful)

        by volxdragon (1297215) on Wednesday November 07, 2012 @04:19PM (#41911321)

        First rule : don't build a data center at a location that gets week-long power outages.

        Building down in that area is pretty much for one reason only: length of the fiber run to wall street. In a world of nanosecond trading, every mile you are away from Wall Street means that much more of an advantage someone else has over you. Is it extremely risky/expensive to build in these locations? Hell yes. Is it likely financially worth doing so? Hell yes. There are costs to doing business, this is just one you have to factor in and see if the overall risk/reward equation works out (I'm betting it does for these folks)

        • by afidel (530433)

          Actually, no. The HFT guys are colocated with NYSE over in Jersey where they've been since a bit after 9/11.

      • First rule : don't build a data center at a location that gets week-long power outages.

        Name that place!
        Northeast? Blizzards (and the rare hurricane)
        Southeast? Hurricanes
        Midwest? Tornadoes and floods
        West? Earthquake and forest fires.
        • by heypete (60671)

          Phoenix [phoenixnap.com] isn't really at risk from any sort of common natural disaster (barring meteorites or something like that). The only real risk is heat, but datacenters have redundant air conditioners for a reason. /not associated with PhoenixNAP //hosts at a facility in Baltimore

      • by Cramer (69040)

        That can happen just about anywhere. You'd have to be down the road from a power plant to have any real assurance of power availability. (something Apple and Facebook, and the state of NC, have done. 'tho NC built their backup facility in Forest City which is a good distance from 3 plants, but on top of the new single turbine coal plant in Cliffside [goo.gl].)

    • by assantisz (881107)
      Yes, the power in that part of town was out for less than a week. My employer also houses two datacenters in the same building and both floors were on generator power all that time w/o incident. So, the only useful thing you can take out of CoreSite's story is "have backup power on standby". You'll never know when it is needed. Or, do it like what NYU did, have your own power plant and hook up your most critical sites to it (http://green.blogs.nytimes.com/2012/11/05/how-n-y-u-stayed-partly-warm-and-lighted/
  • by sunking2 (521698) on Wednesday November 07, 2012 @03:47PM (#41911019)
    "So eight hours in, we already had fuel trucks running. And every 24 hours, we had fuel, even though we didn’t need to."
    I'm sure the people in shelters and waiting in line for 6 hours to fill their own generators so they can keep their family warm at night are happy for you.
    • Re:Excellent! (Score:4, Interesting)

      by mcgrew (92797) * on Wednesday November 07, 2012 @03:53PM (#41911085) Homepage Journal

      If the people in line had paid in advance like CoreSite did, they wouldn't have run out of fuel, either. Sheesh, these folks have foresight and you bash them because others didn't? And what does being in a shelter because your house was demoloshed have to do with fuel?

      • by sunking2 (521698)
        It's not about whether they could do it, but how smug he comes across in patting himself on the back for having an excess of fuel while those around do without. Because his internet hosting is by far the most important thing in the world. And shelters would take people who still have a house, yet have no means of heating it.
        • Re:Excellent! (Score:4, Interesting)

          by TheSpoom (715771) <[ten.00mrebu] [ta] [todhsals]> on Wednesday November 07, 2012 @04:13PM (#41911275) Homepage Journal

          I'm not getting your point here. Are you arguing that they should have simply sent all their fuel to shelters and gone down for the entirety of the storm? In what way was it their job to supply the needs of the shelter?

          Your argument is similar to parents who say that kids should always finish their food because people are starving in China. One has little to do with the other. I'm betting there's a named logical fallacy here but I can't identify it.

          • by evilviper (135110)

            Your argument is similar to parents who say that kids should always finish their food because people are starving in China. One has little to do with the other. I'm betting there's a named logical fallacy here but I can't identify it.

            Couldn't find it? Here ya go:

            http://yourlogicalfallacyis.com/appeal-to-emotion [yourlogicalfallacyis.com]

          • I watched a show the other night that demonstrated animals including monkeys, dogs and even mice have empathy for each-other. In other words a mouse will release another mouse who is trapped and then share its food. What do it say about you that you think it is great they hoarded resources while others suffered?
            • by TheSpoom (715771)

              Pretty simple: it's the job of government to ensure the safety of its populace. If they needed the extra supplies for emergency use, that's why there's eminent domain and police seizures. But then, they were using diesel fuel anyway so most portable generators simply couldn't have used anything they provided. So, in summary, you have your head up your ass and are just looking for a way to appear superior.

        • Re:Excellent! (Score:4, Insightful)

          by hawguy (1600213) on Wednesday November 07, 2012 @04:17PM (#41911307)

          It's not about whether they could do it, but how smug he comes across in patting himself on the back for having an excess of fuel while those around do without. Because his internet hosting is by far the most important thing in the world. And shelters would take people who still have a house, yet have no means of heating it.

          I was struck by his smugness at having prepaid for fuel, but then a few paragraphs later he pointed out that the carrier that prepaid for a 4 hour generator delivery had their generator confiscated by the police. He was lucky they didn't need to refuel the confiscated generator by confiscating his fuel.

          In a big disaster, fuel contracts mean very little - if the government decides that a hospital or police station (or the mayor's mistress's apartment building) needs the fuel more than you do, they will take it.

          • by Hagaric (2591241)

            In a big disaster, fuel contracts mean very little - if the government decides that a hospital or police station (or the mayor's mistress's apartment building) needs the fuel more than you do, they will take it.

            ...Which is why they had *2* contracts for diesel delivery...

            • by hawguy (1600213)

              In a big disaster, fuel contracts mean very little - if the government decides that a hospital or police station (or the mayor's mistress's apartment building) needs the fuel more than you do, they will take it.

              ...Which is why they had *2* contracts for diesel delivery...

              So the police can confiscate twice the fuel?

              In a disaster where fuel is scarce enough to be confiscated by police, I don't think that having 2 contracts is really much better than one - if it was that easy to bring fuel into the city that both of your providers can supply it, then fuel wouldn't be scarce. Which, apparently was the case this time since they got their fuel delivery.

              • by Hagaric (2591241)
                Well they had 3 days fuel to start with & 2x 24-hourly delivery contracts, so unless the emergency services were requisitioning the fuel at source, or the power outage lasted more than 3 days, and assuming that a single delivery could fill the tanks, only 1 delivery in 6 has to make it.
            • by afidel (530433)

              Yep, I've got three because I've been in the situation where neither of my of my contracted suppliers could fulfill and everyones business office was closed because it was late on a Friday. We ended up paying a stupid high trip charge to have someone bring fuel from 4 hours away. Since then I've gone to having three companies on contract and even then I realize that any single site is still vulnerable and so we have a warm DR site in a different calamity zone and on a different national grid (but still with

        • by MightyYar (622222)

          Still, don't blame this guy (or his company). Blame the local politicians who did not evacuate everywhere they should have and blame the people who ignored the evacuation orders. Blame over-restrictive anti-gouging laws which remove any financial incentive to rush fuel to where it is needed.

          Blaming a guy who has regular delivery of diesel when everyone is lining up for gasoline is kind of non-nonsensical, anyway. Diesel would not have helped those people.

        • Generators for data-centers usually run on propane. It's not going to do the locals much good. They likely still have natural gas service without interruption, but their furnaces wont run without electricity. They too could have purchased propane or natural gas generators for their homes for just a couple of hundred dollars. But they didn't. Gasoline powered generators for emergencies are about the stupidest thing you can buy. It's expensive and runs out REALLY fast when there's any sort or service interrup
          • by hawguy (1600213)

            Generators for data-centers usually run on propane. It's not going to do the locals much good.

            I've never seen a propane fueled datacenter class generator. I've seen big natural gas generators, but never propane. They may exist somewhere where the long lifetime of stored propane is a bonus (plus the easy cutover between natural gas and propane). Nearly all of the datacenter generators I've seen are diesel fueled - the generator referenced in the article is diesel fueled.

            They likely still have natural gas service without interruption, but their furnaces wont run without electricity. They too could have purchased propane or natural gas generators for their homes for just a couple of hundred dollars.

            You're not thinking the problem through if you're suggesting that apartment dwellers should just buy $200 propane powered generators

            • by Cramer (69040)

              Unless you live in an area with no laws, I'm pretty sure there are very well defined rules (and LAWS) prohibiting apartments from having generators, and ESPECIALLY storing combustable fuels -- gasoline, propane, butane, hydrogen, etc. They are pretty serious fire risks. While they're designed not to explode, they still can, and even when they don't, they make a small fire a HUGE fire. Further adding to the no-no, apartments aren't readily equiped for alternate power connections, so you'd have drop cords

          • by egamma (572162)

            Generators for data-centers usually run on propane. It's not going to do the locals much good. They likely still have natural gas service without interruption, but their furnaces wont run without electricity. They too could have purchased propane or natural gas generators for their homes for just a couple of hundred dollars. But they didn't. Gasoline powered generators for emergencies are about the stupidest thing you can buy. It's expensive and runs out REALLY fast when there's any sort or service interruption.

            Not sure about "usually", but my companies' data centers runs on diesel. The Wikipedia article on data centers [wikipedia.org] mentions diesel, not propane.

          • Most datacenter generators run on diesel. Onan doesn't even produce a NG or propane genset, Kohler's largest gas genset is 300kW, their largest diesel is 3,250kW. Cat's largest gas is 6,220kW, largest diesel is 17,460kW.

    • by alen (225700)

      this was diesel fuel, not gasoline like you put in your car.

    • It is possible, that they should be thankful for the folks keeping this datacenter up. This one might have even been powering Red Cross's (or your favorite disaster recovery organization's) website or their communication systems. You never know how important the datacenter turned out to be.

    • Most small/personal (portable) generators use UNLEADED gasoline (or if you are really smart, you have one hardwired in to your property fed from propane or natural gas) whereas the big commercial ones use diesel, so even if they wanted to "share" it with the masses, it would be mostly useless...
      • by hawguy (1600213)

        Most small/personal (portable) generators use UNLEADED gasoline (or if you are really smart, you have one hardwired in to your property fed from propane or natural gas) whereas the big commercial ones use diesel, so even if they wanted to "share" it with the masses, it would be mostly useless...

        Since small generators tend to not have catalytic converters, they should run just as well on LEADED gasoline (if you can find it).

    • Re:Excellent! (Score:5, Insightful)

      by egamma (572162) <egamma&gmail,com> on Wednesday November 07, 2012 @04:55PM (#41911705)

      "So eight hours in, we already had fuel trucks running. And every 24 hours, we had fuel, even though we didn’t need to." I'm sure the people in shelters and waiting in line for 6 hours to fill their own generators so they can keep their family warm at night are happy for you.

      It is not the responsibility of CoreSite to provide fuel for everyone in New York City. Billie Haggard did his job well, and he deserves kudos for that.

      Perhaps those people should have heeded the mandatory evacuation warnings and moved further away from the coast--100 miles inland would have made a world of difference to them, come Monday morning. Your house being without power doesn't matter if you're two hours away from it. There's no good reason for putting your children at risk by staying put when a hurricane is coming at you, and you had several days warning.

    • by couchslug (175151)

      The people in shelters had ample time to prepare or prepare to evacuate.

      The people with generators had plenty of time to fill their vehicles and fill reserve fuel containers for their gensets. (They had time to buy generators. Fuel cans are cheap enough.)

      I always fill my vehicles before major storms, and always have reserve fuel for my small generator. (G.I. fuel cans and STA-BIL fuel additive work very well. Fuel small engines off your reserve or burn it in your vehicle to keep it fresh.)

      The best tool in s

  • Actually has 2 *weeks* of fuel supply (huge diesel tanks).
    • by hawguy (1600213)

      Actually has 2 *weeks* of fuel supply (huge diesel tanks).

      So it's probably not in an urban area - fuel codes severely restrict how much fuel you can store, and where you store it. I used to be a tenant in a building that gave up on installing a diesel backup generator since even after exorbitant permit and construction costs, they would have only been able to store about 12 hours of fuel on site. They ended up putting in a natural gas generator instead. Not quite as reliable in a disaster, but unlimited fuel as long as the gas supply is flowing.

  • by timeOday (582209) on Wednesday November 07, 2012 @04:05PM (#41911201)
    I guess Kodak is still selling $5 disposable cameras after all.
  • if you look at the history of NYC area hurricanes Sandy was pretty weak. Category 1 with only 15mph forward velocity. we have had a lot of more powerful hurricanes hit NYC in the past. the 1938 storm was Cat 3 when it struck NYC and had a 70mph forward speed.

    what happened was the it made landfall in southern NJ at high tide and a full moon and the winds pushed the water into NYC and land in general. This is what caused all the flooding. on the west side we had 5 foot deep flooding. on the east side there wa

  • Awesome interview. I loved this part: "The other thing is making sure our customers understand that temporary systems are not good in situations like this. One of our major carriers, their backup system was to bring up a rollup generator. And from what I understand, they paid to have this generator there in four hours, and when they had this generator up, the police confiscated it for emergency use. So their backup generator wasn’t there any more."

    Can anyone elaborate on what radio system he was t
    • Nationwide service radio system? Likely a SAT-phones. Not cheap to use, but if you have a real major emergency (* except for strong X-class solar flares/CMEs in the direction of Earth :), they will usually get thru when all other forms of communication are failing...
    • I was going to guess that it was a trunked radio system like many large law enforcement agencies use. Each site probably has access to a datalink of some sort (possibly IP based, possibly something more specialized, like a satelite link) that can carry traffic between sites, and a central controller at each site relays traffic to individual users. That would probably allow the sort of performance he describes.

    • by hawguy (1600213)

      Can anyone elaborate on what radio system he was talking about? I contributed to a DR plan several years ago, but my concerns about cell phone reliability were shot down.

      Probably a regional UHF business radio network provider like Fisher Wireless [fisherwireless.com]. I think Fisher is a regional provider that only covers California and AZ, but I know national coverage is available. I think all of the interconnects between their radio sites run over UHF links, so they are completely independent of the cellular and wired network (but I'm not 100% sure about that).

      Talk to any business radio provider and they can hook you up. Prices are reasonable, for around $20 - $50/month you can get unlimited

  • Slashdot is traditionally a bad place for language nazism, but the wording in this summary is just plain weird.

    With a data center on the Avenue of the Americas, CoreSite Realty escaped the worst the storm had to offer.

    When I first read this sentence I thought: a real estate company with a data center? Then I looked at the company website and discovered that they just call themselves "CoreSite"; no "Realty". So allow me to ask: Huh?

    But was it coincidence or careful planning?

    "Coincidence"? I think you mean "luck".

  • Just the fact that the fuel truck could even get to their facility says that they did not receive any significant weather.

The shortest distance between two points is under construction. -- Noelie Alito

Working...