Forgot your password?
typodupeerror
Cloud Microsoft Power

Lightning Strike KOs Amazon, Microsoft EuroClouds 189

Posted by timothy
from the this-basket-of-eggs-is-highly-conductive dept.
1sockchuck writes "A lightning strike has caused power outages at the major cloud computing data hubs for Amazon and Microsoft in Dublin, Ireland. The incident has caused downtime for many sites using Amazon's EC2 cloud computing platform and Microsoft's BPOS (Business Productivity Online Suite)."
This discussion has been archived. No new comments can be posted.

Lightning Strike KOs Amazon, Microsoft EuroClouds

Comments Filter:
  • by artor3 (1344997) on Sunday August 07, 2011 @11:04PM (#37018874)

    ...nature wins?

    • by Torinir (870836) <torinir@g[ ]l.com ['mai' in gap]> on Sunday August 07, 2011 @11:07PM (#37018886) Homepage Journal
      Flawless Victory!
    • Sephiroth is rumored to have been stuck in the middle.

    • by SMoynihan (1647997) on Monday August 08, 2011 @04:33AM (#37019976)

      I live in Dublin, and that was some seriously targeted lightning. No sign of storms here, that I saw...

      • I was thinking the EXACT same thing... I know it was raining pretty hard on Saturday, but I didn't see any lightning or hear any thunder
      • by 2phar (137027)
        The lights were flickering here in South Dublin last night around 10pm and there had been an outage of some sort 3-4 hours earlier. I was wondering what was going on, because brownouts/blackouts are extremely rare here. The only real memories I have of power outages in Dublin were rolling blackouts due to industrial action back in the 1970s.
    • by Muad'Dave (255648)

      "History shows again and again how nature points out the folly of men....GODZILLA!" - Blue Oyster Cult

    • Nature bats last.

  • by DWMorse (1816016) on Sunday August 07, 2011 @11:08PM (#37018894) Homepage
    I see how it is. Verizon workers go on strike, MSFT and Amazon gotta call in for something strike-related that's bigger and flashier. Show-offs.
  • My Sympathies (Score:4, Insightful)

    by smpoole7 (1467717) on Sunday August 07, 2011 @11:08PM (#37018896) Homepage

    Considering that my radio stations have been getting hammered for weeks now by this horrible weather in the Southern United States, my sympathies are with them.

    I don't care how much protection you put on your system (and when you have giant lightning rods that are hundreds of feet tall, like we do, you DO try to protect things), an occasional strike is going to slip through. When it does it can get ... messy. :)

    • by Hartree (191324)

      "When it does it can get ... messy. :)"

      I'm often amazed at all the weird failures even when lightning doesn't hit directly and just induces currents.

      Used to see long RS232 runs that woudn't fail instantly, but would act flakey soon after a near strike, and then take a day or two to fail completely.

      • by adolf (21054)

        Used to see long RS232 runs that woudn't fail instantly, but would act flakey soon after a near strike, and then take a day or two to fail completely.

        And that, my friend, is why the good lord gave us RS-422 to use for long runs instead RS-232.

        Balanced, differential signalling and sensible grounding FTW.

    • Re:My Sympathies (Score:5, Interesting)

      by kent_eh (543303) on Monday August 08, 2011 @12:07AM (#37019126)
      I've been in the same spot. (10KW, 3 tower array). It's amazing how far the parts of a capacitor on a P&M panel can spread when propelled by a lightning strike.
      Even with ball gaps, chokes, and all the other effort, ultimately the transmitter has to be connected to the tower. 50 ohms is not that much different than "the shortest path to ground" when you put a few thousand KV against it.

      It took several years after my career change to enjoy the spectacle of a lightning storm
      • by Z00L00K (682162)

        And those 50 ohms aren't resistive ohms either, so the lightning just ignores them.

        A lightning can be very powerful - a 2" steel tube can get disintegrated completely by it.

  • The Cloud (Score:4, Funny)

    by keithpreston (865880) on Sunday August 07, 2011 @11:12PM (#37018914)

    Sounds about like
    http://xkcd.com/908/ [xkcd.com]

  • In my capacity as a Certified Solution Architect(tm), I often warned that The Cloud was suitable only for dynamic workloads. But did you listen? Oh, no, you just went and let your static workloads build up in the Cloud, increasing TCO and, now, bringing down Disaster on your heads!
    • by afidel (530433)
      Dude, if you follow proper procedures then your EC2 instance is mirrored in another availability zone and is using multizone S3 replication. It's not cheap but it's available. If you were really a solutions architect you'd know this =)
      • True enough. I was just trying to eke out a static + clouds = lighting joke.
      • by hedwards (940851)

        Weren't those the same zones that turned out to be hosted in the same facility? Or am I misremembering that.

      • by jimicus (737525)

        Looks like the old "Good, fast, cheap: pick two" adage might need a little rewording. How about "Fast, reliable, cheap: pick two"?

        • by dkf (304284)

          Looks like the old "Good, fast, cheap: pick two" adage might need a little rewording. How about "Fast, reliable, cheap: pick two"?

          Since when was "reliable" anything other than one of the classic metrics for "good"? The old adage needs no changes at all.

    • by plover (150551) *

      In my capacity as a Certified Solution Architect(tm), I often warned that The Cloud was suitable only for dynamic workloads. But did you listen? Oh, no, you just went and let your static workloads build up in the Cloud, increasing TCO and, now, bringing down Disaster on your heads!

      Let me see if I understand you. Static buildup in the clouds caused a sudden discharge redistributing the static to a different cloud with its own static buildup, which then failed to discharge its duties after being charged by the other cloud's discharge.

    • by sgt scrub (869860)

      It just isn't the same without a blond in there somewhere.

  • Cloud fail (Score:5, Insightful)

    by Baloroth (2370816) on Sunday August 07, 2011 @11:20PM (#37018948)

    My understanding of the point of cloud computing was that it would be distributed. I.e. the failure of any one data or computing center would mean the data was still available. Hence, the term "cloud": nebulous, non-localized. Apparently, someone forgot to tell Microsoft and Amazon what the buzzwords they were using actually mean. I more or less expected that of M$, but the fact that Amazon failed too, well, thats pretty a little surprising. I guess it's kinda the norm for all large corporations.

    Glancing at the article, it looks like this outage effected only a certain area, but still, cloud should mean other data centers would take over. I particularly love the quote "Dublin has become a key cloud computing gateway." If one city serves as a "gateway", its not a cloud system. I understand using it as one data center, but others should take over automatically for that area in case of a failure. If you don't have a failover system, you don't have a real cloud computing platform. You have a wannabe cloud computing platform. Or maybe they are just taking a buzzword and redefining it to suit their purposes. That's... exactly what we should expect, I suppose.

    Or am I completely misunderstanding the meaning of this latest buzzword? It's quite possible, I never quite got down what "Web 2.0" was supposed to mean either. Beyond lots and lots of Flash.

    • Re:Cloud fail (Score:5, Informative)

      by HTMLSpinnr (531389) on Sunday August 07, 2011 @11:26PM (#37018982) Homepage

      For EC2, it's only distributed if you pay to have your "service" running in more than one availability zone.

    • As near as I could tell, Web 2.0 boiled down to one thing: The HTTP request object in javascript.

      Where I work, people who are normally major control freaks are seemingly eager to let their processing and data storage go to The Cloud. It will be interesting to see how long The Cloud is down from this lightning strike. The control freaks don't like it when a company web app isn't accessible for even a few minutes, let alone an hour.

      • by DrXym (126579)

        As near as I could tell, Web 2.0 boiled down to one thing: The HTTP request object in javascript.

        Well that was the main thing but greater maturity of DOM, HTML and CSS played their part. Even before XMLHttpRequest turned up it was still possible to do something analogous by doing a form submitting within an invisible iframe and parsing the result. XMLHttpRequest made things a lot easier although same origin policy was a huge pain. Most modern browsers allow cross origin requests these days but it's still a pain to set up.

    • by TubeSteak (669689)

      Or am I completely misunderstanding the meaning of this latest buzzword?

      The main feature of cloud computing is the ability to almost instantly increase your capabilites and capacities.

      At its most basic level, you're talking about a pool of computing resources that can be doled out without regards to the underlying hardware.
      There's no promise that this cloud (the aforementioned pool of hardware) is geographically distributed.
      Like any other hosting, you have to pay extra if you want your data replicated at another hosting center.

    • Re:Cloud fail (Score:5, Insightful)

      by Wolfling1 (1808594) on Monday August 08, 2011 @12:26AM (#37019212) Journal
      Ah, yes. There is that.

      At the moment, my company is aggressively encouraging our customers to avoid the Cloud at all costs. Let me explain why.

      Whilst the technology exists for the cloud to deliver fault tolerant distributed storage, when you choose to put data in the cloud, you are choosing to relinquish control of the data. You are placing it in the hands of someone else. Quite probably an organisation that you do not know intimately. Quite probably an organisation that is based in a different legislative region - probably another country.

      You have little to no capacity to audit their system. You have little to no capacity to test their fault tolerance. And here's the sucker punch - you have little to no legal comeback for the consequences if something bad happens.

      If your data contains any personal information about another person, you are placing the privacy of that person in the hands of an organisation you do not control, and upon whom you cannot enforce any legislative restrictions.

      So, unless you are seriously geared up to investigate and audit your prospective cloud provider - and they are willing for you to do so, the only data you can safely put in the cloud is data that would be basically irrelevant to your core business anyway. Until the fundamental issues of privacy, security and accountability are resolved - or dramatically improved - placing core business data in the Cloud is a massive corporate risk.

      They should not have called it the 'Cloud'. They should have called it the 'Arse' - because if your management are planning to stick their heads in one, they may as well stick their heads up the other. I don't imagine that 'Arse Computing' would be as popular though.
      • by Sarten-X (1102295)

        I quit paying attention to your explanation/rant after one particular choice of words:

        avoid the Cloud at all costs.

        I immediately envision a scenario where the cost of setting up management, infrastructure, and equipment is significantly larger than the cost of losing a portion (or perhaps even all) of a company's data or processing capacity. Rejecting cloud services as a viable option regardless of the actual cost is just as asinine as rejecting the option for turning on hot water in the bathroom sink, because it just might be too warm

        • Re:Cloud fail (Score:4, Interesting)

          by jimicus (737525) on Monday August 08, 2011 @04:31AM (#37019962)

          As for auditing, uptime, and legal consequences, you've apparently never dealt with a service contract. If the contract mandates five nines of uptime, and includes a clause making them liable for all damages and loss, that's a pretty hefty legal comeback.

          I agree with you entirely, it's an absolutely beautiful piece of legal comeback. But every service contract I've ever seen is so full of ifs, buts and other assorted get-outs that it's very rare to actually be able to hold someone to it.

          The one time I have seen an SLA that was actually quite good, the company in question didn't refuse to honour it. Oh no. They went one better - they hadn't even told their staff that it existed, so if you asked about it you'd get a response along the lines of "What's an SLA, then?" The only way you'll get an SLA honoured in those circumstances is to take your provider to court, and you can bet that if you do they'll drop you like a hot potato. So you probably wouldn't bother in any but the most egregious of circumstances.

      • (1) How would you not be backing up to a local resource?
        (2) Why would you not be using encryption at your user site before sending data to the remote server?

        If you're small, choose a small service like SpiderOak.
        If you're large, build a custom front end.

        I don't have particularly sensitive data, so I don't encrypt at my end, though in theory I could (but it would be a pain) and SO had issues with their distributed synchronization back then (note: if I were big enough, I could have hired an IT person to manag

      • Whilst the technology exists for the cloud to deliver fault tolerant distributed storage, when you choose to put data in the cloud, you are choosing to relinquish control of the data. You are placing it in the hands of someone else. Quite probably an organisation that you do not know intimately. Quite probably an organisation that is based in a different legislative region - probably another country.

        Which is the real issue: No way for a European company to use a US cloud provider - Amazon, Azure, Google. The Patriot Act is prohibitive here. [google.com]

      • by codepunk (167897)

        "At the moment, my company is aggressively encouraging our customers to avoid the Cloud at all costs. Let me explain why."

        Because it is eating your profits.

        The same lightning strike we are talking about could have just as easily taken out a in-house data center. I would also argue with that being the case the situation would have likely even been worse for the customer.

    • Distribution means your virtual machine can be on a number of machines inside a cloud. There's nothing in the definition of a cloud that says it has to be in different locations, or running mirrored copies of your instance. Sure, it's possible, just like it's possible with single machines. When will people stop assuming that "cloud" means "indestructible"? This is exactly what happened before with EC2 and lots were hurt then by the same assumption.
    • by mcrbids (148650)

      Perhaps it would be a good idea to start by defining what exactly "cloud computing" means?

      Because looking at the Wikipedia article [wikipedia.org] I see only a brief mention of reliability through redundancy: Reliability is improved if multiple redundant sites are used, which makes well-designed cloud computing suitable for business continuity and disaster recovery.

      As CTO of a data hosting "cloud services" provider myself, I'm proud of our track record for reliability and redundancy. All our systems are backed up offsite t

    • Re:Cloud fail (Score:4, Informative)

      by jimicus (737525) on Monday August 08, 2011 @04:02AM (#37019862)

      Cloud computing is a buzzword meaning "don't run your own hardware, run your business on someone elses". Which might mean anything from a virtual server that you manage at one end of the sophistication scale to a SaaS product at the other.

      All sorts of aspects of this are optional. Including:

      1. Whether or not you manage the underlying operating system - including things like security patches and hardening. You can choose a cloud computing provider that has sysadmins deal with that for you and just run the application yourself; they are a LOT more expensive than Amazon.

      2. How much effort your provider puts into making their systems geographically redundant. Few will talk openly about this; I'm prepared to bet hard cash this is because the vast majority that offer you a virtualised server are just using a web interface to expose a fairly vanilla Xen-with-a-SAN infrastructure to the world with everything sat in one place. Providers that will run the OS for you and can honestly say their infrastructure accounts for complete data centre loss are like hen's teeth.

      3. If you've gone for a SaaS provider - how much effort their developers went to to ensure their application can stand up to everything up to and including total loss of a data centre. And whether or not they test for such an occurrence.

    • by sjames (1099)

      Well, according to the trade press, "the cloud" is both a floor wax and a dessert topping (plus a bag of chips).

      In reality, it's fine grained virtual server rental with an API. Nothing more and nothing less.

      While that is quite useful if you need it, it doesn't magically fix any problems other than highly variable server loads. As we see here, it isn't a magic wand for uptime/availability.

      • by dkf (304284)

        In reality, it's fine grained virtual server rental with an API. Nothing more and nothing less.

        That's exactly what IaaS is. Higher levels (notably SaaS) are a bit more; in those cases you're buying the service — wherever that's running $mdash; and not the server. You get less control but have to do less of the legwork yourself. It's the classic value-add model, and it makes a bunch of sense for many people. (Not everyone, but that's OK. Nobody can be all things to all people.)

    • by Trogre (513942)

      You were lied to.

    • by aug24 (38229)

      Oh silly underinformed person. There is a datacentre in Dublin. It is one of six Amazon datacentres. The others were unaffected, as were (our) public facing services, because only some (of our) servers are placed in Dublin.

      It looks like a cloud from the outside. Those of on the inside know where the servers are because we want to choose where we place them for latency / redundancy reasons.

  • ... And if we aren't 1000%, absolutely, positively reliable may God Strike Us... BLAM!!!!!!

    • ... And if we aren't 1000%, absolutely, positively reliable may God Strike Us... BLAM!!!!!!

      Sorry sir. The thousands place is the sign bit in our percent calculations.

    • (posting to undo wrongful mod, a bug in the system? dropdown mod combo box, use arrows to scroll, press escape to cancel, it then applies the last highlighted mod option :/)

      was meant to mod funny btw :)

      • by Pieroxy (222434)

        (posting to undo wrongful mod, a bug in the system? dropdown mod combo box, use arrows to scroll, press escape to cancel, it then applies the last highlighted mod option :/)

        It's not a bug, it's a feature. If you want to cancel your mod, you have to unplug the RJ45 at the back of your computer first. RTFM for God's sake !!!

  • by countertrolling (1585477) on Sunday August 07, 2011 @11:36PM (#37019018) Journal

    Office 364

  • Serves them right (Score:2, Informative)

    by RobinEggs (1453925)
    Those massive data centers only existed because Microsoft and Amazon channeled profits through Irish subsidiaries to avoid US taxes. They serve some legitimate functions for customers in the UK as a matter of convenience (why build two data centers?), but they're primarily money laundering centers.

    I'd call a few lightning strikes the least of the punishments those data centers - and the entire infrastructures to which they're attached - really deserve.
    • by XaXXon (202882)

      The AWS services out of dublin aren't through an Irish subsidiary. It's just regular AWS.

      If you know differently, please document it.

      • by TubeSteak (669689)

        The AWS services out of dublin aren't through an Irish subsidiary. It's just regular AWS.

        I don't think that's what the GP was getting at.
        Due to their complex tax [strike]avoidance[/strike] management plans, American based multi-nationals end up with billions of dollars overseas.
        They can't bring those billions back to the USA without being taxed, which would completely negate the point of booking them offshore to begin with.
        The end result is that they invest it overseas instead of domestically, because it looks better for their bottom line.

        Just because AWS LLC isn't an Irish subsidiary doesn't m

  • This is what is known as "cloud to cloud lightning"
  • Power Co-Generation (Score:4, Interesting)

    by anubi (640541) on Monday August 08, 2011 @12:12AM (#37019150) Journal
    While working at Chevron Oil Pascagoula Mississippi refinery, I noted Chevron had the same problem. Loss of electrical power to the refinery would be catastrophic. No one wants to be around tons of petrochemical products undergoing serious chemical reactions when one loses control.

    To mitigate this threat, Chevron worked with Mississippi Power to operate a power generation facility at the refinery.

    I would think that anywhere there is a substantial "data processing farm" with critical power requirements, business arrangements should be made with the power generation utilities to run a natgas power plant in the immediate area.

    The utilities often run these plants as "topping" plants, as they needed anyway to even out short-time load variances on the line.

    But, in the event of a serious loss of grid power, it can be awful handy to have a few megawatts of power coming from down the street.
    • RTFA! They do have generators! The lightning strike was apparently so powerful that is affected the backup generators synchronization equipment.

      "Normally, upon dropping the utility power provided by the transformer, electrical load would be seamlessly picked up by backup generators,â Amazon said in an update on its status dashboard. âoeThe transient electric deviation caused by the explosion was large enough that it propagated to a portion of the phase control system that synchronizes the back
      • When I was an undergrad I worked a crappy job at a Florida amusement park. Lightning capital of the world is in Pasco County Florida which was about 30 minutes away.

        They generated their own power and the power lines and even the roller coasters were designed to be struck by lightning. Let me tell you they were struck every 3 or 4 days during the raining season in summer from the monsoons from the Carribean and Gulf. During a bad storm the power lines and rides could be struck 2 to 3 times each in a 30 minut

        • by Monoman (8745)

          Yeah but how many times did the amusement park get taken offline before they came up with a solution. The old saying "it is not a matter of if, it is a matter of when" often applies. Unless compelled by some law, upper management often listens to the bean counters until something goes wrong ... then they start asking questions and opening the checkbook.

        • Sure you could make a datacenter handle a lightning storm like that easily. Simply take it off line every time you see a lightning storm. When the storm has passed, you bring it back online again. Just like they do with the amusement park.

          Some people may have problems with that kind of availability and prefer the risk of the datacenter going down 1 day over a period of years instead of going down every lightning storm. But they are just being overly picky I am sure.
      • They should have had three generators!
  • ...where life is grand and all your apps are immune to...BzzzzzZZZZZT!
  • How shocking! (Score:5, Informative)

    by NicknamesAreStupid (1040118) on Monday August 08, 2011 @12:41AM (#37019270)
  • Murphy was an Irishman

    Every Silver Cloud has a leather lining.

  • by Crypto Gnome (651401) on Monday August 08, 2011 @01:55AM (#37019474) Homepage Journal
    Despite ALl the market-hype and brew-haha going on, the simple fact remains:

    If ALL your computing power is in ONE SINGLE DATACENTRE then what you have is a DAMP SPOT not a CLOUD.
  • All of the cloud computing hype has business everywhere, once again, buying what they don't understand just as they did during the dotCOM bubble. That particular bandwagon caused all sorts of damage to the industry including a flood of people unsuited to the line of work and suppression of wages that don't seem to have ever returned. Now business continues to crave cheap, Walmart-ized IT services and are seeking to get it any way they can; outsourcing to 3rd world nations and most recently entrusting clou

  • (see below) Why does synchronization have to be manual? Or is this a safety feature that the automatic synchronization decides to go offline if things get weird?

    From the Article:

    âoeNormally, upon dropping the utility power provided by the transformer, electrical load would be seamlessly picked up by backup generators,â Amazon said in an update on its status dashboard. âoeThe transient electric deviation caused by the explosion was large enough that it propagated to a portion of the phase cont

  • marketing bullshit (Score:5, Insightful)

    by Tom (822) on Monday August 08, 2011 @04:20AM (#37019914) Homepage Journal

    And there is the marketing bullshit revealed. All the promises of the cloud - down by one lightning strike.

    Because, let's face it, the whole "cloud" thing as they sell it is just advanced virtual hosting with a different name. The only real cloud capabilities are those the big companies build for themselves, and they did things like that 10 years ago already, when nobody had ever heard the term "cloud" used in computing contexts.

    In the end, it's about selling something to people who already have the older version and convincing them to buy the new one. So you give it a different name because a "new" product sells easier than the upgraded version of an "old" product.

    Anyone remember when "Web 2.0" was all the hype? It really wasn't a 2.0 as we all know. There was nothing new in it, all components had been around for a long time. It was a conceptual bundle, but not a new version like the name suggested.But "we're doing more Javascript" now doesn't sell nearly as good as "we're moving to Web 2.0 now".

    • by aug24 (38229)

      But they weren't down. I have servers in Dublin. I also have striped redundancy across other EC2 datacentres. We had 100% uptime last night and I'm now watching the Dublin based machines recover gracefully.

      Yes, I agree it's "advanced virtual hosting with a different name". But it didn't break its promises.

      • by Tom (822)

        Yes, I agree it's "advanced virtual hosting with a different name". But it didn't break its promises.

        That depends on what promises you mean.

        "The Cloud" has been hyped as this mystical thing that means you just move your servers "into the cloud" (whatever that means) and mystically physical location ceases to matter.

        And that promise was broken. You need to worry about things like redundancy, multiple data centers, etc. etc. - just like you did before. I've designed HA systems myself many, many years ago when "cloud" was something in the sky. We set up our own redundant upstreams and taught our routers BGP.

      • by Junta (36770)

        But it didn't break its promises.

        I can read countless stories of people who use things like EC2 and express disdain toward users afflicted by downtime because they didn't understand how to work it. When all is said and done, the users using EC2 'right' seem to end up having to worry all the frustrating details of HA they worried about before, 'Cloud' doesn't change that. The 'promise' is that you naively slap your workload into a 'cloud' provider and you don't have to think so hard about all of that. Even if Amazon doesn't explicitly ma

  • by markbark (174009) on Monday August 08, 2011 @07:11AM (#37020428) Homepage

    Business Productivity Online Suite?
    I always thought it stood for "Big Piece of Sh..... never mind

  • So... Zeus accomplished what Anonymous could not.

In order to dial out, it is necessary to broaden one's dimension.

Working...