Follow Slashdot blog updates by subscribing to our blog RSS feed

 


Forgot your password?
Close
typodupeerror
×
Data Storage Hardware

Data Center Backup Generator Kicks In During Power Outage and Catches Fire (theregister.com) 71

Posted by BeauHD from the halt-and-catch-fire dept.
Joe_Dragon shares a report from The Register: A power outage kicked off a fire in web hosting biz WebNX's Ogden data center in Utah on Sunday, knocking the facility offline temporarily and leaving several servers in need of a rebuild. Kevin Brown, Fire Marshal for the US city's Fire Department told The Register in a phone interview that firefighters responded to a call on Sunday evening. The fire, he said, "originated in a generator in the building and spread to several servers." Brown said the facility's fire suppression system contained the blaze and that fire department personnel assisted with the cleanup. He said power was cut to the building until an electrical engineer could inspect the facility to make sure current could be restored safely, which he added is standard procedure. He also confirmed that some of Ogden City's IT services were down on Sunday and Monday as a result of the data center fire. "Sunday afternoon the city power was disrupted and, as designed, our backup generators automatically switched on," the company said in a Facebook post. "However, during that transition, one of our backup generators that had been recently tested and benchmarked specifically for this situation experienced a catastrophic failure, caught fire, and as a result initiated the fire suppression protocol."

"Some servers will have an extended outage as they may require rebuilds due to some water damage. Those builds have a high probability that data is intact." They added: "Customer's servers in one of our main bays were exposed to water and possible damage may have occurred. No fire damage was inflicted on customer servers."
This discussion has been archived. No new comments can be posted.

Data Center Backup Generator Kicks In During Power Outage and Catches Fire More

Data Center Backup Generator Kicks In During Power Outage and Catches Fire

Comments Filter:

  • Halt And Catch Fire (Score:5, Funny)

    by nospam007 ( 722110 ) * on Saturday April 10, 2021 @05:10AM (#61257892)

    Literally!

    • Give the show a watch if you haven't.

  • Redundancy? (Score:5, Insightful)

    by Registered Coward v2 ( 447531 ) on Saturday April 10, 2021 @05:15AM (#61257896)

    "Some servers will have an extended outage as they may require rebuilds due to some water damage. Those builds have a high probability that data is intact."

    Isn't the whole point of offsite data storage that the servers have redundant backups designed to maintain operation and data should one set get knocked out or fail?

    They added: "Customer's servers in one of our main bays were exposed to water and possible damage may have occurred. No fire damage was inflicted on customer servers."

    Thank god I only lost my data from water damage, the thought of it burning to death is horrifying.

    • I guess they defined "redundancy" as: Another server right on top of the other one, or something like that. ;)

      Hint: Never rely on just one site either.
      Triple software times triple hardware (times triple cables) times triple sites times triple backups. And then add versioning to the entire thing. (From transactions one can roll back over snapshots to keeping a history of backups.)
      Oh, and for cash payment you can rent a 18-ton truck. ;)

      • Re: (Score:1)

        by gwjgwj ( 727408 )

        I guess they defined "redundancy" as: Another server right on top of the other one, or something like that. ;)

        Yeah, better put them in two separate towers. Oh, wait...

    • Re:Redundancy? (Score:4, Informative)

      by Volda ( 1113105 ) on Saturday April 10, 2021 @08:14AM (#61258188)
      Its probably that they organized their racks by customer. This customer gets these racks together this one get those etc. While it does make it easier to physically and logically manage things, it does reduce the redundancy in this case. You would need to spread the customers boxes out and completely mix them up to prevent this situational outage. Would there be an discernable difference in fiber latency going say a football field size distance, additional switches, etc? Maybe there are technical reason why the machines are physically near each other. I'm kind of surprised that they didn't have the generators in a separate building or have better fire suppression and detection system. Usually our large generators are in their own secure building away from where the power is being used. That or we bring in a generator and connect it outside of the building to a big port that leads to our UPS's.

      • Its probably that they organized their racks by customer. This customer gets these racks together this one get those etc. While it does make it easier to physically and logically manage things, it does reduce the redundancy in this case. You would need to spread the customers boxes out and completely mix them up to prevent this situational outage

        If it's like the datacenter where my servers are located, customers rent one or more cabinets (it's also possible to rent 1/2 cabinet).

        Any sensible configuration wi

  • Water damage? (Score:1)

    by Anonymous Coward
    Who puts water-based fire supression in a data center???
    • You can't use halon and other non-water based fire suppression in newer data centers in a lot of geographies.

      • Re: (Score:2)

        by jbengt ( 874751 )
        You can't use Halon(TM) almost anywhere anymore, but there are replacements that are ozone-friendly, and maybe some that are less global-warming, also.
        That doesn't mean you can avoid the water-based fire suppression, though. For example, I was involved in the design of new emergency generators for an airport. They had six generators in a new fire-rated building, each rated to produce 3 MW at 4160 Volts. The fire protection engineer had designed a CO2 fire suppression system, but the insurance carrier, F

    • Re: (Score:3)

      by gmack ( 197796 )

      The fire department if I had to guess.

    • firecode laws / some times after dry supression.

      Some places have both dry and wet supression. The wet supression has an melt point on the head that can go off with out needing any power.

    • Re: (Score:3)

      by jbengt ( 874751 )

      Who puts water-based fire supression in a data center???

      Actually, that's the norm. It's usually a pre-action system that requires fire/smoke detectors to go off before letting water into the system, mainly to prevent damage from accidental discharge from a sprinkler head.
      A clean agent system like FM-200 is extremely expensive for a large data center; and it's essentially impractical unless you split larger areas up into smaller individual closed spaces. And even if you have clean agent fire suppression,

    • Re:Water damage? (Score:5, Interesting)

      by aaarrrgggh ( 9205 ) on Saturday April 10, 2021 @11:26AM (#61258658)

      Everybody. The real question is how the hell a generator fire could cause fire suppression systems inside the data center to activate. It isn’t like they are in the same room... usually at least a 2-hour fire separation is required.

      Fire can spread when sprinklers activate with a diesel spill associated with a fire, but even that is usually a low risk. Some material information seems to be missing.

  • Water in a data center? (Score:4, Insightful)

    by LordWabbit2 ( 2440804 ) on Saturday April 10, 2021 @05:30AM (#61257920)
    What shit show of a data center uses water for fire suppression?
    What a joke.
    Also was the generator so close to the actual servers that they had to get doused with water to quench a fire in the generator?

    No fire damage was inflicted on customer servers

    Because the water damaged them first, hard to set fire to a sopping wet server.
    If this whole debacle doesn't make you take your hosting elsewhere then you deserve to lose data.

    • Re:Water in a data center? (Score:5, Insightful)

      by kent_eh ( 543303 ) on Saturday April 10, 2021 @07:39AM (#61258128)

      What shit show of a data center uses water for fire suppression?

      One that is following fire code? In my experience data centres will have both pre-action sprinklers and a clean agent (Something like Inergen) Depending on the local fire code, tehy will will require sprinklers to protect the structure and any adjacent structures. The Inergen (or other clean agent) is there to protect the equipment, but the sprinklers will operate to protect the building if the clean agent fails to stop the fire.
      Also, if the fire department found active fire after the power was cut, they'll use water to knock that fire out.

      • I have worked extensively on BMS systems, I understand that standard fire suppression is going to be present (ie water).
        What I don't understand is why a generator fire which caused a fire suppression event ended up dousing servers? Why were they on the same loop? This "data center" was an adhoc crap fest that does not deserve to be called a data center.

    • What shit show of a data center uses water for fire suppression?
      What a joke.
      Also was the generator so close to the actual servers that they had to get doused with water to quench a fire in the generator?

      A real joke is making a lot of assumptions based on a throwaway line in a media release from a PR department that a server suffered water damage.

      • Yeah, I'm not getting the real joke. Their generator caught fire and they pissed on the servers. If that was my server I would be seriously fucked off. However you seem to intimate that you have more knowledge about the situation, perhaps you can explain more? At the moment it looks like they didn't know what they were doing and the "data center" was not worth it's name in quotes. Their fire suppression for the generator was linked to the fire suppression for the rest of the building, so as soon as tha

    • Re: (Score:3)

      by gmack ( 197796 )

      This reminds of of rackspace Texas data center indecent over a decade ago. They had an explosion and everything worked on the backup power until the fire department showed up and orderered them t turn it all off so they could safely put out the fire. The fire department doesn't care about uptimes or data integrity, they only care about saving lives.

      Also, a few years ago I worked in a building that housed the head office and servers for a multi billion dollar business. One day some construction crew brok

  • At which point... (Score:5, Funny)

    by h33t l4x0r ( 4107715 ) on Saturday April 10, 2021 @05:31AM (#61257922)
    The secondary backup generators came online, put out the fire from the primary backup generators, and proceeded to spray raw sewage from the secondary sprinklers. A senior datacenter engineer who was reached for comment replied: "It could have been worse".

  • One Day of credit per 15 minutes of downtime? Ouch (Score:4, Informative)

    by bardrt ( 1831426 ) on Saturday April 10, 2021 @05:33AM (#61257924)
    per the article: WebNX's SLA guarantees 100 per cent uptime and uninterrupted power every month, with account credits of one day per 15 minutes of downtime in each case.

    If they're down for a week that's 672 days free for each customer...

    • I guess they bet on people thinking "Why would I want credit for using a data center that just drowned my servers??" and ending their contracts early. ;)

    • Re: (Score:2)

      by decep ( 137319 )

      If the WebNX contracts are anything like other contracts, the SLA will not apply to this event.

      SLAs only apply to standard business impacting events, not DR events. Once you have a DR event (like an "act of god" situation), the SLA is basically thrown out the window and DR timelines start to apply. WebNX could have 90 days to restore service before service credits are applied.

    • The SLA is capped at 30 days of credit in any 30 day period.

  • Odd-Looking Installation (Score:5, Interesting)

    by ytene ( 4376651 ) on Saturday April 10, 2021 @05:59AM (#61257950)
    With a little bit of help from this article [datacenterfrontier.com], I was able to locate the data center in question using Google Maps.

    Here [google.com] is a Google Maps view of the same building from a near-identical location as the photograph from the "Data Center Frontier".

    If you explore the Google Maps view, one of the things that you will not see is an external generator associated with this WebNX facility. The closes I could get to spotting it can be seen in this Google Streetview image" [google.com]. If you look at the roof line, near the centre of the image, you will see a rotating-vent exhaust that looks like it might be connected to say a central heating system or similar. Its the only visible sign I could see of anything that looks like it could handle hot exhaust gas from an internal combustion engine powering an electrical generator.

    Obviously I'm just speculating here - and more importantly, I don't have any information about the internal layout of the structure. It's entirely possible that the operators have a purpose-built interior room that is comprehensively fireproof and robustly separated from the technology part of the data center facility.

    On the other hand... Unless there is an external generator installation at the facility [I am entirely happy to concede this point if I've missed it!!!] then it looks like this host has built their diesel backup generator inside their hosting facility. Which would rather suggest that any sort of fire or similar problem with the generator could have the reported impact of affecting racked hosts.

    It is all to easy and not remotely helpful to cast opinions based on lack of concrete data and on information sources as incomplete as Google Streetview. So I won't. But I will say that this location does look rather unusual, given the lack of apparent distinct backup generator location.

    Interesting to see the Fire Department write-up if/when they publish it.

    • then it looks like this host has built their diesel backup generator inside their hosting facility.

      I'm anything but an expert in this area, but this is the first question I had while reading TFA: How would a generator incident trigger a water fire suppression protocol inside technical rooms ? It must be very close, or even inside the data center rooms, right?

      Is there a real interest, or a particular constraint, to place generators so close to the data center ?

      • Re: (Score:2)

        by ytene ( 4376651 )
        We both had exactly the same thought then? I read TFA and thought, “No way should a generator and server racks be in close enough proximity for a fire in the former to have *any* impact on the latter. Just no way.” That send me off exploring and led me to conclude that the generator is physically inside the core building.

        Which, sorry, is nuts.

        Apart from all the fire hazard risks, the challenges of storing sufficient fuel for decent run-time, leaves you with the risks of things like mainten
        • Yeah this didn't make sense to me either. All the data centers I've been to (or worked at) have the generators outside and away from the building. Looking at the building in question, it seems pretty obvious that they just repurposed some generic existing structure instead of doing things properly.

          I bet it was cheaper though.

          • Re: (Score:2)

            by ytene ( 4376651 )
            A long time ago, I designed and commissioned the build of a relatively small (4800 sq ft) data center / comms room setup). We ended up with a pair of decent-sized 6-cylinder diesel generators and a UPS that gave us 20 minutes of standby time, just in case we had trouble starting either generator (each of which could handle 100% of the load on commissioning date and was intended to be able to handle at least 75% through the life of the generator.

            I simply wasn’t allowed to put the generators close to

          • Re: (Score:2)

            by jbengt ( 874751 )
            Mounting generators outside is cheaper. That's the main reason you usually see it that way. But there are plenty of places where mounting generators outside is impractical due to space, noise, etc. And codes often insist that they be on the first floor; if so they can't go outside on the roof
            Storing fuel would not be an issue for gas generators. If you have a back-up diesel generator that is meant to run for any significant length of time, you would have to have separate diesel storage tanks anyway, al

      • Re: (Score:2)

        by jbengt ( 874751 )

        Is there a real interest, or a particular constraint, to place generators so close to the data center ?

        The interest is to place the generators close to the electrical service, so you don't need a bunch of huge conduits and cables connecting the generators to the transfer switch.

    • That whole area of the building (behind the green transformers and white shelter thing) appears to be a large electrical room with 2 distinct electric utility service entrances (3 door grey boxes each have revenue meters on them). I would agree with you that this is likely the location of the generators and related switchgear.

      In multiple articles, they quote the operator as saying "generators" but typically exhaust systems would be independent of one another. As you pointed out, there is only one vent sta

    • You can see the genset in streetview (Score:1)

      by Anonymous Coward

      https://www.google.com/maps/@4... [google.com]

      The electric radiator louvers are above motercycle on the loading dock. When the generator starts they open.

      The exhaust pipe is the the brown pipe you see running horizontal above the silver station wagon.

      • Re: (Score:2)

        by jbengt ( 874751 )
        Interesting.
        It looks like they repurposed a loading dock bay for the generator. That doesn't seem like a very big engine exhaust vent, so, if it's the only one, it would seem they did not have a lot of back-up power. (Also, don't park under it if you don't want condensation from the exhaust to drip on your car.)

    • It is all to easy and not remotely helpful to cast opinions based on lack of concrete data and on information sources as incomplete as Google Streetview. So I won't.

      What are you talking about. This is Slashdot. You're supposed to just assume the generator was in the middle of the datacentre and that the entire building got insta-flooded with water including all the servers, and then call everyone involved an incompetent moron. That's what everyone else on Slashdot does. Don't go doing investigations for details, and on top of that then not make an assessment. That would be talking too much sense for this site.

      • Yes, absolutely, and well said!
        OTOH for more along that line checkout theregister for 'BOFH' (which used to be funny) or many of their 'on_call' stories for real world examples.

    • Pretty sure you are looking at the wrong building— it is across the street from what you seem to be referring to. It isn’t too hard to hide generators from aerial view, especially if they are hardened for ballistics, but I couldn’t spot them in a cursory review either.

      • Re: (Score:2)

        by ytene ( 4376651 )
        If you navigate in streetview to the junction of 1st and 600W, you can see the company sign on the corner of their plot. And the starting point matches the image on the linked article... So I'm *reasonably* sure I got the right building...

        • That one looks like a sales gallery to me— the functional space is the newer building with much more metal and protective fencing— but still tries to fit in.

    • Re: (Score:2)

      by imidan ( 559239 )

      it looks like this host has built their diesel backup generator inside their hosting facility.

      I mean, TFS summary says: "Kevin Brown, Fire Marshal for the US city's Fire Department told The Register ... 'the fire ... originated in a generator in the building and spread to several servers.'" Doesn't that suggest that their generator is inside the building?

  • I *love* "standard procedure". (Score:3, Interesting)

    by BAReFO0t ( 6240524 ) on Saturday April 10, 2021 @06:57AM (#61258052)

    Like "security reasons" it relieves one of all need to actually state any reasons. (In this case, there were valid reasons, in the contect of an incompetent data center design, but they were not stated.)

    Notable mention: "It's the law!". (Invalid implication: It is right!)

  • No C4 needed.

  • and as luck would have, some of the water damaged servers were for EEVBlog.

  • I wonder how long Tesla Megapacks or the competitive battery systems could keep the data center running?

    • I wonder how long Tesla Megapacks or the competitive battery systems could keep the data center running?

      I don't know what the capacity of a Megapack is, but data centers require a pretty substantial amount of energy. There are various UPS systems available just for that purpose, so I don't see any reason why a large Tesla battery pack couldn't be used, but you'd still want a generator for any extended outages.

      I guess what I'm trying to say is, generators make electricity and batteries don't.

    • Re: (Score:2)

      by jbengt ( 874751 )

      I wonder how long Tesla Megapacks or the competitive battery systems could keep the data center running?

      I don't know how big this data center is, but I once worked on a project where they had about 2.5 megawatts of UPS and batteries in order to provide 15 minutes of power for an orderly shutdown. (Chances are, the batteries would have lasted 30 minutes, but it was designed for worst case.) Most UPS use lead-acid batteries, anyway, since that's a mature, inexpensive technology and weight doesn't matter mu

    • Data centers generally already have expensive battery backup systems, but they aren't designed to run long term. They are generally sized for maybe 5-10min of power, so that they can instantly take the data center load when the power cuts out, while the generator starts, gets up to speed and stabilizes before cutting over from battery to generator. Then there is the amount of space you need for say a 1 megawatt battery compared to a 1 megawatt generator. A 1 megawatt generator could fit in a shipping contai

  • "However, during that transition, one of our backup generators that had been recently tested and benchmarked specifically for this situation experienced a catastrophic failure, caught fire, and as a result initiated the fire suppression protocol."

    I remember my first corporate sysadmin job. Built a server farm, full UPS support, self-tested every week, rundown test every month, and yanking the power cord out of the wall once in a while, just for good measure.

    First time we have an actual power outage, the UPS powering the gateway server locks up.

  • Some of our water customers test-run their backup power generators weekly. It also keeps the fuel from getting stale.

  • OVH datacenter fire not long ago (Score:3)

    by ZiggyZiggyZig ( 5490070 ) on Saturday April 10, 2021 @01:24PM (#61259046)

    There was a fire in one of the datacenters of a major European web and server host in March, impacting many customers. It seems that they didn't have any offsite redundancy and some customers lost their data. So far, we do not know about the cause.

    People who are hosting stuff online should backup their stuff to their local disk, just in case... Do not put too much trust in "the cloud".

    https://www.itpro.co.uk/server... [itpro.co.uk]

  • Time to buy some backup backup generators.

  • Anybody else noticed that the article says this happened on Sunday? Either this happened LAST Sunday, in which case the article is WAY behind the times, or they're describing the damage that won't happen until TOMMOROW? In case you hadn't noticed, here in the good ole US of A, it IS Saturday, April 10th.. Weird..

  • Servers generate heat. Companies have to pay for fans and air conditioning to get rid of that heat.

    Generators produce heat when operating.

    Why would you house the emergency generators in the same building as the servers, thus increasing the heat which needs to pumped out of he building when you turn the generators on? House the generators in a separate building outside. And run their power via cables into the building with the servers.

  • I have five servers in that datacenter. Four came back online in the first 48 hours. The fifth is in the water-damage area and I'm still waiting for them to get to it.

    Sure, we can criticise them having a generator close enough to trigger the sprinkler system, but it's still a better outcome than OVH had in Strasbourg last month.

Slashdot Top Deals

How many hardware guys does it take to change a light bulb? "Well the diagnostics say it's fine buddy, so it's a software problem."

Close