Become a fan of Slashdot on Facebook

 


Forgot your password?
Close
typodupeerror
×
The Internet Hardware IT

Water Main Break Floods Dallas Data Center 230

Posted by Soulskill from the five-thousand-hats-worth dept.
miller60 writes "IT systems in Dallas County were offline for three days last week after a water main break flooded the basement of the Dallas County Records Building, which houses the UPS systems and other electrical equipment supporting a data center in the building. The county does not have a backup data center, despite warnings that it faced the risk of service disruption without one."
This discussion has been archived. No new comments can be posted.

Water Main Break Floods Dallas Data Center More

Water Main Break Floods Dallas Data Center

Comments Filter:

  • Silly rabbit. (Score:5, Insightful)

    by migla ( 1099771 ) on Monday June 07, 2010 @05:28PM (#32489456)

    There should always be duplication of critical components of a system with the intention of increasing reliability of the system, usually in the case of a backup or fail-safe.

    • Re:Silly rabbit. (Score:5, Funny)

      by jo_ham ( 604554 ) <joham999 AT gmail DOT com> on Monday June 07, 2010 @05:33PM (#32489520)

      This is Texas - God is their backup solution.

      Every night they pray for no hardware failures.

      • Re:Silly rabbit. (Score:5, Funny)

        by Mordok-DestroyerOfWo ( 1000167 ) on Monday June 07, 2010 @05:43PM (#32489628)

        God is their backup solution.

        If this is true, shouldn't they have been prepared for a flood?

      • Re:Silly rabbit. (Score:4, Informative)

        by Crudely_Indecent ( 739699 ) on Monday June 07, 2010 @06:22PM (#32490032) Journal

        This is Dallas, where the City Council is doing such a good job, they're trying to get 270% raises and to double the length of their terms.

        • Re:Silly rabbit. (Score:4, Informative)

          by AK Marc ( 707885 ) on Monday June 07, 2010 @09:33PM (#32491366)
          I know you meant to be funny, but the City of Dallas and the County of Dallas are distinct and unrelated governmental entities. You are complaining about the city, but the county is the one affected by the flood. Don't tarnish the bad name of the city with the bad name of the county.

      • Re:Silly rabbit. (Score:5, Funny)

        by couchslug ( 175151 ) on Monday June 07, 2010 @07:12PM (#32490442)

        "This is Texas - God is their backup solution."

        So much for that. I prayed to Allah for a flood.

    • Re:Silly rabbit. (Score:5, Interesting)

      by vlm ( 69642 ) on Monday June 07, 2010 @05:33PM (#32489522)

      There should always be duplication of critical components of a system with the intention of increasing reliability of the system,

      Yeah that might be the intent, but it only works if the combined reliability is higher than individual reliability. Transfer switches I'm looking at you! I have worked at numerous facilities with data centers, and inevitably the transfer switch is less reliable than either wall AC power, or the diesels (youch!). Yes I know exactly what I'm saying, that at every facility I've worked at, power reliability would have been higher without the transfer switch and the generators. But its politically incorrect as the rare wall AC power failure would be unacceptable unless we spent money on switches and gens. As long as you spend money on switches and gens, any low level of reliability is acceptable.

      Your mileage may vary, maybe coastie cities have less reliable power. Don't know.

      • Re: (Score:3, Interesting)

        by headhot ( 137860 )

        Transfer switches are the bane of my existence. I work for a major MSO, and every site we have lost has been to a transfer switch problem. Equipment with 2 ps is the best solution, so you have have 2 sets of transfer switches, UPS, Generator, and Mains.

    • Texas doesn't need backup. (Score:5, Insightful)

      by copponex ( 13876 ) on Monday June 07, 2010 @05:44PM (#32489644) Homepage

      In a state as blessed as Texas, they were told that God would provide protection against acts of God. I imagine many of the faithful are confused, especially when Jesus day is only a few days away.

      Maybe they didn't execute enough retarded people this year?

      http://en.wikipedia.org/wiki/Jesus_Day [wikipedia.org]

      • Re: (Score:2, Funny)

        by Anonymous Coward

        Maybe they didn't execute enough retarded people this year?

        They'll get around to you, don't worry.

      • Re:Texas doesn't need backup. (Score:5, Funny)

        by PopeRatzo ( 965947 ) * on Monday June 07, 2010 @06:00PM (#32489818) Journal

        While Governor of Texas, Former President George W. Bush, signed a bill into law proclaiming June 10, 2000 to be Jesus Day

        Wow.

        This year, they decided to re-write textbooks to eliminate the "liberal bias".

        You know what? Maybe the Dallas datacenter going out without a backup is a blessing in disquise.

        You think we could get Texas to secede if we asked them real nice? We can give them Arizona and a state to be named later, as long as they promise to never tell anyone that they were once part of the United States.

        • "You think we could get Texas to secede if we asked them real nice? We can give them Arizona and a state to be named later, as long as they promise to never tell anyone that they were once part of the United States."

          That's actually not a bad idea. The US is a bit large to be one country, political polarization means people who are dire enemies must fight for power (producing results no one likes), and it could be cut in two or three sections with no great loss. It would be less menacing to the rest of the w

          • ...dire enemies must fight for power

            Would you rather they fight for power within the common political system, or with guns as separate countries? Besides, the divisions you talk about also exist within the states, even Texas.

            "Diversity" isn't bad, but there is no reason everyone should be forced to yield to everyone else when there are such profound differences in opinion, politics, and culture.

            When the laws are done right, everyone gets along because everyone is free to make their own choices. Against ab

    • They probably have lots of tapes, and backups. the data was not harmed, it was safe, it was just not available.

    • People often refuse to learn absent being bitch-slapped by outcomes they chose not to prepare for.

      When "what to do" is so well-known, refusal to do it merits scorn, contempt, and frequent use as an example to others.

    • Re:Silly rabbit. (Score:5, Insightful)

      by Leebert ( 1694 ) * on Monday June 07, 2010 @10:19PM (#32491730)

      There should always be duplication of critical components of a system with the intention of increasing reliability of the system, usually in the case of a backup or fail-safe.

      Let me try to reply a bit more constructively than some of the others here.

      It is never a foregone conclusion that you will always have duplication of critical components of a system, if you are doing proper risk management.

      Essentially, the art of risk management is figuring out how far to go with mitigations of various risks.

      To illustrate with an excessively simplistic example (Assume a perfect vacuum and a frictionless environment):

      Let's say you sell something online, you sell W products/hour, and if you miss a sale, that's it, you're not getting it back.

      So that means that you lose the profit on W products every hour, let's call that X.

      Next, you look at the potential hazards, and calculate how often you expect to have each hazard occur per year. For example, to be simple, let's pretend your only hazard is that you expect the basement to flood once every 20 years, causing a complete outage of your data center. This means your Annualized Rate of Occurrence (ARO) is 0.05 basement floods/year.

      Further suppose that you expect a downtime from a basement flooding to last, say, 24 hours. That means your Single Loss Expectancy is your profits per hour, X, times 24 hours, let's call that Y.

      From Y and 0.05, we can calculate the Annualized Loss Expectancy, that is, the cost of a single occurrence times the probability of occurrence in any given year. So let's let Z be the ALE of (Y * 0.05).

      If the annualized cost of having an alternate data center to mitigate only the risk of flooding exceeds Z, the Annualized Loss Expectancy, you do not invest in an alternate data center, because it makes no business sense. You just take the loss when it happens, because it's cheaper than dealing preventing it.

      Of course, it's *never* quite this simple, and sometimes the SLE is essentially infinite (such as when loss of life could occur) and thus you spare no expense in mitigating the risk. Sometimes, you can't easily quantify the cost, because it isn't always money, it could be, for example, reputation.

      But it is *never* a foregone conclusion that you should automatically spend money mitigating risk without first thinking about if the mitigation costs more than the risk itself.

  • Shit happens (Score:2, Interesting)

    by Anonymous Coward

    Sometimes it's cheaper to deal with it when it happens than to take precautions.

    • Re: (Score:2, Insightful)

      by KevinKnSC ( 744603 )

      I don't disagree with you, but I strongly suspect this will be one of those times that it really would have been worth it to take precautions.

      • Not if it's just some power supplies and UPSs that got wrecked. A duplicate datacenter just amounts to pre-emptive purchasing of the very same equipment, plus more they wouldn't have ended up using.
    • Yeah, like that oil spill.

      • Re: (Score:3, Interesting)

        by kiwimate ( 458274 )

        Exactly. "Principles of Corporate Finance", Brealey/Myers/Allen.

        Major public companies typically buy insurance against large potential losses...

        BP has challenged this conventional wisdom...

        BP...took a hard look at its insurance strategy...BP decided not to insure against most losses over $10 million. For these larger, more specialized risks BP felt that insurance companies had less ability to assess risk and were less well placed to advise on safety measures. As a result, BP concluded, insurance against large risks was not competitively priced.

        How much extra risk did BP assume by its decision not to insure against major losses? BP estimated that large losses of above $500 million could be expected to occur once in 30 years. But BP is a huge company with equity worth about $200 billion... BP concluded that this was a risk worth taking. In other words, it concluded that for large, low-probability risks the stock market was a more efficient risk-absorber than the insurance industry.

        Now we get to see how well their hedging will work out.

    • Re: (Score:3, Interesting)

      by Monkeedude1212 ( 1560403 )

      I have not seen a case with a Data Center where this has EVER been the case.

      Essentially, look at it this way:

      Everything gets destroyed. You have to buy new ones. Then you have to start all over. It took you 3 days to get operational again, and now you've got to some-how start over from scratch.

      Or what they could have done:

      Everything gets destroyed. You already bought a spare system just in case, and had it stored off-site. For about a negligable amount, you went through the effort of backing up the data onc

    • "Sometimes it's cheaper to deal with it when it happens than to take precautions."

      It certainly makes for enhanced Schadenfruede.

  • Backups (Score:2)

    by vlm ( 69642 )

    The county does not have a backup data center

    Traditionally, facilities with that level of management oversight don't even have backups. No not backup centers or facilities or hardware, I'm talking about backup tapes. Am I right or wrong?

    I know rotating hard drives don't like immersion. Are SSDs any better, or do they die from galvanic corrosion? It might be an interesting race, which will survive longest.

  • Sewer line (Score:4, Funny)

    by Anonymous Coward on Monday June 07, 2010 @05:29PM (#32489470)
    At least it wasn't a sewer line break, that would be real shitty.

  • The county does not have a backup data center, despite warnings that it faced the risk of service disruption without one.

    That alone spells disaster. If I had a nickel for everything that could go wrong in a data center, I could buy a new data center.

    • No, because inflation would render those nickels worthless.

      • Re: (Score:2)

        by vlm ( 69642 )

        No, because inflation would render those nickels worthless.

        Recently, the value of the nickel content in a nickel briefly exceeded 5 cents. If the cost of everything, including nickel metal, exploded up by a factor of ten, then you'd be able to buy nickels at the bank or whatever for 5 cents and melt them into nickel ingots worth 50 cents. You'd be OK. Metals "always" hold value during inflation. Even hyperinflation, in that case especially if the metal happens to be cast and jacketed lead...

        • Re: (Score:2, Insightful)

          by Monkeedude1212 ( 1560403 )

          The problem is that not all nickels are full nickel anymore, just like not all pennies are made from pure copper.

          I know people who go searching for the older coins simply so they can melt them down into ingots worth more than the cash value of the coins.

          The newer stuff just isn't worth it though.

          • Comment removed based on user account deletion

            • The metal in a penny is already worth more than a penny. Congress solved this dilemma by making it illegal to melt down pennies to resell the metal.

              There's a plan to move all cash transactions to 5 cents by making them all round up or down appropriately, while having all electronic transactions remain accurate to the penny. But, as you can guess, certain members of Congress who are more attached to sentimental ways vowed to save the American penny, and so they instead outlawed melting them down as a I men

            • They will like it just fine. The rule will be if $amount is less than or equal to .0499 round down in favor of bank. It's like The superman with Richard Pryor all over again.

  • Tested Backups? (Score:2, Redundant)

    by Ichijo ( 607641 )

    Officials stressed, however, that no data was lost during the shutdown because of automatic nightly backups at an off-site location.

    Next question. Has the backup and restore process been tested?

  • Something like that happened to us... (Score:3, Insightful)

    by snowraver1 ( 1052510 ) on Monday June 07, 2010 @05:36PM (#32489548)
    About a year ago...

    At the time, we had all our WLAN connections carried through Bell Canada VIA Frame circuits. I guess many of these circuits went through a facility in Edmonton. This facility was being rennovated, and some poor worker drilled through a pipe that they thought was empty... As it turns out, that pipe was filled with pressurized water, and so the water started spraying everywhere/everything and ended up taking down all our frame services north of Edmonton (about 30 sites). It took about 2 (very stressful) days for Bell to route our frame circuits through another data center.

    It sucked, but I really feel bad for the poor guy that drilled through the wrong pipe.
    • BTW... WLAN - Our Greater LAN that spans thousands of KMs, not the wireless...

    • Re: (Score:3, Interesting)

      by vlm ( 69642 )

      At the time, we had all our WLAN connections carried through Bell Canada VIA Frame circuits.

      Was it ever possible to buy a frame relay switch that doesn't have an automatically rerouting ATM backbone as the underlaying technology? As far as I know, the answer is no.

      ended up taking down all our frame services north of Edmonton

      Somebody's got a single point of failure in Edmonton. Huge design mistake, not inherent technological limitation.

      I worked for a carrier for many years that was properly designed. PVCs would drop over the dead trunk and reroute over the live trunk transparently, assuming you were using a real packet based protocol like TCP/IP and not S

  • Who's idea... (Score:5, Insightful)

    by Pharmboy ( 216950 ) on Monday June 07, 2010 @05:39PM (#32489594) Journal

    Whose bright idea was it to put the UPS and backup systems in the portion of the building that is first to be flooded, and the most devastated in just about any natural disaster, AND the least accessible afterward? Sounds like something a government would do....

    • Re:Who's idea... (Score:5, Insightful)

      by NewWorldDan ( 899800 ) <dan@gen-tracker.com> on Monday June 07, 2010 @05:49PM (#32489688) Homepage Journal

      .... and it's also the part of the building that's easier to cool and isn't in demand for office space. A lot of businesses put their data centers in basements. I've seen a few places that built dedicated buildings for the data center, but usually, cost dictates that they stick it where they can.

      Frankly, while it will be a pain in the butt for 2 weeks, they'll get through this just fine. If they had a redundant data center, people would be whining about the waste of money and so on. There's no right answer here.

    • Re:Who's idea... (Score:4, Interesting)

      by fuzzyfuzzyfungus ( 1223518 ) on Monday June 07, 2010 @05:50PM (#32489698) Journal
      It's a terrible plan; but putting the heavy stuff in the basement makes life easier(unless your building was purpose-built, or you have serious renovation funds, putting thousands of pounds of UPSes on one of the upper floors isn't always one of your choices). And, more generally, there seems to be this perverse part of human nature that clings to the atavistic belief that stuff you don't have to look at isn't an issue. We put the ugly, heavy, parts of the system in the basement, we bury dangerous chemical wastes(where it is virtually impossible to inspect them for leakage, and all leakage goes directly to the water table) rather than putting them on the second floor(where discovering leaks is as easy as walking through the first floor)...

    • I would guess a gov't worker

      • Re: (Score:2)

        by Jawnn ( 445279 )

        I would guess a gov't worker

        Right. That would be the "conservative" fuck-wit that has been campaigning on "lower taxes" for the last twenty years. He "saved" the taxpayers hundreds of thousands of dollars by funding a half-assed infrastructure that couldn't survive a single fault in a critical system.

        • How fucking stupid are you? You can't blame every thing on Bush and Republicans. I don't even like Bush (or the current Republicans) but only a fuck-wit would think that all of life's problems are caused by one political party. Are all the tornadoes and floods the fault of GW Bush as well?

    • Re: (Score:3, Interesting)

      by Nemesisghost ( 1720424 )
      Actually most buildings in Texas are built where the basement is the most structurally sound part of the building. This is because the biggest 'natural' disaster are tornadoes. This is especially true of older buildings(like most government buildings). There's only one state run building I know of that was built any other way. It's one of the state backup centers in West Central Texas. They spent 2-3x the amount of a comparable building so that the top floor could withstand hurricane force winds. They
    • Some of the replies are saying the basement is a fine place to put these things, I can appreciate that. I'm wondering if this is less a problem of basement and more a problem of sewer. I don't know about Dallas, but I live in Houston, and Houston has the most terrible sewers I've ever seen. Storm drains are located very far away from each other, and the amount of space underground for carrying water seems to be miniscule. Once a year or so, we get just a day's worth of heavy storms which result in flash

    • Well many businesses locate such stuff at the lowest levels to protect it from severe weather.

      Yet I have met many people in IT from small government IT setups who never had backups, some even had charged water sprinkler systems in their data centers. Many vendors sell systems without backups because if the cost of the backup systems were included they would be too expensive. Plus never overestimate the ability of people to put off to the next group to follow to finish their job or suffer from the incomple

    • Whose bright idea was it to put the UPS and backup systems in the portion of the building that is first to be flooded... Sounds like something a government would do....

      Now if only we could re-play the situation with the equipment on an upper floor and destroyed by a tornado. I'll bet your response would be identical.

    • Re: (Score:2)

      by antdude ( 79039 )

      Whose, not who's.

      I am not surprised by the government. Probably a bunch of non-computer people!

  • Comment removed (Score:5, Insightful)

    by account_deleted ( 4530225 ) on Monday June 07, 2010 @05:53PM (#32489750)
    Comment removed based on user account deletion

    • Re: (Score:2, Informative)

      by Anonymous Coward

      In what world do you live in that the government is ultimately responsible for a massive oil spill that was caused by a corporation?

      While the government may be responsible for the fact that the regulations weren't followed every step of the way, the company that is behind the actions on the rig are ultimately fully responsible for not following those rules and regulations. I'm glad that the government can help in any way possible, but BP should foot the bill.

      • Tell that to Bobby Jindal.

      • Re: (Score:2)

        by Jawnn ( 445279 )

        While the government may be responsible for the fact that the regulations weren't followed every step of the way, the company that is behind the actions on the rig are ultimately fully responsible for not following those rules and regulations. I'm glad that the government can help in any way possible, but BP should foot the bill.

        Wanna bet? Seriously. If you really believe that (and a brief review of the history will clearly show how "responsible" the oil industry has been when it comes to their messes), I'll even offer generous odds.

    • It's entirely possible that the reason they're complaining is that they want a smaller government with less things they're responsible for, so they'll have fewer things they're capable of screwing up, and want the things that government does taken over by private industry. Having a larger government does not necessarily mean having a more prepared government - it generally means having more layers of bureaucracy and more people who's jobs are not directly tied to their performance.

      While I don't promote hand

      • Having a larger government does not necessarily mean having a more prepared government - it generally means having more layers of bureaucracy and more people who's jobs are not directly tied to their performance.

        And this is different from larger industry how?

    • What your conservative friends are objecting to is a massively bloated government that appears to be completely inept at EVERYTHING regardless of how much money we give them.

      If they can't be prepared with the tens of billions that they spend NOW what makes you so sure that they would do any better with additional billions?

    • Re: (Score:3, Interesting)

      by scamper_22 ( 1073470 )

      That is a very false false dichotomy... and a very poor argument.

      Your basic argument is that people who favor small government expect the government to do nothing.
      It's like talking about small government, and someone says: don't you like safe food inspections?
      Yes, I 'like' those things. I also don't mind the government doing them. That's why I believe in small government. Not no government.

      Just take a look at the government's spending. The things 'small government' folks want the federal to do would cos

  • Well sirs I have this here deed that says I own the Cowboys Stadium. What do you mean it looks like I typed it up? You got any proof I don't own this here stadium? OK Then. lol can see it now.

  • Someone forgot the rules... (Score:3, Insightful)

    by U8MyData ( 1281010 ) on Monday June 07, 2010 @06:24PM (#32490042)
    From the movie Contact: "First rule in government spending: why build one when you can have two at twice the price?" Load balanced and replicated no less. Basement + critical systems = bad idea. Not only for flooding potential, but bad sewer systems as well. Yes been there done that. Curious, is there a systems engineer that could make a good argument for building data center infrastructure in a basement. Two points already for shielding from severe weather; anything else?

    • Re:Someone forgot the rules... (Score:5, Insightful)

      by Renraku ( 518261 ) on Monday June 07, 2010 @06:44PM (#32490216) Homepage

      Climate control is easier in the basement. You can build big fuckoff heat exchangers that go under ground level and surface however far from the building you want them to surface.

      Simpler wiring plans because you don't have to run big industrial power cables up to the top floor and the data lines don't have to go far to get to the basement.

      All that being said, below-ground server rooms should have some method to be able to seal themselves off from the rest of the world in case of flooding. Perhaps the elevator or hallway door can form a decent seal, whereas everything else is already as sealed as it can be. Perhaps sealing everything also cuts power so nothing overheats.

      • Re: (Score:3, Insightful)

        by Locke2005 ( 849178 )
        Climate control is easier in the basement. Assuming you are using ground source heat exchange, yes. Otherwise, it's about as far as possible from the A/C heat exchangers on the roof. Which do you think Dallas was using?

    • Re: (Score:2)

      by sn00ker ( 172521 )
      Weight. Unless you're building from scratch, with a hefty engineering budget, putting heavy shit higher than the foundation level means lots of load-bearing components. That means losing lots of space in the floors below, due to the greater size and/or density of pillars, and also increases significantly the cost of the floors that must take this weight.

      If your DC is in the basement, or whichever floor is closest to the foundations, you've got the in-built load-bearing functionality of the whole planet, a

  • Confidentiality? Check!
    Integrity? Check!
    Availability? ... Can I borrow a pen?

  • I was at a $BigSwitchMakerResearchSite and they located their servers on the second floor of a 3 story building. They had a flood in the area in the past which wiped out their first floor data center. Since this place was also in tornado country some one made a sane and sensible decision to locate all the important servers on the second floor away from the outer walls.

    Take heed all you junior sysapes.

  • Some clarifications (Score:5, Insightful)

    by rabtech ( 223758 ) on Monday June 07, 2010 @07:57PM (#32490780) Homepage

    First, this is Dallas County, not Dallas city.

    Second, they knew about the potential for failure and were working on setting up a backup data center. TxDOT denied them rights of way to lay fiber along the highway into a facility in Tarrant county, so they were looking at other potential sites in Garland. Unfortunately this happened before they got it all resolved.

    TxDOT might have had good reasons for denying the request, I don't know, but I would wager that the backup site would be a lot further along if they had been able to run that fiber. Sometimes you know there is a problem, management agrees, and you even have a budget to fix it... but someone else (another department, another company, a government agency, etc) stands in the way.

  • Dont worry (Score:2, Funny)

    by AHuxley ( 892839 )
    They nice people from a regional http://en.wikipedia.org/wiki/Fusion_center [wikipedia.org] will have all the data backed up for any state database. eg http://www.fusionsystem.us/ [fusionsystem.us]
    Staffed by IT experts eg the NSA, they all have clearance and just love to take good care of all of your regional data 24/7.
    Been state and federally funded they will have fancy off site backups in real time.
    Many snapshots of the data will be flowing around the USA at any time.

  • Root Cause (Score:3, Insightful)

    by ThatsNotPudding ( 1045640 ) on Tuesday June 08, 2010 @07:44AM (#32494272)
    From TFA: "The 90-year-old water main ruptured..."

    90 years old because taxpayers (read: well-heeled conservatives) never want to pay for maintaining and replacing infrastructure until after the disaster occurs. No doubt they will somehow get federal funds to help defray the costs - all the while cursing the federales' very existence.

Slashdot Top Deals

It is easier to write an incorrect program than understand a correct one.

Close