Water Main Break Floods Dallas Data Center 230
miller60 writes "IT systems in Dallas County were offline for three days last week after a water main break flooded the basement of the Dallas County Records Building, which houses the UPS systems and other electrical equipment supporting a data center in the building. The county does not have a backup data center, despite warnings that it faced the risk of service disruption without one."
Silly rabbit. (Score:5, Insightful)
There should always be duplication of critical components of a system with the intention of increasing reliability of the system, usually in the case of a backup or fail-safe.
Re:Shit happens (Score:2, Insightful)
I don't disagree with you, but I strongly suspect this will be one of those times that it really would have been worth it to take precautions.
Re:Silly rabbit. (Score:3, Insightful)
>And the funding comes from where?
If Dallas having a data center is not that critical, then never mind. Otherwise, from taxes, where civilized civilization usually comes from. But actually I just put that first post there to spoil it for prospective first posters. It was a copy/paste from the start of the wikipedia article on redundancy(engineering).
Something like that happened to us... (Score:3, Insightful)
At the time, we had all our WLAN connections carried through Bell Canada VIA Frame circuits. I guess many of these circuits went through a facility in Edmonton. This facility was being rennovated, and some poor worker drilled through a pipe that they thought was empty... As it turns out, that pipe was filled with pressurized water, and so the water started spraying everywhere/everything and ended up taking down all our frame services north of Edmonton (about 30 sites). It took about 2 (very stressful) days for Bell to route our frame circuits through another data center.
It sucked, but I really feel bad for the poor guy that drilled through the wrong pipe.
Re:Tested Backups? (Score:3, Insightful)
Who's idea... (Score:5, Insightful)
Whose bright idea was it to put the UPS and backup systems in the portion of the building that is first to be flooded, and the most devastated in just about any natural disaster, AND the least accessible afterward? Sounds like something a government would do....
Texas doesn't need backup. (Score:5, Insightful)
In a state as blessed as Texas, they were told that God would provide protection against acts of God. I imagine many of the faithful are confused, especially when Jesus day is only a few days away.
Maybe they didn't execute enough retarded people this year?
http://en.wikipedia.org/wiki/Jesus_Day [wikipedia.org]
Re:Silly rabbit. (Score:2, Insightful)
And the funding comes from where?
A redundant source of income that is.
Sadly, no matter how you design a system there is always a single point of failure. Just depends on how much you want to spend and what you want to risk going south.
Re:Seriously? (Score:2, Insightful)
The problem is that not all nickels are full nickel anymore, just like not all pennies are made from pure copper.
I know people who go searching for the older coins simply so they can melt them down into ingots worth more than the cash value of the coins.
The newer stuff just isn't worth it though.
Re:Who's idea... (Score:5, Insightful)
.... and it's also the part of the building that's easier to cool and isn't in demand for office space. A lot of businesses put their data centers in basements. I've seen a few places that built dedicated buildings for the data center, but usually, cost dictates that they stick it where they can.
Frankly, while it will be a pain in the butt for 2 weeks, they'll get through this just fine. If they had a redundant data center, people would be whining about the waste of money and so on. There's no right answer here.
Don't know (Score:1, Insightful)
Maybe they were more worried about tornadoes than floods there.
Of course the police/para/military mindset is bunker/underground everything critical as much as possible as well. I know my county just built a new jail/police station a few years ago and more than half that sucker is underground, you can see it from the road so it was easy to see while it was being constructed. It's built more like some fort than what you might think of as a traditional jail or copshop facility. Well sort of, it is half underground now, what they did was build it on level ground, the first level, then they bermed that over and built the next level you can still see now on top of it.
Comment removed (Score:5, Insightful)
Someone forgot the rules... (Score:3, Insightful)
Re:Texas doesn't need backup. (Score:3, Insightful)
By the way, what's wrong with re-writing the textbooks to eliminate the hardcore pro-materialist bias?
Pro-materialist bias? Is that what you ingrates are calling science and history now?
Re:Someone forgot the rules... (Score:5, Insightful)
Climate control is easier in the basement. You can build big fuckoff heat exchangers that go under ground level and surface however far from the building you want them to surface.
Simpler wiring plans because you don't have to run big industrial power cables up to the top floor and the data lines don't have to go far to get to the basement.
All that being said, below-ground server rooms should have some method to be able to seal themselves off from the rest of the world in case of flooding. Perhaps the elevator or hallway door can form a decent seal, whereas everything else is already as sealed as it can be. Perhaps sealing everything also cuts power so nothing overheats.
Re:Silly rabbit. (Score:1, Insightful)
Hey, give us some credit... we TRIED to send him back 4 years earlier, but we couldn't get everyone to agree.
Re:Texas doesn't need backup. (Score:1, Insightful)
Hey, I don't have to listen to spewing your blatant pro-reality propaganda. I like my fantasy world perfectly fine, thank you very much.
Re:Someone forgot the rules... (Score:1, Insightful)
It is probably cheaper to build a 2nd data center than make the 1st one submersible (that's what would happen during a flood ;)
Re:Someone forgot the rules... (Score:3, Insightful)
Some clarifications (Score:5, Insightful)
First, this is Dallas County, not Dallas city.
Second, they knew about the potential for failure and were working on setting up a backup data center. TxDOT denied them rights of way to lay fiber along the highway into a facility in Tarrant county, so they were looking at other potential sites in Garland. Unfortunately this happened before they got it all resolved.
TxDOT might have had good reasons for denying the request, I don't know, but I would wager that the backup site would be a lot further along if they had been able to run that fiber. Sometimes you know there is a problem, management agrees, and you even have a budget to fix it... but someone else (another department, another company, a government agency, etc) stands in the way.
Re:Texas doesn't need backup. (Score:5, Insightful)
We could re-write the textbooks to eliminate the bias toward truth.
It would only be fitting.
Re:Silly rabbit. (Score:5, Insightful)
There should always be duplication of critical components of a system with the intention of increasing reliability of the system, usually in the case of a backup or fail-safe.
Let me try to reply a bit more constructively than some of the others here.
It is never a foregone conclusion that you will always have duplication of critical components of a system, if you are doing proper risk management.
Essentially, the art of risk management is figuring out how far to go with mitigations of various risks.
To illustrate with an excessively simplistic example (Assume a perfect vacuum and a frictionless environment):
Let's say you sell something online, you sell W products/hour, and if you miss a sale, that's it, you're not getting it back.
So that means that you lose the profit on W products every hour, let's call that X.
Next, you look at the potential hazards, and calculate how often you expect to have each hazard occur per year. For example, to be simple, let's pretend your only hazard is that you expect the basement to flood once every 20 years, causing a complete outage of your data center. This means your Annualized Rate of Occurrence (ARO) is 0.05 basement floods/year.
Further suppose that you expect a downtime from a basement flooding to last, say, 24 hours. That means your Single Loss Expectancy is your profits per hour, X, times 24 hours, let's call that Y.
From Y and 0.05, we can calculate the Annualized Loss Expectancy, that is, the cost of a single occurrence times the probability of occurrence in any given year. So let's let Z be the ALE of (Y * 0.05).
If the annualized cost of having an alternate data center to mitigate only the risk of flooding exceeds Z, the Annualized Loss Expectancy, you do not invest in an alternate data center, because it makes no business sense. You just take the loss when it happens, because it's cheaper than dealing preventing it.
Of course, it's *never* quite this simple, and sometimes the SLE is essentially infinite (such as when loss of life could occur) and thus you spare no expense in mitigating the risk. Sometimes, you can't easily quantify the cost, because it isn't always money, it could be, for example, reputation.
But it is *never* a foregone conclusion that you should automatically spend money mitigating risk without first thinking about if the mitigation costs more than the risk itself.
Re:Silly rabbit. (Score:4, Insightful)
Seriously, have you ever actually read any respectable theology? No, probably not, which is why you promote these misinterpretations and equivocations. Go read some of the top theologians of our day, and then tell me how stupid you think it all sounds. You are not qualified to engage in reasoned discussion on this issue until you have.
And this, boys and girls, would be what sinful pride looks like.
Root Cause (Score:3, Insightful)
90 years old because taxpayers (read: well-heeled conservatives) never want to pay for maintaining and replacing infrastructure until after the disaster occurs. No doubt they will somehow get federal funds to help defray the costs - all the while cursing the federales' very existence.
Re:Silly rabbit. (Score:3, Insightful)
Seriously, have you ever actually read any respectable theology? No, probably not, which is why you promote these misinterpretations and equivocations. Go read some of the top theologians of our day, and then tell me how stupid you think it all sounds. You are not qualified to engage in reasoned discussion on this issue until you have.
I've read "respectable theology" and I'd like to submit the following quote for you to ponder: "the greatest single cause of atheism in the world today are Christians who acknowledge Jesus with their lips and walk out the door and deny Him by their lifestyle."
Churchgoing or considering myself of any particular religion really isn't my thing, I have tried it at least, but I've always believed that the surest sign of someone secure in their faith is that they can defend it without attacking others. If you consider yourself a good christian you should really evaluate your participation in today's discussion because you are seriously damaging the reputation of people who are better because of their religion.