Water Main Break Floods Dallas Data Center

miller60 writes "IT systems in Dallas County were offline for three days last week after a water main break flooded the basement of the Dallas County Records Building, which houses the UPS systems and other electrical equipment supporting a data center in the building. The county does not have a backup data center, despite warnings that it faced the risk of service disruption without one."

Shit happens

[Anonymous Coward]

Sometimes it's cheaper to deal with it when it happens than to take precautions.

Re:Silly rabbit.

[Anonymous Coward]

And the funding comes from where?

Re:Silly rabbit.

[vlm]

There should always be duplication of critical components of a system with the intention of increasing reliability of the system,

Yeah that might be the intent, but it only works if the combined reliability is higher than individual reliability. Transfer switches I'm looking at you! I have worked at numerous facilities with data centers, and inevitably the transfer switch is less reliable than either wall AC power, or the diesels (youch!). Yes I know exactly what I'm saying, that at every facility I've worked at, power reliability would have been higher without the transfer switch and the generators. But its politically incorrect as the rare wall AC power failure would be unacceptable unless we spent money on switches and gens. As long as you spend money on switches and gens, any low level of reliability is acceptable.

Your mileage may vary, maybe coastie cities have less reliable power. Don't know.

Re:Shit happens

[vlm]

The criminal justice system of Dallas - maybe not in the same category.

No. Real high availability is like the AC power to the respirator in a hospital surgical room.

The criminal justice system of Dallas was closed exactly one week ago today for the holiday. Giving the parole officers, judges, juries, attorneys off is no big deal. The inmates, unless they had a trial scheduled today, frankly probably won't know the difference.

Re:Shit happens

[Monkeedude1212]

I have not seen a case with a Data Center where this has EVER been the case.

Essentially, look at it this way:

Everything gets destroyed. You have to buy new ones. Then you have to start all over. It took you 3 days to get operational again, and now you've got to some-how start over from scratch.

Or what they could have done:

Everything gets destroyed. You already bought a spare system just in case, and had it stored off-site. For about a negligable amount, you went through the effort of backing up the data once a month, to either a spare hard drive, or to the old computer your IT admin has in his basement that he never uses. It'll take you 3 days to get operational again, but at least you'll have that data somewhere and you don't have to play catchup.

Or if they wanted to go crazy

Everything gets destroyed. They had a spare system set up with full redundancy in case this kind of disaster. The cost was high when setting it up, but you have essentially no down-time. Ideally, the 3 days you saved will cover the cost of the damaged equipment. Or if the data is particularily sensative, that is the cost you pay to keep it safe.

Re:Who's idea...

[fuzzyfuzzyfungus]

It's a terrible plan; but putting the heavy stuff in the basement makes life easier(unless your building was purpose-built, or you have serious renovation funds, putting thousands of pounds of UPSes on one of the upper floors isn't always one of your choices). And, more generally, there seems to be this perverse part of human nature that clings to the atavistic belief that stuff you don't have to look at isn't an issue. We put the ugly, heavy, parts of the system in the basement, we bury dangerous chemical wastes(where it is virtually impossible to inspect them for leakage, and all leakage goes directly to the water table) rather than putting them on the second floor(where discovering leaks is as easy as walking through the first floor)...

Re:Something like that happened to us...

[vlm]

At the time, we had all our WLAN connections carried through Bell Canada VIA Frame circuits.

Was it ever possible to buy a frame relay switch that doesn't have an automatically rerouting ATM backbone as the underlaying technology? As far as I know, the answer is no.

ended up taking down all our frame services north of Edmonton

Somebody's got a single point of failure in Edmonton. Huge design mistake, not inherent technological limitation.

I worked for a carrier for many years that was properly designed. PVCs would drop over the dead trunk and reroute over the live trunk transparently, assuming you were using a real packet based protocol like TCP/IP and not SNA/SDLC where a dropped packet means reset the SNA communications controller...

It sucked, but I really feel bad for the poor guy that drilled through the wrong pipe.

The dude with the drill might be in ice cold Edmonton water. But the dude whom designed in the single point of failure in Edmonton is probably in really hot water.

Re:Silly rabbit.

[Anonymous Coward]

In Texas, we are proud of our faith.

Which is incredibly funny, especially considering that pride is considered one of the deadly sins (along with lust, greed, gluttony, and several others that Texans are well known for). Apparently too many of you are too busy thumping your chests about how you are such good Christians to even stop to consider what it really means to be a good Christian.

Re:Silly rabbit.

[headhot]

Transfer switches are the bane of my existence. I work for a major MSO, and every site we have lost has been to a transfer switch problem. Equipment with 2 ps is the best solution, so you have have 2 sets of transfer switches, UPS, Generator, and Mains.

Re:Who's idea...

[Nemesisghost]

Actually most buildings in Texas are built where the basement is the most structurally sound part of the building. This is because the biggest 'natural' disaster are tornadoes. This is especially true of older buildings(like most government buildings). There's only one state run building I know of that was built any other way. It's one of the state backup centers in West Central Texas. They spent 2-3x the amount of a comparable building so that the top floor could withstand hurricane force winds. They didn't do it because it was easy or economical, they did it because if you want to remain safe during a major storm, you head to the basement.

Re:Silly rabbit.

[Anonymous Coward]

You no doubt have a UPS hold the data-center while the generators come up to speed. I know it might not be fun but what about a manual cut switch and just size the UPS to last long enough for whoever is on call to get there? Obviously you have to have at least one other person who also gets the page a responds if the primary on call does not indicate they have gotten to site after a set amount of time; and you want to make sure the jars in the UPS are tested often.

This is what we do and what most of the bigger shops I have seen do. We had an auto-switch but after its repeated failures determined it was best to remove it.

Re:Silly rabbit.

[jo_ham]

It is quite possible to take Christians seriously and still make jokes about them. Some of the most devout Christians I know make jokes about their faith and God all the time.

One of the defining human characteristics is being able to laugh at yourself.

I make similar jokes about Apple users, Linux users, Football fans, people who drink Mountain Dew, atheists, rock fans, sci fi nerds, people who watch reality TV...

Humour is part of human character. You appear to be missing yours. Perhaps you should pray for it to return.

Re:Silly rabbit.

[Anonymous Coward]

I'd like to see your joke about the atheist, drinking Mt Dew, while watch his favorite rock band during a football halftime show, while using his iPad to schedule his Linux DVR box for the next Stargate.

Re:Prepared government or small government pick on

[scamper_22]

That is a very false false dichotomy... and a very poor argument.

Your basic argument is that people who favor small government expect the government to do nothing.
It's like talking about small government, and someone says: don't you like safe food inspections?
Yes, I 'like' those things. I also don't mind the government doing them. That's why I believe in small government. Not no government.

Just take a look at the government's spending. The things 'small government' folks want the federal to do would cost next to nothing.
Our biggest costs are healthcare, military...

So go ahead and cut those things down to focus on small government. It will free up hundreds and hundreds of billions of dollars.
Then, I'd be more than happy if the government spent what it costs to have proper data centers.

Re:Shit happens

[kiwimate]

Exactly. "Principles of Corporate Finance", Brealey/Myers/Allen.

Major public companies typically buy insurance against large potential losses...

BP has challenged this conventional wisdom...

BP...took a hard look at its insurance strategy...BP decided not to insure against most losses over $10 million. For these larger, more specialized risks BP felt that insurance companies had less ability to assess risk and were less well placed to advise on safety measures. As a result, BP concluded, insurance against large risks was not competitively priced.

How much extra risk did BP assume by its decision not to insure against major losses? BP estimated that large losses of above $500 million could be expected to occur once in 30 years. But BP is a huge company with equity worth about $200 billion... BP concluded that this was a risk worth taking. In other words, it concluded that for large, low-probability risks the stock market was a more efficient risk-absorber than the insurance industry.

Now we get to see how well their hedging will work out.

Re:Silly rabbit.

[victorhooi]

heya,

Haha, silly little boy...*grins*.

Ok, firstly, the seven deadly sins is actually a Catholic device, and even then it's just a categorisation thing, it's not Biblical, from what I can tell. The majority of Texans are, from memory, Evangelical Christians? As a Christian, a sin is a sin is a sin - they're all bad, and we all do them, Christian or otherwise. In the eyes of God, they're actually all "equally bad", if that makes any sense. I'm not going to go into a lengthy discourse on why, but ask any Christian, and they'll be happy to help answer your question.

Read the bible - there is only one unforgivable sin - and that is to blaspheme the Holy Spirit, such as the Pharisees did. Everything else is by definition, forgivable via Jesus. Also, put it this way, if you're worried you've committed the unforgivable sin, then by definition you haven't (I'm paraphrasing Larry Richards here).

And in terms of proud of their faith...err, there's nothing wrong with that. In fact, according to the Bible, your God is actually one of the few things you're meant to be proud of, that and being saved from sin by Jesus, and all that. The Bible repeatedly tells you to be proud about your God, the living God etc. etc.

So your feeble attempt to accuse Christians of not knowing their faith just fell flat on it's face.

Now, as a Christian, if you were to accuse us of other things, I'd be happy to entertain you, and indulge in self-reflection. However, please use an actual valid point if you're going to try to do that. We've done some questionable things in the past, and I'm sure we'll continue to stumble, and pick ourselves up, going forward. The one thing that differentiates us if that we're saved via Jesus, and that's an external thing.

So I don't think Texans (or Christians) are proud, in the sense that you might use the word pride (e.g. proud of what you've achieved), they're simply proud that God has saved them.

Cheers,
Victor