Can Maintenance Make Data Centers Less Reliable? 185
miller60 writes "Is preventive maintenance on data center equipment not really that preventive after all? With human error cited as a leading cause of downtime, a vigorous maintenance schedule can actually make a data center less reliable, according to some industry experts.'The most common threat to reliability is excessive maintenance,' said Steve Fairfax of 'science risk' consultant MTechnology. 'We get the perception that lots of testing improves component reliability. It does not.' In some cases, poorly documented maintenance can lead to conflicts with automated systems, he warned. Other speakers at the recent 7x24 Exchange conference urged data center operators to focus on understanding their own facilities, and then evaluating which maintenance programs are essential, including offerings from equipment vendors."
In between maybe? (Score:5, Insightful)
Maybe there's a sweet spot between "no testing at all" and "replacing everything every three months"? In my experience, there is a lot of work to do in most places to make sure that proper testing is done, or at least that emergency procedures are known and people are well trained in them. Very often documentation is lacking and the onsite support staff have no clue where that circuit breaker is. That is the most common scenario in my experience, not overzealous maintenance.
Re:In between maybe? (Score:5, Interesting)
I suppose that I'd agree. Back in the early 90s, I inherited from a friend a fear of rebooting, turning off, or performing maintenance on a computer. Half the time he opened the case, the computer would become unbootable or never turn back on. Luckily, as a talented engineer, he could usually fix whatever the problem was, but it was a huge pain in the ass. Of course, back then, commodity computer hardware was hugely unreliable, with vast gaps in quality between price ranges, and we were working with pretty cheap stuff. Still, to this day, I dread the thought of turning off a computer that has been working reliably. You never know when some piece of crap component is nearing the end of its life, and the stress of a power cycle could what pushes it over the edge into oblivion (or highly unreliably behavior). I used to be fond of constantly messing with everything, fixing it until it broke, but his influence moderated that impulse in me, to the point where I usually freak out when anyone suggests unnecessarily rebooting a computer. Surely, there's something to say for preventative maintenance, and I'd rather be caught with an unbootable PC during regularly scheduled maintenance than suddenly experiencing catastrophic failure randomly, but there's something to be said for just leaving the shit alone and not messing with it. Every time you touch that computer, there's a slight chance that you'll accidentally delete a critical file directory, pull out a cable, or knock loose a power connector. The fewer the times you come into contact with the thing, the better. If I could build a force field around every PC, I probably would.
Re:In between maybe? (Score:5, Funny)
fixing it until it broke
Thats the spirit!!
Re:In between maybe? (Score:5, Interesting)
i can't agree. i used to but now i cannot afford to.
we recently experienced 2 catastrophes (datacentre-wide downtimes, you know things that NEVER happen) and the results were unbelievable. GRUBs failed to load OSes, machines were without a bootloader (due to emergency disk hotswaps), some machines simply didn't turn on, services didn't autostart, a few virtual servers autostarted on multiple hosts (instead of just one), fsck on some of our volumes took hours to finish, 30% of supermicro IPMI cards were unresponsive, etc. it revealed that almost nobody had followed procedures properly.
after that, every single service we have is built in a clustered manner with nodes spread across multiple datacentres. I now restart machines and pull cables at regular intervals to test bgp/ospf, clustering, recoveries, to check filesystems, etc. i am now also ABLE TO SLEEP.
Re:In between maybe? (Score:4, Informative)
===
Back in the early 90s, I inherited from a friend a fear of rebooting, turning off, or performing maintenance on a computer. Half the time he opened the case, the computer would become unbootable or never turn back on.
===
Neither you nor your friend are alone in thinking that:
AD-A066579, RELIABILITY-CENTERED MAINTENANCE, Nowlan & Heap, (DEC 1978) [this used to be available for download from the US Dept of Commerce web site; now appears to be behind a US government paywall (!)]
A more recent summary:
http://reliabilityweb.com/index.php/articles/maintenance_management_a_new_paradigm/ [reliabilityweb.com]
sPh
Re:In between maybe? (Score:5, Informative)
Re: (Score:2)
Thanks!
sPh
Re: (Score:2)
soft vs hard reboot (Score:2)
Some times software / os needed at least a soft reboot from time to time to clean up stuck software and remove memory leaks.
Now some stuff like firmware updates may need a hard reboot.
As for power cycleing some times you need to do it to get back into a crashed system.
Re: (Score:2)
Some times software / os needed at least a soft reboot from time to time to clean up stuck software and remove memory leaks.
What operating system are you using, Windows 98? The worst case I've had with Linux (CentOS 5.4) is NFS locking up and taking one of the CPUs for a ride to jump the load to around 8 to 9. Even then, a little patience killed the process after a while. There have been times when it was FASTER to hard boot (but the risks suck), but most modern applications and operating system ON THE S
well some windows updates still need reboots less (Score:2)
well some windows updates still need reboots it's less then it was in the past but still more then with linux.
Also a lot of NON OS software updates / installers at least say they need a reboot.
Re:soft vs hard reboot (Score:4, Insightful)
You must not deal with any Oracle database servers. They leak like a sieve.
[John]
Re: (Score:3)
Re: (Score:3)
NFS locking up is ultimately a part of the spec. It was originally a stateless filesystem that operated over UDP. Unless you're using a more recent revision of the protocol and have it configured as such, you're going to have issues with it locking up regularly.
Re: (Score:3)
Desktop PCs and servers seem to have largely overcome the need to reboot regularly, but other segments of the industry seem to be moving backwards. My Android handset actually says in the manual that you should power cycle it regularly. With a firmware upgrade, it even started giving me a warning from time to time, telling me I had not power cycled the phone in X amount of times and that I should do that now or risk instability. (Am I crazy for assuming that a phone OS is a markedly less complex environmen
Re: (Score:2)
Zombies on a SERVER, no. (the topic of the article) I'm more conservative what I run on a server, however.
Re:In between maybe? (Score:5, Interesting)
"If it's not broken, don't fix it."
Re:In between maybe? (Score:5, Insightful)
Do you know why satellites last so long in a hostile environment?... because nobody touches them.
"If it's not broken, don't fix it."
Actually I'm pretty sure it's the millions that are spent engineering each individual one so that it specifically can survive many years in said hostile enviroment.
If we spent anywhere near what is spent on proper engineering in time and money, everyday crap would be pretty damn reliable too, just not nearly as cost effective
Re: (Score:2)
Actually, the components would be pretty close to as cost effective. The executives just wouldn't get their uber-fat rewards.
Re: (Score:2)
how many are hooked into the internet running an os that gets attacked?
Re: (Score:3)
I am still that way with firmware upgrades. I think it probably has something to do with our generation. In the 90s, computer hardware was touchy and was expensive to replace. If you're like me, you probably grew up blowing into Nintendo game cartridges when they did not work. But back to firmware, I only upgrade it when necessary. Over the last fifteen years I have seen too many firmware upgrades bork hardware that was working just fine. With security patches I do them monthly, but not firmware. And
The threat today is automated updates (Score:2)
One bad automated update can lead to your system hosed or obscure reliability problems, perhaps not showing up for a while and the worst ones again leaving you with little option but to rebuild a system.
So I turn off auto update on everything I can, and manually update periodically. I consider the security risk smaller this way. I get it stable, and let it run that way for a few months at least. Then update security fixes etc.
Re: (Score:2)
Re: (Score:2)
Re:In between maybe? (Score:5, Insightful)
if that's the case, you don't have CONTROL over your equipment.
That was acceptable for Windows 95 but not even for desktop PCs anymore, let alone server equipment. My opinion is that your equipment isn't stable UNTIL you can turn it off and on again reliably. And yes... that is an ENORMOUS amount of work.
If you can't reliably replace individual pieces then you don't have control for maintenance... sure you can stick your head in the sand and just not touch anything... but that's just piling up all the things you didn't take time to figure out until come critical time later.
Re: (Score:2)
Re: (Score:2, Interesting)
If your buying new or refurbished electronics are THAT unreliable, why the !%!@#$!@%! are you using them?
If a router fails to come up because a cap is ready to blow, what happens when it blows WHILE IT'S RUNNING?
I had that happen with 2 Cisco ASA firewalls. One was 5 years old, the other was a few months. They were using HSRP and decided fighting amongst each-other for control was a great idea because one of the ports was going out. We took the old one offline; wouldn't turn on anymore. The new one? Wo
Re: (Score:2)
I have seen the same, my long running hard drives wouldn't boot up again. When I opened one of these stopped hard drives the head was glued to the boot sector. I assumed that since boot sector is never used during the 3-5 years of operation it collects some trash that will attach to the reading head when HD is stopped and cools down.
Sometimes moving the case is enough to get bad connections appear (or opposite, when computer is brought for maintenance it starts to work) but pressing down all the cards and c
Maintenance and prevention are not always the same (Score:4, Interesting)
Re: (Score:3)
After that blunder, I was asked to check on all the cameras servers once a week and make sure I could actually open up and view recordings from days past. This is a preventative action, but not really a maintenance one.
No, it's not preventative. It does nothing to prevent the problem. It detects the problem earlier (before, say, a business user does). That's monitoring. It's proactive, not reactive - perhaps that's what you mean?
Re: (Score:3)
===
No, it's not preventative. It does nothing to prevent the problem. It detects the problem earlier (before, say, a business user does). That's monitoring. It's proactive, not reactive - perhaps that's what you mean?
===
It is deeply unclear whether what is traditionally termed "preventative maintenance" (intrusive work involving disassembling, eyeballing, software probing, etc) actually improves reliability over conditioning monitoring tests followed by break-fix work as described by the parent post. More
In the Army we had "PMCS". (Score:2)
https://en.wikipedia.org/wiki/Preventive_Maintenance_Checks_and_Services [wikipedia.org]
Or to use a more common example, think about changing the oil in your car every (time interval) or (distance interval). Will it stop failures? Maybe. Maybe not.
On the other hand, every time you "work" on a system you introduce entropy.
As long as you remove more entropy than you introduce, you should have a more reliable system (than if you hadn't worked on it at all). But that gets into the training/knowledge of the person performing th
Re:Maintenance and prevention are not always the s (Score:4, Interesting)
Planned obsolescence has been promoted in all aspects of life since post WW2 and now it is hard to imagine the world without it. That line of thinking has been creeping into everything even in areas where it doesn't seem to apply.
Does this play a factor on the perception of preventative maintenance or its frequent application? I think it probably does in at least a couple ways, don't you?
Re: (Score:2)
Internal monitoring of components is a lot better now than it used to be. We used to go around and check all of the supply fans once a month because it was the highest failure rate component on the desktops we were using and there was no indication until the machine started crashing from overheating.
Security updates (Score:5, Informative)
I can think of many occasions that a security update has broken a server/router/etc. Obviously the lack of a security update can lead to a bigger headache in the future. But the typical user doesn't understand and has the attitude "IT broke the server again".
If a virus or hacker causes an issue the attitude is "I hope they fix that soon. I hate viruses/hackers" (obviously this is a huge generalization).
Re: (Score:2)
Some vendors will push security updates and other updates through the same channel, sometimes even without the user knowing if a particular update is fixing a security problem. Occasionally the update will even be installed without the user's accept.
If it was just a matter of only installing updates to fix known security problems and no other changes were made to the software, I think cases
Reliability Centered Maintenance (Score:5, Interesting)
===
"Is preventive maintenance on data center equipment not really that preventive after all? With human error cited as a leading cause of downtime, a vigorous maintenance schedule can actually make a data center less reliable, according to some industry experts.'
===
It isn't just human error: the very act of performing intrusive tasks under the theory of "preventative maintenance" can greatly reduce reliability of systems built of reasonably reliable components. This was studied extensively by the US airlines, US FAA, and later the USAF in the 1970s when the concept of reliability centered maintenance was developed for turbine engines and eventually full airliners. Look up the classic report by Nowland & Heap. Very much counter-intuitive if one has been trained to believe in the classics of "preventative teardowns" and fully known failure probability distribution functions, but matches up well to what experience field mechanics have been saying since the days of the pyramid construction.
sPh
Of course, today there is a huge "RCM" consulting industry, 7-step programs, etc that bears little resemblance to the original research and theories; don't confuse that with the core work.
Re: (Score:2)
===
n the 1970s when
===
Sorry; that should be "1960s" not 70s.
Maintenance took down Chernobyl (Score:3, Informative)
Re:Maintenance took down Chernobyl (Score:5, Informative)
That being said, it was because their procedures were shit, not because they were doing maintenance.
Actually, no, the Chernobyl disaster was sparked with a 'live' test of a new, untested mechanism for powering reactor cooling systems in the event of a disaster that brought down the power grid. http://en.wikipedia.org/wiki/Chernobyl_disaster#The_attempted_experiment [wikipedia.org] (And even that test was delayed several hours, into a shift of workers that weren't properly prepared to conduct the test.)
Re: (Score:2)
Like the GP said, it was because their procedures were shit, not because they were doing maintenance.
Re: (Score:2)
Actually, it was pretty much everything. Flawed design, untested technology, substandard construction, cost cutting, politics galore, arrogant and overconfident leaders, nuclear cowboys with something to prove, unqualified staff, etc.
The disaster itself was huge, but I'm surprised that nuclear power has been as safe as it is given how badly humans are at coping with such enormous responsibilities. Looking bad at the the Communist years, having a culture driven by financial profit doesn't seem as bad as a
Re: (Score:2)
Unfortunately, that is where real world data and experience comes into play.
GIVEN that their procedures were shit, maintenance actually made things worse and thus cased Chernobyl.
Now theoretically, you just need better procedures to make maintenance be a net positive. However, that doesn't change the advice that you shouldn't do such maintenance... GIVEN that your procedures are bad.
Given that humans are error prone and the IT procedures are shit, it is probably good advice to not do the maintenance.
Re: (Score:3)
GIVEN that their procedures were shit, maintenance actually made things worse and thus cased Chernobyl.
I'm guessing you were going for the sarcasm points, but for those who don't know about nuke eng as much as myself and presumably scamper, they had perfectly good procedures for experiment engineering evaluation that they mysteriously chose not to follow, and there was no maintenance involvement at all. Its the opposite of what he was claiming.
The quickie one liner of what happened is a RBMK has an extremely sensitive control loop by the very nature of what it means to be a RBMK, and the engineers who know
Re: (Score:2)
I thought about it a bit, and it would have worked on a PWR, but it would have worked REALLY WELL on a BWR, if you can survive the pressure fluctuations, which it probably could have. Of course if they're not running the reactivity control loop past engineering, they're not going to run the hydrodynamic control loop past engineering, so they might have industriously found a way to blow themselves up that way too.
PWRs are dead stable, not terribly sexy, quite heavy and bulky, and have more moving parts. If
where's the car analogy? (Score:4, Funny)
The guy at the garage always recommends I do an $80 transmission flush.
Re: (Score:3)
More MBA Constultant BS... (Score:3, Interesting)
Seriously...I sometimes think the average IQ is dropping on a daily basis (and, yes, I get the irony)...Both with what I read, and my own experiences working in IT, I become more and more convinced that society will eventually collapse under the weight of bad advice from consultants (and, no, I don't own a fallout shelter)...and I spend more and more time thinking about ways that I can profit off of the stupidity of leadership.
Re: (Score:2)
Seriously...I sometimes think the average IQ is dropping on a daily basis (and, yes, I get the irony)...Both with what I read, and my own experiences working in IT, I become more and more convinced that society will eventually collapse under the weight of bad advice from consultants (and, no, I don't own a fallout shelter)...and I spend more and more time thinking about ways that I can profit off of the stupidity of leadership.
Read it [google.com] and grab your tin foil and duct tape. You're gonna need lots.
Another lesson relearned (Score:2)
Can faulty logic make data centers less reliable? (Score:5, Insightful)
From TFS:
"... poorly documented maintenance can lead to conflicts with automated systems ..."
That doesn't mean maintenance makes datacenters less reliable. It means cluelessness makes datacenters less reliable.
Sheesh.
Re:Can faulty logic make data centers less reliabl (Score:4, Insightful)
vigorous maintenance
excessive maintenance
poorly documented maintenance
Those are all qualified as out of the ordinary. Anything in excess (on either side of the scale, whether it is too much or not enough) is a problem. Of course maintenance must be performed, but I guess some data centers have a strange idea of best practices, or they do not follow them.
Re:Can faulty logic make data centers less reliabl (Score:5, Insightful)
Maintenance, like anything else you do in a datacenter or wherever you work, must be done correctly. If maintenance reduces the reliability of the maintained entity, then per definition, it was not correctly performed.
Doing something correctly requires knowledge, planning and training. Just like everything else.
Maintenance-induced failure. (Score:5, Insightful)
There's something to be said for this. Back when Tandem was the gold standard of uptime (they ran 10 years between crashes, and had a plan to get to 50), they reported that about half of failures were maintenance-induced. That's also military experience.
The future of data centers may be "no user serviceable parts inside". The unit of replacement may be the shipping container. When 10% or so of units have failed, the entire container is replaced. Inktomi ran that way at one time.
You need the ability to cut power off of units remotely, very good inlet air filters to prevent dust buildup, and power supplies which meet all UL requirements for not catching fire when they fail. Once you have that, why should a homogeneous cluster ever need to be entered during its life?
Re:Maintenance-induced failure. (Score:5, Insightful)
There's also been a shift in the mentality of how well computers operate. It went from not tolerating any kind of downtime to the Windows mentality of crashing and "That's just how computers are".
Re: (Score:3)
Re: (Score:2)
If you read the Unix Haters Handbook, you'll see that they notice that "Unix boots fast", a lot of times.
I wasn't using computers (even less "real" computers) by that time, so I don't know how reliable is their description. But they describe exactly that shift in mentality, and way before Windows time. Then free unixes came, and everything changed...
Re: (Score:2)
The future of data centers may be "no user serviceable parts inside". The unit of replacement may be the shipping container. When 10% or so of units have failed, the entire container is replaced. Inktomi ran that way at one time.
I saw a while back (probably a year or two) that this is the way Microsoft will run (runs?) their Azure systems.
By the time 10% of the units are not working, it may be time to upgrade to the latest technology anyway. If you exclude disks, then I am certain you could run such a container for more than 10 years that way.
Re: (Score:3)
Possibly because ten out of 100 units have failed because a $200 hard drive has failed in each one? Does that mean that the whole $100,000 cluster needs to be replaced? Spending $100,000 instead of $2000 is not a great decision.
Re: (Score:2)
My guess is, the logic is here:
If after a couple of years running a bunch of systems and 10% has already failed (in a certain short amount of time), the others will probably follow soon.
Re: (Score:2)
That logic applies to mechanical parts and electronic circuits that operate way above their current limit. It is not valid for electronic circuits operating with reasonable currents.
Try "If after a couple of years 10% has already failed, we still have a couple of years until the next 9% (10% of 90%) fail".
In other news ... (Score:2)
Rather simplistic, but... (Score:2)
If it isn't broken, don't fix it.
The key to achieving high uptime ... (Score:3)
... is actually quite simple: You keep your hands off the systems. Period.
In detail, you plan, install and _test_ your setup before it enters production. You make sure that you can survive whatever you throw at it wrt. errors and incidents. You then figure out how much downtime you are allowed to have according to SLA. You then divide this number into equal sized maintaince windows together with the customer. And then you adhere to these windows! No manager should ever be allowed to demand downtime out of band. Period. In between you basically minimize your involvement with the systems and plan your activities for the next scheduled closing window.
And you ofcourse only deploy stable, true and tested versions of software and operating systems. And even though your OS supports online capacity expansion on the fly, you really shouldn't use the capability unless you absolutely have to. Instead you plan ahead in your capacity management procedure and add capacity in the closing windows. And you do not test and rehearse failures! It only introduces risks ... besides that you have already tested and documented them. And as you haven't changed the configuration, there is no need to test again.
So in essence. Common sense will easily yield 99.9%. Carefull planning and execution will yield 99.99%. The really hard part is 99.999%... /zensonic
Re:The key to achieving high uptime ... (Score:5, Insightful)
Which means for every online server you need an offline test machine -- and a way to simulate the operating environment in order to test. Not many companies have the skill of cash to do that.
That's where planning comes in. (Score:2)
You only need a server for each item that is different. So if you standardize on hardware / OS then you only need 1 server to test hardware drivers and OS updates and so forth.
Beyond that, you really should have a test database system and a test app system. You never want to deploy updates into a production environment without going through a test system first (which is NOT the same as a development environment).
Virtual systems can help a lot with the server requirements. But you still need to understand th
Re: (Score:3)
And I would have my own personal unicorn that craps Skittles on demand. Also, I could eat candy and poop diamonds.
Meanwhile, here in the real world... systems experience unexpected failures that will require them to be patched/rebooted/etc at the most inconvenient of times.
Missing the point... (Score:2)
Obviously not doing maintenance is much worse than the risk incurred by doing maintenance. However in the 2 years of using the datacenter my company relies on, I can say the only 3 major outages have been due to non-routine maintenance. Once was during a power upgrade, the datacenter supposedly has redundant power company connections, but the plug was somehow pulled during the upgrade anyhow. Another was during network maintenance, our dual redundant internet connections turned out not to be so redundant wh
Far from obvious (Score:2)
===
Obviously not doing maintenance is much worse than the risk incurred by doing maintenance.
===
That's far from obvious, actually, and is demonstrably wrong for many types of systems and installations.
sPh
Re: (Score:2)
It is time to switch data centers. We're with AT&T (and AT&T is far from the best) and they do quarterly power tests without a single problem. They've done core infrastructure router upgrades with zero down time. All in all, I'm very happy with the service. Any competent co-lo provider should be able to handle the issues you've had without any hiccups.
Transfer switch ratings (Score:5, Interesting)
Check your transfer switch ratings. I guarantee it will be spec'd much lower than you think. The electricians think it'll only be switched a couple times in its life. The diesel service provider thinks you're running it twice a week. Whoops. If you run it once a week, it'll only survive a couple years, then you'll get a facility wide multi-hour outage. I've personally seen it over and over again over the past two decades. The best part is "we have a procedure" so it'll only be run during maint hours and the desk jockeys 200 miles away will run it rain or shine, so its guaranteed that the xfer switch destroys itself at 2 am during a blizzard and it'll take half a day to repair.
Very few xfer switches are more reliable than commercial utility power. Installing a UPS actually lowers reliability in almost all professional situations.
My favorite power outage was caused by a gas leak a couple blocks away, where the utility co shut down the AC and then threatened to take an axe to the gen/UPS if not also shut off. This was not in the official written report, just word of mouth.
Useless article with no data. (Score:5, Interesting)
I read through the entire article, and saw zero data to support his assertion. I'm sure he has the data, but the article didn't reference a single piece of it. Without any data to support the theory all we have is a fluff opinion piece. Shame on Data Center Knowledge for writing an article about a scientific investigation, and not presenting a single piece of scientific evidence.
This is well known from Formula One (Score:5, Interesting)
The purpose was partly to stop qualifying being its own arms race, with cars in completely different specification than for the race, and partly to reduce costs and the number of travelling staff. At the same time, "T Cars" --- a third car, available as a spare --- were banned, so that if a driver destroys a car in practice the team either have to rebuild it or not race. They're allowed to travel with a spare monocoque, but it cannot be built-up and it does not get pit space.
There were endless howlings from the teams, claiming that without a complete strip-down after qualifying, with a large crew working overnight to check everything on the car, reliability would go through the floor and races would finish with only a handful of stragglers fighting a durability battle (our US viewers may find this ironic in light of a certain US Grand Prix, of course).
The same argument was advanced, mutatis mutandis, over limitations on engines and gearboxes, limitations on the number of gear clusters available, limitations on certain forms of telemetry and a wide variety of "the cars can't just be left to run themselves, you know" interventions.
In fact, reliability is now far greater than ten years ago. It's not uncommon for there to be no mechanical retirements, certainly not from the longer-standing teams, and the days of engines imploding on the track are long gone. A front-running driver will probably only have one, if even that, mechanical DNF per season. The teams deliver a functioning car when the pit lane opens at 1pm Saturday, and that car then runs twenty or thirty laps in qualifying and sixty or seventy in the race, a total of perhaps 250 miles, without much maintenance work beyond tyres, fluids and batteries (section 34.1 on page 18 of the sporting regulations [fia.com]).
So again, we see that "preventative maintenance" turns out to really be "provocative maintenance", and leaving working machines alone is the best medicine for them.
Re:This is well known from Formula One (Score:5, Insightful)
Re: (Score:2)
Also don't forget driving style is modified. "F it, you guys are going to tear the thing down anyway so I'm gonna lean it out to sneak in a couple more laps per tank" is replaced with babying the car.
Re: (Score:2)
So they removed the maintenance between qualifying and race. That does not mean that the race team does not do "maintenance" between races. Yes there is "less maintenance" but not "no maintenance".
Re: (Score:2)
Another way of looking at it, is that it doesn't matter what the rules are, so long as everyone follows the same rules.
Shorter races allow people to run closer to the edge, longer races require conservative builds. In the end, all the teams have the same problems to overcome, so in theory, nobody has the competitive advantage.
This is a different situation than, say, international trade, where one country is forced to meet safety regulations and another is not. Now that's when you end up with "wrecks and b
"The most reliable machine... (Score:4, Funny)
...is the one farthest from the nearest engineer."
Consider The Pioneer and Voyager spacecraft and the Mars landers.
battery maintenance / changing out battery is nee (Score:2)
battery maintenance / changing out old battery's is need as a dieing battery can fail to work when need or at the worst they can have a explosion.
Classic article on this problem: ValuJet 592 (Score:2)
http://www.theatlantic.com/magazine/archive/1998/03/the-lessons-of-valujet-592/6534/ [theatlantic.com]
William Langewiche, Atlantic Monthly, "The Lessons of ValuJet 592". It was basically done in because it was transporting safety equipment itself, which was vulnerable to a hard-to-predict failure. The more complex we make air travel, with its multiple checks and layers of protection, the more opportunities for failure. Adding another check to avoid 592, as they did, creates yet another opportunity.
It is, as they say, a H
Its All In the Process ... (Score:3)
For example, when working for one of the big 401k's, the first DR exercise evaluated the data center completely being leveled and re-locating both technical services as well as the ~300 on site employees to another location. Long story short, the first exercise of this was scheduled for 2 days, and while it worked, we identified dozens of issues. We scheduled the next test 6 months later and addressed what we believed were all of the issues; on next test, we ran into perhaps ~10 issues. The next test we scheduled 3 months ahead and ran into ~2 issues. All awhile, things continue to change and innovation is occurring, change process control is ensuring that new things are being factored into the continual DR process/exercise. For a small telecom I worked for, the same type of testing was accomplished with ~2-3 week turn around time (smaller team, less change points, more dynamic response), but with same underlying principles.
Documentation of such things is critical, and employee turnover is often one of the greatest risk points. Having a diversified staff with overlapping knowledge should minimize the later risk to some degree, and if implemented fully, risk should be diminished.
So how does all this tie back into maint? Well, it is anticipated that if any system runs long enough, their will be opportunity for failure. It is preparation for when such failure occurs, one can balance the capability of providing a measured window of downtime (if any) and provide some degree of predictability (i.e. I test once a quarter). The counter to this can certainly be overzealous maint, so certainly their is a point to being reasonable. For example, what many of go through with our cars - the dealer wants us to come in every 3k miles for an oil change, whereas realistically most mfr's and my own experience dictates that ~5k (if not longer depending on circumstance) is much more cost effective. Either way, this is providing some degree of confidence that this should prolong engine life.
Re: (Score:2)
For example, what many of go through with our cars - the dealer wants us to come in every 3k miles for an oil change, whereas realistically most mfr's and my own experience dictates that ~5k (if not longer depending on circumstance) is much more cost effective.
LOL old Saturn cars were famous for a valve issue where the engine suddenly starts to burn about a quart of oil per 1k mile after about 125K miles of service ... engine capacity is 4 quarts oil... "most" owners don't even know what a dip stick is, much less how to read it... lots of saturn engines dead with an empty oil pan...
You'd be surprised how often the manufacturers actually know what they're doing with stuff like that.
Something I never understood about that whole mentality.. pay the bank $40000 in pa
It depends (Score:2)
If "maintenance" means doing a forklift upgrade of all the computer and networking equipment every year or two then of course your reliability is going to suck, especially the human error factor with all of that new, unfamiliar equipment.
On the other side of things if someone thinks that never changing the oil in the generator is going to make it more reliable then they're in for a surprise. When I think about datacenter "maintenance" I think: changing the CRAC air filters, cleaning any outdoor coils, chang
Illuminating web page (Score:2)
See especially the sections "How equipment fails" and "Operating Context and Functions"
Re: (Score:2)
Add some really heavy-duty math to that with "Mathematical Aspects of Reliability-Centered Maintenance" by H. L. Resnikoff ! Way over my head. But the basic idea is simple. In a reasonably well-designed system with reasonably reliable components, you have the least information about that which interests you the most: failure rates. Making standard probability-distribution failure analysis virtually impossible (even if one discards the questionable "everything has a bathtub failure distribution" assumpt
The quality of the people matters a lot (Score:4, Insightful)
Re: (Score:2)
Was just writing my posting while you did yours. I could not agree more. Additional aspects are
- Engineers and managers that try to justify their existence by performing a lot of maintenance
- Incompetence due to bad training, arrogance and inexperience
Example: I recently pulled an Ethernet cable with broken connector out of a mission critical server (was not in production, we were reviewing cabling correctness). Turns out that some brain-dead person did the cabling with old used cables. 1 Minute of downtime
Re: (Score:2)
This sounds very much like a phenomenon I called "al dente programming"; throw code at a problem until something sticks. There is very little thought to the consequences of actions and assumptions that any issues will be fixed later. If people would just slow down a bit, do some research and think about the action maybe they would be starting fewer fires that need to be put out. It is circular logic; I don't have time to think about something because it is a fire but not thinking about something creates fir
Depends on the people (Score:2)
If you have rushed, underqualified people do the maintenance, then sure, it decreases reliability. If you have careful, non-rushed and competent people doing it, I doubt very much that the same is true. These people tend to be a bit more expensive, but cutting cost in the wrong places is a traditional occupation of managers in IT.
Re: (Score:2)
===
If you have rushed, underqualified people do the maintenance, then sure, it decreases reliability. If you have careful, non-rushed and competent people doing it, I doubt very much that the same is true.
===
Go read some of the original references on Reliability Centered Maintenance, particularly the Nowlan & Heap report referenced upthread by multiple posters. Your basic assumption has been shown to be very often incorrect in practice.
sPh
Re: (Score:2)
I think what is more likely is that competence is overstated in practice. Nobody will admit they use low-competence people for difficult jobs. Also, competent people will take a mostly hands-off approach to maintenance.Of course if maintenance always means "change something", the assumption is wrong. But that approach to maintenance is the wrong one in the first place.
All I want... (Score:2)
It's funny...I have cases where that sort of filter exists for a bottom-mounted power supply, but the case's own fans? Have to take 'em apart to properly clean the filters. And please don't say "Just lug a vacuum cleaner around." - they rarely do a good job if they actually are "luggable"; they (in a rare phrase) d
When I was a computer tech... (Score:2)
... a very long time ago, we had a saying about this.
"It's called preventive maintenance because it prevents the computer from working".
oblig Dilbert (Score:4, Funny)
Re: (Score:2)
That's maybe the only opportunity one have to see Alice afraid :)
nothing new (Score:2)
Short term gain, long term pain (Score:2)
Whenever I hear this meme touted (and I've heard it a *lot* over the last 40 years) I immediately think -- someone wants to shave a few maintenance dollars, trading short-term gain for long-term pain.
Your money, your choice.
Slashdot turned into InfoWorld? (Score:2)
according to some industry experts.'The most common threat to reliability is excessive maintenance,' said Steve Fairfax of 'science risk' consultant MTechnology. 'We get the perception that lots of testing improves component reliability. It does not.' In some cases, poorly documented maintenance can lead to conflicts with automated systems, he warned. Other speakers at the recent 7x24 Exchange conference urged data center operators to focus on understanding their own facilities, and then evaluating which ma
Re:Can too much FU make BAR unreliable? (Score:2)
I mostly agree. (Score:2)
Just notice that there is some value in forcing your hardware to fail in a time your downtime will be cheaper. Also, if you are smart, you'll induce redundant components to fail in different times, so downtime will cost just the maintaince price.
Of course, places tha fail to see that they are antecipating failures by doing maintence won't ever plan for that.