Data Center Power Failures Mount 100
1sockchuck writes "It was a bad week to be a piece of electrical equipment inside a major data center. There have been five major incidents in the past week in which generator or UPS failures have caused data center power outages that left customers offline. Generators were apparently the culprit in a Rackspace outage in Dallas and a fire at Fisher Plaza in Seattle (which disrupted e-commerce Friday), while UPS units were cited in brief outages at Equinix data centers in Sydney and Paris on Thursday and a fire at 151 Front Street in Toronto early Sunday. Google App Engine also had a lengthy outage Thursday, but it was attributed to a data store failure."
If only you had listened... (Score:5, Funny)
-B. Mays
Re: (Score:1, Offtopic)
Re: (Score:2)
I'm guessing that the majority of these were caused by leaks or spilled drinks. If only you guys had listened to me and gotten Zorbeez(tm)[SOAKS UP 10x ITS OWN WEIGHT!] [wikipedia.org].
Even that wouldn't work. What you have here is your textbook Pepsi Syndrome and only a President in yellow booties can fix it.
Re:"bad week to be a piece of electrical equipment (Score:5, Funny)
Re: (Score:2)
Re: (Score:2, Funny)
Indeed, 18465. And we shall get off your lawn as well.
Re: (Score:1)
Re:"bad week to be a piece of electrical equipment (Score:5, Interesting)
Because out of all of the data centers in the world, there were problems at five? Riiiiight. Good reporting, Slashdot.
Can I sign up for broken water main notices here, too, or do I need to go to another website?
100+ million people daily are "serviced" by these 5 data centers.
Company's such as authorize.net where COMPLETELY unavailable for payments to hundred of thousands of webmasters sites (ya know the people who make money)
If you don't think this is serious news then you are still living at home.
Ya that's what I thought.
Re: (Score:3, Insightful)
Re: (Score:2)
The Fisher Plaza story is big. I happened to be walking by right after it happened, noticed the generators running and went, 'Hm-m-m". We've toured their facility in the past, and wanted to use them, but they didn't have capacity at the time. They seemed first rate. If a first tier provider can have this happen...
Re: (Score:2)
by right after it happened, noticed the generators running and went, 'Hm-m-m
You're some kind of witch aren't you? You broke my internet!
BURN THE WITCH!!!
Wrong (Score:2)
Damn you Michael Bay! (Score:5, Funny)
"A blown transformer appears to be the culprit"
I'd heard the new movie was crude, but I didn't realize how crude it actually was!
Re: (Score:2)
The good quote from the Transformers movie character Ratchet: Wow... that was tingly!
Re: (Score:2)
I guess Megan Fox's character is upgrading.
Re: (Score:1)
Re: (Score:2)
Methinks that electrical standards are falling behind the demand created by computing resources.
XO Communications Genesis Hosting (Score:1)
Re:XO Communications Genesis Hosting (Score:4, Funny)
Re: (Score:1)
Outages (Score:3, Interesting)
Re:Outages (Score:5, Interesting)
I've had equipment and/or worked in many datacenters over the last decade or so. I've worked with even more clients who have had equipment in other datacenters.
I've only experienced 3 power related outages that I can think of.
One was a brownout in that area, which cooked the contactors that switched between grid power and their own DC room.
One was an accident, where a contractor accidentally shorted out a subpanel, and took out about a row of cabinets. I was there for that one. I saw the flash out of the corner of my eye, and by the time I turned my head, he was just flying into the row of cabinets.
One was a mistake in the colo, where there was a mislabeled circuit, so they cut power to 1/3 of one of our racks.
There have been even more outages related to connectivity problems. With one major provider who was just terrible (and is now out of business), they had a fault about once a week or less. Every time we called, they said "there was a train derailment that cut a section of fiber in [arbitrary state], which effected their whole network." It was funny at first, but annoying when we started questioning them about why there was no news about all these train derailments. We had to make up our own excuses for the customers, because we couldn't keep telling them the BS story the provider gave. We were smart about it though, and at least had decent excuses, and the whole staff knew which BS story to give for a particular day. The sad part was, we had a T3, and that was huge at the time.
At my last job, they wanted a full post-mortum done on any fault. If a customer across the country suffered bad latency or packet loss, it was our job to find out why and "fix" it. The management wouldn't accept that there are 3rd party providers who handle some of the transit. So, we'd call our provider demanding it to be fixed (which they couldn't do), and then call the broken provider (who hung up since we weren't their customer), and then got reamed by the boss because we couldn't fix it. Delay tactics worked best after a while. If you're "investigating" a problem long enough, and hold the phone up to your ear enough, the problem will likely be fixed by those who really can. We'd still log a ticket with our provider, because the boss would eventually call the provider referencing the ticket number, and find out there was still nothing that could be done.
There's pretty much guaranteed to be a fault of some sort between two points on the Internet every day. All anyone can really do is make sure it isn't with your own equipment. That's something I always did before calling to complain about anything. It's embarrassing to hear "did you reboot your router?" and that turns out to really be the problem.
The only real solution to this is, redundancy. Not just in one facility, but across multiple facilities. If you spread things out enough, sure an isolated problem will effect some people, but not everyone. You want a service to be reliable, redundant machines in each datacenter is the only way to go. When I was running the network (and everything technical) at one job, a datacenter outage wasn't a concern, it was just a minor annoyance. I filed a trouble ticket, and told them to call me when it was fixed. We'd demand reimbursement on the outage time, and made them handle the difference on our 95th percentile bandwidth charges at the end of the month. I wasn't going to take a hit on the bill just because they had an outage in a city, and my other cities had to take the traffic during the outage. When your bill is measured in multiple Gb/s, you have a little more say in how they handle the billing. :)
Re: (Score:2)
I read that as contractors. I apparently saw 'contractor' in the next sentence and did the switcheroo. I was going to call you callous for using the term 'cooked'.
FYI, arc flash [wikipedia.org] is not something to be taken lightly (no pun intended). It's dangerous as all get-out in high voltage panels that have a lot of available fault current. A typical 480V, 20kA fault [wikipedia.org] can release the same amount of energy as 1.5 lbs of TNT.
Re: (Score:2)
I figured some people would make that mistake. Some others (like you) actually know the difference. Ya, I'd prefer to not have a cooked contractor. They kinda smell, and it tends to make other contractors not want to work with you. :)
I was very happy to have not been at the site when it happened, but I did have to fly up to check all of our equipment.
About half our servers wouldn't come back online, either due to power or networking faults. The network switch
Re: (Score:1)
it seems you had some bad experiences lately.
under normal circumstances you can always have 2 out of this 3 - regardless of which topic we are speaking (datacenters, code quality, cars*, ... - just name it)
*) and not even a bad analogy :)
Re: (Score:2)
Be Redundant! (Score:5, Insightful)
Re:Be Redundant! (Score:5, Informative)
Re: (Score:3, Funny)
In the event of a nuclear attack, you probably have more pressing issues to deal with than your server uptime.
Re: (Score:2)
Re:Be Redundant! (Score:5, Interesting)
Be nice, people don't read the books nor RFC's any more.
At the biggest operation I ran, I had redundant servers in multiple cities, and DNS servers in each city. If we lost a city, it was never a big deal, other than the others needing to handle the load. With say 3 cities, a one-city outage only accounted for a 16.6% increase in the other two. Each city was set up to handle >100% of the typical peak day traffic, so it was never a big deal. I don't think we ever suffered a two-city simultaneous failure, even though we simulated them by shutting down a city for a few minutes. Testing days were always my favorite. I loved to prove what we could or couldn't do. I peaked out one provider in a city once. We had the capacity as far as the lines went, but they couldn't handle the bandwidth. It was entertaining when they argued, so I dumped the other two cities to the one in question, and they were begging me to stop. "Oh, so there is a fault. Care to fix it?"
I could quantify anything (and everything) at that place. I could tell you a month or so in advance what the peak bandwidth would be on a given day, and how many of which class of servers we needed to have operating to handle it. I classed servers by CPU and memory, which in turn gave how many users and how much bandwidth each could do. I only wanted our machines to every peak out at 80%, but sometimes it was fun to run them up through 100%. I set the limits a little low, so we could run at say 105% without a failure.
Such information let us know if we had a server problem, before we knew we did. I'd notice a server was running 10% low, and that really means that it is going to fail. We'd watch for a little while, and it would. :) We'd power it down, and leave it in the datacenter until we had another scheduled site visit.
Re: (Score:1)
You're mistaken. Each of the other two cities would see their load increase 50%.
Re: (Score:2)
Normal Operations:
City 1 - 33.3%
City 2 - 33.3%
City 3 - 33.3%
City 1 stops:
City 1 - 0%
City 2 - 33.3%
City 3 - 33.3%
Other cities take up the slack:
City 1 - 0%
City 2 - 33.3% + 16.6% = 49.9% (mol)
City 3 - 33.3% + 16.6% = 49.9% (mol)
If you are really bent about the missing 0.2%, you can work it out in fractions instead. :)
City 1 - 0/3
City 2 - 1/3 + 1/6 = 3/6 = 1/2
City 3 - 1/3 + 1/6 = 3/6 = 1/2
I have a better time adding and rounding decimals in my head. Since nothing on the Internet is actually perfect, 1% off on
Re: (Score:1)
"City 2 - 33.3% + 16.6%" -- note that the load on City 2 just increased 50%. If it were making 33 widgets and had to make another 16 widgets, it's now making 50% more widgets.
To be more explicit, each city in your analysis is taking up 16.6% extra of the total load, which means that each city is individually seeing a 50% increase in its own load.
Re: (Score:2)
Well, ya, 50% more than it had been handling before. I see.
My concern was the percentage of totals. That was divided up by the available servers, and then each server was calculated to know it could handle. We had generally 3 classes of servers. They could be considered "old", "not so old", and "new". :) We didn't quite class them like that, but it would be a decent evaluation. If "old" could handle 1x, "not so old" could handle x2, and "new" could handle 3x. It wasn't u
Re: (Score:2)
Hehe.. Umm.. Well.. Ya, I guess we could. :) We never referred to it as that though. :)
I did turn them up to "lets see when smoke comes out", but somehow we never had the magic smoke released. Except that one new years, but an old server seemed like a great launch platform for fireworks. :) It was flat, hard, and cost a few thousand dollars. What better to put explosives on. :)
Re: (Score:2)
It's required that you have two name servers when you register a domain name.
Physical separation is not required. It's just good practice. (I do, in separate cities on different ISP networks) Having separate nameservers in different geo regions is implicit because you have to register at least two for each domain name. I've seen some people game this by having a single nameserver with two IP addresses, which strikes me as the height of stupidity, but it's not happening on my watch.
Re: (Score:2)
If everything referenced by the DNS records (web and email services or whatever) is hosted on the same machine as the name server, then it isn't particularly stupid. It's just a small operation that has a single point of failure; redundant DNS isn't going to change that.
Re: (Score:2)
If everything referenced by the DNS records (web and email services or whatever) is hosted on the same machine as the name server, then it isn't particularly stupid. It's just a small operation that has a single point of failure; redundant DNS isn't going to change that.
WITH the single exception I know of, that incoming email will bounce with something like "domain not found" if there is no DNS response at all, vs if there is DNS but the MX record servers can't be reached it'll silently retry. Some totally brain-dead MTAs will bounce, but anything remotely usable will transparently retry later and no one will know it happened.
And it's not so much a "small operation", as a non-relevant risk. People have a certain expectation of how (un-)reliable email is, due to filtering
Re: (Score:2)
Common myth but quite untrue (try it for yourself). If there is no response from any DNS server then it will be considered a temporary failure and delivery attempts will continue at intervals in the background just as if the MX target(s) were not responding.
Only if a server can be re
Re: (Score:2)
While geographic diversity is certainly an excellent goal, it's not always that simple. My ISP's network core was located in the Peer 1 suite at 151 Front (whose UPS caused the fire). Power was cut to Peer 1's suite, but not the rest of the building (151 Front has independent power/cooling/etc. per-suite to the extent where each tenant is responsible for getting their own solution).
Redundant power sources could have mitigated the issue had there not been a fire; running two independent circuits to critical
Re: (Score:3, Interesting)
Best solution for big outfits is to have at least this setup:
1) One party being the main contractor. This party doesn't do ANY hosting per say but only manages the fail-over strategy, doing the relevant testing once in a while.
2) A second party being involved in hosting and managing data centers.
3) A third party, completely independent from party 2, a competitor of 2 is preferable, which also does hosting and manages data centers.
It is the same principle when you bring redundant internet connectivity to a b
Re: (Score:2)
1) Have the fiber from one provider come into the building from, say, the north side of the building.
2) Have a competitor, unrelated business wise, that doesn't use the same upstream providers bring his fibers in from the South side of the building.
3) Discover that both fiber runs connect to the same L.E.C. vault 100 feet away and then run parallel the whole way back to the same central office, and/or they both are carried on the same SONET ring just connected to different ADMs (which would at least give you ADM redundancy).
Seriously though, step 3) is get a copy of the DLR / CLR of the local loop, and have someone analyze them. Of course how the circuit is designed is not necessarily how it is actually routed, which is even funnier.
Everyone in the t
Re: (Score:2)
President Jimmy Carter.
Nuclear attack evac test.
Lots of embarassed people.
It is *epically* difficult to get and keep true physical diversity.
Re: (Score:2)
Re: (Score:2)
On the other hand, if your servers are all down, is there a lot of point to people knowing their IP address?
One secondary DNS per location is just fine. If there is just one datacenter, then use 2 DNS servers there.
It is recommended, but not required. The server police won't take your hardware away if you don't do it.
No preventative maintenance? (Score:5, Insightful)
My wild guess is they are deferring preventative maintenance on these data centers so we are seeing these major outages now. Fire suppression, UPS, transfer switches, generators, distribution panels, transformers, network gear, server, storage devices and other gear will fail if you don't maintain them properly. As loads increase, the equipment will fail earlier and my guess the people have pushed the limit of this equipment beyond they the lifespan of load rating.
Re: (Score:3, Insightful)
I would not try to convince him. Just write a memo describing the issues without sounding alarmist. It is up to the boss to evaluate the risks and to take the decision. Once you have written your memo, you are basically covered.
Now could be a good time to write this memo, just remember not to sound alarmist, just describe the possible issues although the risk is slim. You could say that you have been inspired by recent events in big data centers ;-))
As per licensing issues, call your Oracle/MS representativ
Power Fail Often (Score:3, Interesting)
Frankly, if data centers are going to proclaim their redundancy, they should test by power failing the entire data center once every two weeks at a minimum. A data center that goes down twice in a month would get ahead of any issue pretty fast. Lessons learned from the staff and the management are very valuable.
The marketing messaging:
"We power fail our data center every two weeks to ensure our backups work..."
Sound scary? Just think about the data center that has never been through this process. at tha
Re: (Score:2)
Semi-monthly pull-the-plug tests would reduce reliability. Monthly load tests on generator and a battery monitoring system ensure electrical reliability quite effectively. Only the most inadequate facilities fail to do this.
The larger problems come from improper change control, a lack of scripting, or an abnormal failure mode. Lack of testing and maintenance is a real problem, and in data centers it is far too often that it is caused by the IT team not understanding the risks of inaction. If you have an ac
Re: (Score:2)
Frankly, if data centers are going to proclaim their redundancy, they should test by power failing the entire data center once every two weeks at a minimum.
I disagree. If you perform a full load test of your facility every 2 weeks (or heaven forbid more frequently) you will be buying LOTS and LOTS of UPS batteries. Not to mention putting additoinal wear and tear on your generators, transfer switches, UPS Modules, Control Cabinets, etc.
You are correct, data centers should do "pull the plug" tests, but not as frequently as you suggest, otherwise they'll effectively be reducing their availability by introducing more risk to the equation.
Are they over drawing the power out the units? poo (Score:1)
Are they over drawing the power out the units? poor battery that blow up? Not having the right wire gauge? Not cooling the power buses and switches?
Downside to consolidation (Score:5, Insightful)
Surprise surprise...there's a downside to consolidation. Hey morons, the internet was invented as a means to ensure redundant communications paths given nuclear warfare. The old central switch (physical switching) was seen as too cumbersome and vulnerable. Now that we have wonderfully redundant communications, and have done away with most of the downsides of physically distributed systems, morons are building logically centralized systems.
NEWSFLASH - Redundant communications and physical virtualization do very little for you if you build a logical mainframe.
Truly distributed systems must be physically AND logically DISTRIBUTED with redundant comms paths in order to gain the full benefits of decentralization. (e.g. Distributed isn't distributed if all your authentication is done at one site or all your traffic must pass through .)
Re: (Score:1)
Someone mod this coward up.
Re: (Score:2)
done !! oups, posting seems to have voided my mod ;-((
Re: (Score:3, Insightful)
>> "morons are building logically centralized systems"
I have worked with such a moron doing architecture on a big government project ;-)) unbelievable...
His argument was that "The government likes centralized systems" ;-))
Re: (Score:2)
Putting all your eggs in one basket is nothing new under the sun. You ever see Ma Bell's idea of a "redundant" circuit? Two wires in the same condiut. But at least Ma Bell was doing it out of thriftiness and laziness, not ignorance and superstition.
Re: (Score:3, Insightful)
No, I think you have it exactly backwards, or at least you're missing an important nuance. It's really, really expensive to duplicate everything across two (or more) data centers. And it's full scope increase in IT costs: most or all cost categories increase. We're talking more than double the costs, in round numbers. Beyond the cost, it's very hard technically to recover hundreds or thousands of servers simultaneously or even near-simultaneously, because you are typically trying to recover not hundreds or
Former critical power field engineer here... (Score:5, Interesting)
... saying that it's time to reconsider cost cutting measures. In 15 years in the field I never saw a well designed and well maintained critical power system drop its load. I saw many poorly designed and/or poorly maintained systems drop loads, even catching fire in the process. One such fire in a poorly designed and poorly maintained system took the entire building with it, data center and all. The fire suppression system in that one was never upgraded to meet the needs of the "repurposed space" which was originally a light industrial/office space.
Qld Health datacentre disaster (Score:1, Interesting)
See story of Qld Health datacentre disaster on ZDnet recently:
http://www.zdnet.com.au/news/hardware/soa/Horror-story-Qld-Health-datacentre-disaster/0,130061702,339297206,00.htm
Even worse... (Score:5, Informative)
I'm one of the guys that services the security system in Fisher Plaza. The damn sprinklers killed half my panels near the scene. Turns out they use gas suppression methods in the data centers, not so much in the utility closets. And the city of Seattle REQUIRES sprinklers throughout the building, even right over the precious, precious servers. In defense of the staff there however, they do not keep them all charged 24/7. Other then that, I have no more info, as they're pretty locked down.
Re: (Score:2)
Re: (Score:1)
Good (Score:1)
Or at least it will keep the sales staff busy writing up quotes that will be rejected for being too expensive (although much less than the cost of a prolonged outage...)
Re: (Score:2)
So the real question is... (Score:4, Interesting)
The solution is Geographical diversity (Score:1)
Re: (Score:1)
An individual system will always eventually fail. A single part, a server, a power system, a communication system. We have an architecture of related systems that can survive the isolation of one group or large swaths of systems with no impact on the persistence or availability of data.
We call it life.
Been through too many of these. (Score:5, Insightful)
"Major" data center or not, the one your company employing you at the time is using is the important one.
In my experiences, data center backups fail about a third the time power is interupted somewhere.
Servers in an Oakland California center were the victim of the loss of one of three power phases, while the monitoring that would have switched over to the diesel generators was looking at the power level of other phases. UPS systems ran out of power. An extra level of redundancy in the form of rack mount UPSes allowed servers to shut down properly despite the data center's loss of routing.
Data center #2 was the victim of a simple power outage and immediate failure of the main data center UPS system. According to a security guard I talked to, "it exploded". The diesel backup never had a chance to start.
Then the doubly-sourced Power Distribution Unit supplying a rack at a third ISP failed in a way that turned off both sources supplying the servers.
Hint: Add an extra level of UPS redundancy and safe shutdown software daemons, at least. Multiple data centers if you need more nines.
Rackspace in Dallas (Score:5, Informative)
We're a Rackspace customer in their DFW datacenter. This is the third power-related outage they've had in the last two years at that supposedly world-class facility.
The first wasn't really their fault: truck driver with health condition runs into their transformers. Generators kick in, but chillers don't re-start quickly enough. Temps skyrocket in minutes, emergency shutdowns. Maybe the transformes should have had some $50 concrete pylons surrounding them?
The second outage was the result of a botched generator upgrade.
This latest outage was the result of a botched UPS maintenance.
None of the outages was long enough to trigger our failover policy to our DR site, but our customers definitely noticed.
While their messaging has been very open and honest about the problems, and the SLA credits have been immediate, we pay them nearly $20K per month. Nedless to say, we are shopping, and looking into a "multiple cheap colos" architecture instead of "Tier-1 managed hosting". Nothing beats geographic redundancy.
Re:Rackspace in Dallas (Score:5, Informative)
It wasn't a power issue as such, but the way their chillers reponded to two quick power fluctuations in succession:
This is what they said:
Without notifying us, the utility providers cut power, and at that exact moment we were 15 minutes into cycling up the data centerâ(TM)s chillers. Our back up generators kicked in instantaneously, but the transfer to backup power triggered the chillers to stop cycling and then to begin cycling back up againâ"a process that would take on average 30 minutes. Those additional 30 minutes without chillers meant temperatures would rise to levels that could irreparably damage customersâ(TM) servers and devices. We made the decision to gradually pull servers offline before that would happen. And I know we made the right decision, even if it was a hard one to make.
Re: (Score:1)
Re: (Score:1)
We're a Rackspace customer in their DFW datacenter. This is the third power-related outage they've had in the last two years at that supposedly world-class facility.
The first wasn't really their fault: truck driver with health condition runs into their transformers. Generators kick in, but chillers don't re-start quickly enough. Temps skyrocket in minutes, emergency shutdowns. Maybe the transformes should have had some $50 concrete pylons surrounding them?
The second outage was the result of a botched generator upgrade.
This latest outage was the result of a botched UPS maintenance.
None of the outages was long enough to trigger our failover policy to our DR site, but our customers definitely noticed.
While their messaging has been very open and honest about the problems, and the SLA credits have been immediate, we pay them nearly $20K per month. Nedless to say, we are shopping, and looking into a "multiple cheap colos" architecture instead of "Tier-1 managed hosting". Nothing beats geographic redundancy.
Thundersnatch - I'm sorry to hear you've had the same types of troubles over the past few years at Rackspace. I can't blame you for feeling burned by it, you're paying a good amount to have many 9's of uptime because whatever you're running online is clearly critical to your business. It sounds like this is definitely something we can help you out with. The company I work for (INetU) focuses on working with businesses who run critical operations online. Do you have some time for a call? My contact info
it was my fault (Score:1, Funny)
Ugh. I just finished re-watching, "Die Hard 4". (Score:1, Offtopic)
It was a terrible movie, not the kind I like to watch anyway, but for some reason I felt compelled to view the damned thing twice in two days.
The Big Bad Threat in the film was all about something called a Fire Sale, ("It All Has To Go"), where the population's fear level is spiked up into a panic by a group of bad guys deliberately crashing the national infrastructure by way of hacking all the most important computer systems. --All to create a giant distraction so that the stock market could be plundered
Sunspots, Anyone? (Score:3, Interesting)
All these data centers failed at roughly the same time as the sunspots returned [space.com], but that's just a coincidence, right?
UPSs cause more failures than they prevent! (Score:2)
I'm almost thinking of taking UPS out of the loop here. They cause nearly all the downtime we have. It would be better to just let the machines power off rather than allowing the UPSs to CAUSE the machines to be taken offline. At least if the UPS isn't in circuit, the machines power back up again when the power comes back, but if there's a fault with the UPS or it's batteries, then the machines stay offline until the batteries have been replaced.
Why the hell the idiots that design UPSs seem to think it's a
Re: (Score:2)
The UPS isn't the issue and taking it out of the loop would be absolutely dumb. The issue is proper maintaining, testing, most of all the design of HOW it's installed. (scheme if you will)
We had a power failure (UPS failure) at our backup facility (Peer1 in lower Manhattan) The problem was, they had a UPS, but if that UPS failed, there was nothing behind it. What happen was the UPS failed and the passthough controller burnt up during the failover and took the entire wing down. They have replaced the ol
Re: (Score:2)
I'm almost thinking of taking UPS out of the loop here.
That would be suicide. If you think that all of the pieces of gear in a data center could go down and then come back up with no problem, I may have a bridge that you can buy for a very reasonable price.
Why the hell the idiots that design UPSs seem to think it's a good idea to prevent them turning on if they sense a problem with the batteries is beyond me. Why not let the machines power back up but just make a loud beeping noise until the batteries are fixed. Don't they realise that most of the time the UPS will only properly test the batteries when there's an actual power cut?
Oh where to start. First of all, enterprise level UPS Systems (not the little "shoe box" APC unit under your desk) do not shut down on battery issues. At worst, during a catastrophic failure, they will trip to bypass. If properly arranged in a 2N or 2N+1 configuration, your Critical Load will migrate
Re: (Score:2)
Ain't it amazin' that all those UPS engineers whose workdays are nothing but thinking about the design of systems all arrive at the same conclusion and refuse to start their machines if the battery is failed? It's just incredible that these guys with EE degrees come out of college smart enough to do right things and somehow get it so wrong. Oh wait... I'm being sarcastic, which isn't very nice. I'd never suggest that the idiot isn't them, might instead be you, because that would be bad for my karma.
Re: (Score:1)
That's because you keep buying the cheap "Back-UPS" home computer crap to run servers on. I'm using a 15-year-old Smart-UPS 600. I've replaced the batteries once. When the original set wore out, it didn't refuse to power up, it complained about the batteries with das blinkenlights and the warning beeper until they were replaced.