Why Power Failures Can Always Lead To Data Loss 456
bigsmoke writes "So, all your servers run on RAID. You back up religiously. You're even sure that your backups are recoverable. But do you also need a UPS? According to Halfgaar (on Slashdot before to promote better Linux backup practices), yes, usually you do. He argues that despite technological advancements such as file system journaling, power failures can still cause data loss in most setups."
Well no shit, Sherlock (Score:5, Insightful)
Power losses can cause data loss? Gee, you mean that my system that relies on electricity for everything it does can be adversely effected by power outages even if I take precautions? That's some good admin work there, Lou -- if only there was some sort of law that covered the tendency of things that can go wrong to go wrong...
Next week: Fires can make things warm, floods can make things wet.
Re:Well no shit, Sherlock (Score:5, Funny)
I don't know about you, but my servers run on the power of cotton candy and happy thoughts.
Re:Well no shit, Sherlock (Score:5, Funny)
I don't know about you, but my servers run on the power of cotton candy and happy thoughts.
As a former sysadmin, I would think that any machine reliant on 'happy thoughts' would be the most crash-prone system in the history of computing.
Re:Well no shit, Sherlock (Score:5, Funny)
I can offer you a Happy Thought UPS. It's a box of puppies. Be careful though, it only has 500 puppy Amps of capacity.
Re:Well no shit, Sherlock (Score:5, Funny)
Mmmm! Puppies!!! (Score:5, Interesting)
Ok back on subject
A UPS isn't even a panacea... I had a server lose 3 out of 4 HDs in a 4 hour period. (The 3rd drive went at 4:57 PM Thursday Dec 11th 1997. Not that I would remember...) When I looked at the service history on it it had been losing drives for 8 months at an accelerating rate.
Turns out that the 3000va rack mount wonder UPS from that big, well known vendor was the problem. The switching unit in it was sending spikes into the equipment.
They wouldn't warranty it so I ended up putting a Triplite ISObar surge suppressor between it and the server in our test environment and it was in service for years after that.
Never trust any piece of equipment...
Re:Mmmm! Puppies!!! (Score:5, Funny)
They wouldn't warranty it so I ended up putting a Triplite ISObar surge suppressor between it and the server in our test environment and it was in service for years after that.
Never trust any piece of equipment...
You mean like a Triplite ISObar surge suppressor?
Re: (Score:3, Interesting)
The reason I remember the exact minute it failed was I had my bag in hand and was walking toward the door when the server alarm went off.
18 hours later I found out from the backup software vendor that there was a bug in the software that meant it wouldn't restore any rights
It can be done! (Score:3, Funny)
...If you're a Mac fanboy running a network of Apple computers. If anything goes wrong, it's an artistic expression and anyone who criticizes the problem is a closed-minded square who "doesn't get it." Then you sit back in self satisfaction listening to alternative pop, thinking about how hip and different and enlightened you are.
Happy thoughts power supply: Dead stable.
Linux networks can run on happy thoughts as well as long as you run on electricity during the setup and installation stages and then switch
Re:Well no shit, Sherlock (Score:5, Funny)
Except the server that runs http://youporn.com/ [youporn.com]
Re:Well no shit, Sherlock (Score:5, Funny)
My servers run on Electricity but the RAID controller has battery backed up RAM so any cached data will persist a power failure and the disks are in writethrough mode.
I like this setup, but please. Tell me more about this cotton candy technology? Is it superior.
Re:Well no shit, Sherlock (Score:4, Insightful)
That is until the 10,000 volt spike when the power company improperly brings the grid back up bakes the RAM, the battery, RAID controller and the hard drives.
Voltage Spikes (Score:5, Informative)
The typical small UPS system has some amount of surge protection built-in. But it's typically only good for at most a couple thousand joules. But then, if you get a spike that is big enough to blow a varister, you also get to buy a new ups.
A better solution is to put a "whole house" surge protector on the circuit-breaker panel. It protects everything, with a much higher number of joules. Five or six pounds of varisters can absorb a lot more shock than one ounce of varisters. They cost about $100, and can be found at most big hardware stores or electrical supply houses. That doesn't eliminate the need for a ups. It does protect the ups, along with the other equipment, from most voltage spikes.
Last year, lightning hit the power pole 20 feet from my house. We know where it hit because the pole caught fire. My next-door neighbors on both sides lost every single piece of electrical equipment -- not just computers, TV's, and stereos, but also fridge, microwave, water heater, and range. All of it was damaged beyond repair. We barely noticed the hit, except for the bright flash of light, and had no damage at all.
Re: (Score:3, Insightful)
I'm not convinced that whole-house protection helps much either. A few years ago, there was some event during a thunderstorm - we never quite figured out what - that fried two TiVo modems, a garage door opener (the circuit board was visibly burned and light bulb shattered), a few Wirsbo hot-water thermostats (not even connected to the mains power, just low-voltage from the boiler), a few Vantage whole-house dimmer modules, an intercom, and a printer.
The house was, at the time, "protected" with two Cutler-H
Re:Voltage Spikes (Score:5, Informative)
The path-to-ground is really important, as is the quality of the ground. The length of the path is the reason why whole-house devices are installed at the service entrance panel. But, that assumes that your service-entrance ground is a good ground.
If your ground is not good, shorting to ground won't do much good. A lot of houses around here are grounded to plumbing pipe that is buried just 12" deep. During a dry spell a few years ago, I detected variable voltage where it shouldn't have been. The voltage problems cleared up after I added an 8-foot vertical ground rod to the system.
The thing that kills a surge protector is too many amps for too long. If it shorts the power to ground (low-resistance), but the ground is not really well-grounded, then the whole thing can float close to line-voltage. In that case, that voltage can destroy your other devices, while the surge unit never gets enough current to burn the varisters.
Re: (Score:3, Informative)
Common-mode spike. The power line was fine, but your gro
Re: (Score:3, Informative)
Neutral and ground are supposed to be bonded at the service entrance panel, and not anywhere else. If the ground is actually grounded, with a big copper wire to a big copper spike that goes deeper than the water table, that will normally provide the path of least resistance for the electricity to follow.
A lot of houses don't have a good ground connection. Most building codes (and the NEC) allow 25 ohms resistance on the ground connection. But it's hard to measure, so the building inspectors don't meas
Re: (Score:3, Interesting)
That's why any datacenter worth putting your servers in pipes its power through a flywheel or some other electricity "cleaner". A 1-ton lead ball spinning at 10,000 RPMs isn't going to speed up that much on a spike like that.
Re: (Score:3, Insightful)
Ah, that's easy (Score:5, Funny)
All you need to do is have the grid power feed some high wattage light bulbs. And near the light bulbs is some solar cells. The output from the solar cells is used to charge batteries which feed an inverter that actually powers the computer. Of course there is some power loss in the conversion process, and you need to have some (ok, a lot), of the input power to the system commited towards running a cooling unit to keep things at a reasonable temperature. But the resulting device provides clean power with no possibility of any surges getting thru to the protected equipment.
Of course, if you go to this level of trouble for your power source, then I'd also suggest opto-isolating all signal lines to and from the server. And enclose the server in a well grounded faraday cage. And it wouldn't be a bad idea to have a dedicated comm link to a duplicate server located else where. Preferably on a different tectonic plate.
Re:Ah, that's easy (Score:4, Interesting)
You now have a 1% efficient power supply.
A slightly more practical option (with better isolation than a standard electromagnetic transformer, but unfortunately also some inductive effects) would be to couple two motors with an insulative shaft.
Re: (Score:2)
My servers run on Electricity
My servers run on Love.
Re:Well no shit, Sherlock (Score:5, Funny)
Your mom loves you and pays for the electricity. That doesn't mean that your servers run on love.
Re:Well no shit, Sherlock (Score:5, Informative)
Mine run on evil thoughts and hatred (Score:2)
And I bet they has a longer uptime than yours....
Re: (Score:2)
Yes, where is the 'Duh' tag when you really need it? Or maybe slownewsday...
Re:Well no shit, Sherlock (Score:5, Informative)
Ok, people who don't just read the executive summary knew this all along, but perhaps it's necessary that someone spells it out for the rest: Journaling and RAID do not prevent data loss in case of a power outage (and many more circumstances). If you know why, just skip the article. If you're wondering how you can lose data if you write everything to two disks and your filesystem guarantees its own consistency, then perhaps this is the wake up call that you need.
Re: (Score:3, Insightful)
Ok, people who don't just read the executive summary knew this all along, but perhaps it's necessary that someone spells it out for the rest: Journaling and RAID do not prevent data loss in case of a power outage (and many more circumstances). If you know why, just skip the article. If you're wondering how you can lose data if you write everything to two disks and your filesystem guarantees its own consistency, then perhaps this is the wake up call that you need.
Any Server Admin who didnt realize that isnt really a server admin. And the rest of the world probably doesnt care or need to know.
Just a thought... ;-)
Re: (Score:3, Insightful)
Any Server Admin who didnt realize that isnt really a server admin. And the rest of the world probably doesnt care or need to know.
Just a thought... ;-)
The fact that they're not really server admins doesn't stop them from running servers, though!
Re: (Score:3, Funny)
if only there was some sort of law that covered the tendency of things that can go wrong to go wrong.
I hear Murphy might have one :)
Re:Well no shit, Sherlock (Score:5, Funny)
One of the first things that will happen, is that the memory DIMMs will no longer be refreshed properly (DRAM needs to be refreshed constantly otherwise it will loose it's data) and very rapidly, the memory will contain only garbage. The hard drives and DMA controller however, will run a bit longer; so if data is being written to disk, the DMA controller will keep reading data from memory, but it has no idea that this data is corrupted.
However, we've recently seen that RAM holds state well enough to preserve crypto keys thru a power cycle [hackaday.com]. This has very scary implications: the RAM knows what's happening, and behaves differently (loses data immediately on power-off or remembers it for several seconds) in order to cause the most difficulty for the owner of the machine.
Not only are computer components intelligent and self-aware, they're also out to get us!
no, that's not the scary thing (Score:3, Funny)
Re:no, that's not the scary thing (Score:5, Funny)
its not worth loosing you're cool about grammer misteaks and etc.
Re: (Score:3, Informative)
Actually when power drops the "power good" line from the power supply goes low, which causes a system reset and locks everything up.
This is also how the computer knows how long to keep the reset line engaged on startup, it stays asserted until the power supply says the power is good, and everything has proper voltage.
Re: (Score:3, Funny)
We just need to get that guy that declared Pluto is no longer a planet to declare that electricity no longer causes data loss.
Side note: He also declared that north is no longer a direction, blue is no longer a color, and your sister is no longer a virgin.
Re:Well no shit, Sherlock (Score:4, Funny)
Illiteracy (Score:5, Funny)
From TFA:
(DRAM needs to be refreshed constantly otherwise it will loose it's data)
Fly, little data! Be free!
Re: (Score:3, Funny)
Get off my lawn, you little bits!
can always lead to data loss? (Score:5, Funny)
UPS - more than just a backup. (Score:4, Informative)
UPS is more than just saving your data.
Re:UPS - more than just a backup. (Score:4, Informative)
It's also important to get a decent UPS too, if you're using it for something like a server. I think the cheapy ones basically just use a transfer relay, where as the higher end ones actually run the hardware off of the battery via the inverter all the time. While I would think that with the former (called "standby" UPSs maybe?) the transfer time wouldn't be enough to cause too many problems, you still don't have the buffer that you'd get with a true uninterruptible power supply.
I think a lot of the cheaper ones don't put out a true sine wave either, though for their intended purpose of letting you shutdown your desktop cleanly again they're probably fine.
Re:UPS - more than just a backup. (Score:5, Informative)
Yup the 3 major types of battery UPSs I know of:
Offline - Relay or simple failover. (APC Backups)
Line Interactive - Can correct line over/under voltage to a point (APC Smartups)
Online - Full AC -> DC -> AC conversion. (APC Symetra, Liebert, anything that doesn't suck)
Basically outside of home use you want an online type UPS.
There are other systems like motor/generator flywheel types, but they need a very fast backup generator to sustain anything more than 30 seconds of outage. But they're great for smoothing out some types of line issues.
Re: (Score:2, Informative)
>UPS smooths out all those nasty spikes as well as stopping your servers from going down to a 1 second power cut.
A true UPS smooths out the spikes. Most of today's UPSes (at least consumer models) are off-line [wikipedia.org] supplies. The batteries don't kick in unless the power is out. Worse than that, the cheap ones don't output sine waves, they output square waves. These UPSes also take some time to switch to batteries, leaving your computer without power for that time.
Now, some of those UPSes have filtering tec
Re: (Score:2)
Exactly. Even more important than simple power backup is AVR. The microfractures that can be created in system chips by voltage as little as +/- 10 volts, over time, will cause systems to malfunction.
I can't find the article, but at one point as a reseller for APC (maybe it was Leibert), their marketing material used to state that 95% of all system component failure was due to voltage irregularity, and that properly filtered line voltage could extend the life of electronics 3 fold.
You should not only have
Duh! (Score:5, Insightful)
I remember a discussion on the PostgreSQL hacker's list about recoverability and transaction logs.
You can't make a system that will not lose data, you can only make a system that knows the last save point of 100% integrity.
There are too many variables and too much randomness on a cold hard power failure. You absolutely need a UPS that gives you time to shut down cleanly.
Re:Duh! (Score:4, Insightful)
You're still hosed if your server's power supply goes titsup. Or if your hard drive crashes. Or if the building burns down.
Gotta love these slashvertisements, I wonder whose UPSes they're pimping? Its not like we don't all know you need a UPS. What's next, a FA about how you need fire insurance?
Well of course you need UPSs, but (Score:5, Informative)
Re: (Score:2)
Re: (Score:2, Interesting)
Re: (Score:2, Interesting)
been there, and done that! We recently moved a few servers this way. Just be careful, and go slow.
What this really points out... (Score:3, Insightful)
is a weak spot in the design of most computers.
Computer power supplies should be built with enough spare capacitance to run things long enough for the computer to save critical data, and operating systems and critical apps should be able to handle an emergency shutdown and save critical data in very short order.
This is old hat in embedded systems.
Re:What this really points out... (Score:5, Informative)
Computer power supplies should be built with enough spare capacitance to run things long enough for the computer to save critical data
Here's a question for you: Calculate the size of the capacitor needed that can hold enough power to run a 200W load for 5 minutes and maintain a voltage level within a specific usable range.
Hint: its BIG. batteries are more space efficient, but the chemicals and outgassing make them inappropriate for location INSIDE the computer box.
Re:What this really points out... (Score:5, Insightful)
Who the hell is talking about 5 minutes!? I'm saying you should be able to get a clean shutdown in 5 seconds if you prioritize it correctly.
Re: (Score:2)
Who the hell is talking about 5 minutes!? I'm saying you should be able to get a clean shutdown in 5 seconds if you prioritize it correctly.
I'm not sure what your system is, but for this to be a general purpose device, it needs to work within the realm of real life systems. Have you ever typed "sync" on a busy system and had it go away for a minute or more?
5 minutes is a "safe" number. It takes time to detect a power failure more than a mere "spike." You don't want to start a shut down and suddenly have the
Re:What this really points out... (Score:4, Insightful)
Why 5 minutes? It usually takes less than a second to run a sync on the disks depending on how active they are. A couple seconds of runtime should be enough to do an "emergency shutdown" and avoid data corruption.
####@johncash:~$ time sync
real 0m0.004s
user 0m0.004s
sys 0m0.000s
Re: (Score:2)
A couple seconds of runtime should be enough to do an "emergency shutdown" and avoid data corruption.
As I said in another post, it is very much more complicated than just a few seconds.
Re: (Score:3, Interesting)
I think you're making it more complicated than it needs to be.
If the system gets a signal that power is going away very, very soon, drops everything else, and just devotes its last seconds to getting things in order - it should be doable in a few seconds and be vastly preferable to the alternative of just having power go away without warning.
Obviously a UPS is an even better option, but it's not every place that could use a UPS is ever going to get one and it would be good if we could work on the problem fr
Re:What this really points out... (Score:4, Informative)
Other than the lack of communication at present between the PSU and the rest of the system (on a hardware and software level), what you're describing really seems to be the computer equivalent of throwing your hands in front of your nuts as you spot the incoming baseball. It helps the immediate problem of data (or testicle) loss, but it's really just a small amount of damage control.
This is why a proper UPS that can trigger a full system shutdown once you hit a certain power remaining threshold is far preferable. Granted I'd rather have a controlled crash than the risky nonsense that would come from the power cord being yanked, but (right now) computers can only go so far to help themselves in a couple-second window.
Re:What this really points out... (Score:5, Informative)
Why 5 minutes? It usually takes less than a second to run a sync on the disks depending on how active they are. A couple seconds of runtime should be enough to do an "emergency shutdown" and avoid data corruption.
####@johncash:~$ time sync
real 0m0.004s
user 0m0.004s
sys 0m0.000s
That will sync the disks, but it won't stop the database from accepting incoming data. It won't stop cron jobs which might be just about to trigger. It won't deal with tasks that are in the middle of a big operation which involves a lot of writing to disk.
Re: (Score:3, Insightful)
You mean, the battery location on my laptop is not appropriate ?
I know laptop and servers are very different but still, if my laptop can run 2 or 3 hours on a battery (including the LCD), it should not be that difficult to use the same technology to power a server for a 5 minutes (with no screen needed).
Re: (Score:3, Interesting)
I agree with you.
My point was that just because a battery can power a laptop for several hours doesn't mean a single battery can supply a server for 5 minutes. So, the GP was claiming that because: (laptop power consumption) * (2-3 hours) == (server power consumption) * (5 minutes) it shouldn't be hard for the same battery to power both. The point I was trying to make is that a device that provides a certain range of performance, (in this case the car at 70 MPH), doesn't mean it is easy for it to perform
Re:What this really points out... (Score:5, Interesting)
This is old hat in embedded systems.
Yes, but embedded systems usually have lower power requirements, or at the very least, a smaller range of power requirements. You can't add 3 PCIe cards, a few extra drives, and a few more GB of RAM to most embedded systems.
I worked on the design of an embedded system a few years ago that had a holdup spec - I think it was supposed to survive for 50 ms with no power. So a 50 ms power interruption would result in continued operation, while an outage longer than that was allowed to reset the board. However, the power draw on the board was around 200 Watts; being able to supply that much power for that long in a fairly compact form factor was a huge hurdle. It also caused airflow problems, because the giant capacitors would prevent air from getting to other components on the board, like the CPU. In the next version of the spec, I believe the holdup requirement was eliminated - apparently we weren't the only ones having trouble meeting that requirement.
Our Tandem (Score:5, Interesting)
We spent most of the day getting our servers back up from various states of disrepair (confirming the article, power loss is superbad). It turns out that our main medical software ran on a Tandem. Though the drives and such lost power, the CPU had a backup of D-batteries and survived the power loss just fine. Needless to say, we stopped making fun of their seemingly primitive emergency backup power.
Re: (Score:3)
This is my favorite power loss story.
It's great.
http://www.alioth.net/tmp/vaxen.html [alioth.net]
Re: (Score:3, Interesting)
The problem is that different applications systems have different amounts of stats that must be saved. An RT app usually only has a memory buffer that can be written in a small number of IO's. Many business apps have relatively lots of data, in non-contiguous buffers, that require hundreds of IOs to store. Many business systems have hundreds of such apps running in the machine at the same time. Some systems can have gigs of data, in thousands of buffers, in their write-behind cache. And, some businesse
It happened to someone (Score:5, Insightful)
Re:It happened to someone (Score:5, Funny)
Yes. My first reaction upon reading the summary was.. "Duh?" What, did they have it plugged into the wall before that? A UPS becomes MORE critical, not less, as the cost of hardware (RAID arrays are expensive) goes up.
Re: (Score:2)
Would you believe that a certain major UK university runs its entire computer science dept without either UPS or power spike protection?
I was surprised, especially since I saw how the regular power spikes blew computer after computer and nothing was done.
As for trying to run experiments that took more then a day or two to complete, well, can you picture a post grad who's just found that a weeks work has once again been wasted because some bean counter refused to pay for department wide UPS?
Alas I don't have
Don't for get to test people, TEST! (Score:5, Insightful)
We recently had a test night where all we did was test the UPS system and shutdown procedures, and there was a couple gotchas. Interestingly, by default the APC powerchute app we were using defaulted to shutting down the UPS completely after the [first] server went down - not good. This was buried fairly deeply in the configuration.
Equally important to any protection measure, be it RAID, Power Protection, whatever - is testing!
Re:Don't for get to test people, TEST! (Score:4, Interesting)
I 100% agree with the idea of testing under controlled conditions. The oops you guys discovered is a good thing to be caught early on. I can imagine the look on your support team's faces when the UPS suddenly turned itself off while the remaining servers still trying to perform a safe shutdown. I'm sure the secondary UPS was left running as a precaution until the test is successful.
I have seen a screw up where somebody cut into a live power cord thinking it was a tie wrap caused a major short in the PDU. The guy thought he was safe until he discovered whoever installed the servers didn't double check the power connections and loads so it created a cascade failure in several racks and lost several tons of data. Recovery took awhile.
Least to say it was not a good day.
Re: (Score:2)
On the other hand.. (Score:3, Interesting)
you can recover your RAM minutes after loosing power.. no kidding! http://citp.princeton.edu/memory/ [princeton.edu]
I know PHB's try to cut costs.... (Score:2)
I know that PHB's will try to cut costs, and that unnecessary hardware is the first to be cut, but is there ANYONE who believes that a UPS is not needed? Are there really people out there that think, "We don't need the UPS right now. We can wait until we have more money."
It boggles my mind that there is even a need for such an article
That's what I always say sometimes (Score:2)
Well, duh. Thank you Captain Obvious.
Here's question for you all. I have a cheap Conext (made by APC) IPS. Yes, it's an interruptible power supply. It used to work fine, but once I added a Samsung b/w laser printer, whenever the printer's heating element first comes on, the UPS drops out immediately and the computer restarts. Even put a new battery in it; no help. The printer, btw, is NOT plugged into the UPS. The line voltage appears to get yanked down just momentarily and the computer ignores it, whe
Re: (Score:2)
Re: (Score:2)
The docs for every UPS I've ever seen say do NOT attach a laser printer to them, for exactly the reasons you've seen -- when the printer comes on, the startup drawdown is just too much (it's generally about four computers worth).
Your printer should be on a good surge protector, but there is no reason it needs to be attached to the UPS. Some UPSs now have spare plugs for exactly this use -- they provide surge protection but not continuous power.
Re: (Score:2)
You should read his whole comment -- the printer is not plugged in to the UPS.
Re:That's what I always say sometimes (Score:5, Interesting)
Rule #1.
NEVER plug a laser printer into a UPS. The power that the fuser draws is WAY too much.
Look at some of the cheap office units, they show little pictures on them, notice the printer icon is on the surge side, NOT battery/surge side.
If the power goes out, you should NOT be trying to print.
http://articles.techrepublic.com.com/5100-10878_11-6085460.html [com.com] See #6
http://arstechnica.com/guides/other/ups.ars/3 [arstechnica.com]
http://www.jetcafe.org/npc/doc/ups-faq.html#0405 [jetcafe.org] see 04.05
Would you put a space heater on a UPS? Shredder? Vacuum? Table Saw? If you put a laser printer on it, you may as well.
Re: (Score:2)
Maybe I'm the dumbass here, WTF is an interruptible power supply? And why is it called a UPS, when a UPS is an UNinterruptible power supply?
Re: (Score:3, Informative)
Yes, quite. It can't handle the substantial inrush current needed by the laser printer.
The "click" you hear in the UPS when the laser printer warms up is the UPS noting the drops on the power mains, which gives you some idea just how much current that printer needs.
I have a Samsung ML2150, and have noticed the same thing. Lights flicker, etc. whenever I submit a print job and the printer transitions from standby to active. The various UPSes in my office sense that, and respond with clicks and beeps.
Take
Get a UPS (Score:4, Insightful)
I really can't understand people who don't have a UPS. Don't you care about your data? At all? The UPS is not very expensive (My BackUPS 900 is very nice and only $100), and will last a long time (you just replace the batteries now and then). Once you are on UPS, you can stop worrying about any power issues, journalling file systems, crash recovery, and all that. The computer will never fail due to power. If you run Linux, it will also never fail due to the OS. If you are a normal user, that means your computer will never fail, period. Seriously, there is no excuse for not having a UPS. Go and get one right now!
Re: (Score:2)
I always tell people the same thing. For about $100 for a decent home-type UPS, you will never have your hardware trashed by power spikes and sags, and you'll never have your work rudely interrupted or destroyed by a power outage.
Re: (Score:2, Insightful)
Re: (Score:3, Interesting)
If we were talking about a datacenter, then yes, UPS on everything important. But for home use, nah.
Is this bring your kid to work day? (Score:5, Funny)
Ok, now everyone has something to give to your kid for the sysadmin-in-traning class.
For the rest of us... back to work, nothing here you didn't learn your first year.
For the poster... Shame shame... Turn in your card.
Uhm, no..? (Score:2)
If you back up religiously, assuming you have the backups on some sort of removable media, why would recovering from them be impossible when data loss via electrical outage occur?
Dur-durdur!
Carefully proofredded article (Score:3, Funny)
"3.2. (Ecrypted) file systems"
Please tell me more about these ecrypted file systems. Do they also do gurnalling?
Don't forget the simple case... (Score:2)
Re: (Score:3, Informative)
Real text editors will recover gracefully from such situations. :-)
(I'm think along the lines of @UEDIT on OS2200 which saves its entire virtual memory state to disk periodically and can recover it with ease at the next startup, or the old EDT editor on VMS which saved the commands one entered and could replay them when a recovery was specified).
I'm surprised more text editors don't have a similar feature. I think vim does, tho...?
And this is what ZFS looks out for (Score:4, Interesting)
...by design. TFA doesn't delve into too much detail, but a sudden power loss on such software RAID systems is a condition that ZFS accounts for. Its Copy-on-write (COW) and write-length stiping strategy prevents things such as the RAID5 write hole [sun.com] condition, a condition that has the biggest chance of occurring when a power loss event happens.
A UPS is good to have. Even at home. (Score:4, Interesting)
Last night we had a power outage. I shut down the desktop and was able to continue working for almost 2 hours on the laptop because with the Desktop down the UPS was only carrying the DSL router and the WiFi box.
At work. Power is a whole enterprise within the company I work for.
Dual gas powered Generators at each location, Rooms full of Batteries for the Telecoms gear (most is straight DC) and Inverters for the Servers. (DC PSUs are available for some of the servers we use but at so high a premium that the inverters are cheaper.)
We can handle a dozen Power cuts in a day with no service interruption or data loss ("Tested" 2 weeks ago) and we can stay up without external power for more than a week. After that we have to start trucking in additional diesel.
Yep. That's right. With sufficient fuel we can be online indefinably. Which we will have to do if we get hit by a major hurricane.
Which means the phone network is a lot more reliable than the Power grid where I live.
As for Data loss. I have over the years done a lot of recovery work. "Morfy" of "Murfy's Law" fame isn't a guy or a girl. He is a deamon from the darkest pits of hell sent to torment the souls of IT workers everywhere.
Imagine a server, where UPS #2 is down for repairs, UPS #1 fails during a power cut, When everything comes back up we find 2 failed hard drives in the RAID 5 on the email server.
despite previous testing and confirmation that the backups work the most recent tapes failed to read.
Eventually we sent the failed drives off to a Data recovery company in Florida because
#1. The customer can afford it.
#2. Simply "skipping" a few days of Email is not an option for a bank (hence the ability to afford data recovery).
So yeah. A UPS is essential. Just like RAID, Clustering and Backups but in the end it can all fail.
Best advise? Memorize all your important data. That way if you loose your mind, you are not responsible for the lost Data (or anything else).
Re: (Score:3, Informative)
Last night we had a power outage. I shut down the desktop and was able to continue working for almost 2 hours on the laptop because with the Desktop down the UPS was only carrying the DSL router and the WiFi box.
good uptime for a laptop. got a second battery? (I know I do)
Inverters for the Servers. (DC PSUs are available for some of the servers we use but at so high a premium that the inverters are cheaper.)
that's because it just has to invert it before it can step it up or down. If you supply DC you are
Other reasons to run a UPS (Score:4, Interesting)
UPS units are relatively cheap, it's well worthwhile to invest in one, not just to protect from data loss:
* Hardware loss: I've seen a lot of hardware blown up from power interruptions. Do you trust your power company that much to provide clean power to you? Sure surge protectors help a bit, but a decent UPS costs maybe twice as much as a good surge protector.
* Time lost restoring your session after blackouts / brownouts: OK, maybe you're used to restarting your computer every morning anyway. But I like to leave things open and return to my desktop just the way I left it arranged.
* Stats: Using NUT and Munin, you get to monitor and log your power, so you can see things like exactly when your electricity went out and for how long, what load your PC is drawing after that last upgrade, etc. e.g.: http://hairball.bumba.net/cgi-bin/nut/upsstats.cgi?host=apc@localhost [bumba.net]
* Graceful shutdown: you have a chance to tell your buddies that your power just went out, and you'll be coming back once it's restored.
Frankly, I'm a little surprised a backup battery isn't built into PC power supplies already, so they'd work a bit more like laptops. Same with networking gear.
Re: (Score:3, Informative)
What I have is a Tripp-Lite SB-2000, which is an oldie but a goodie. Only link I can find now is here [vancebaldwin.com]. It runs on 24v external power, so I just set two car batteries on top of it. Picked it up years ago for a song on ebay.
That unit though really is meant to have massive batteries on it. (looks like 24v golf cart batteries maybe, it has large binding posts on it for the external battery, there is no internal battery)
You can't just hook a car battery up to some old APC you have sitting around. It may run
Chose your UPS carefully, and TEST it the hard way (Score:4, Informative)
This morning we had a planned shutdown of 100 servers for eletricity works, all were on the same 40 kVA UPS. All went fine, we shutdown all servers to be safe, and kept some stuff online for montoring and the like, then main power was shut off. The UPS gladly took the load, with an estimated battery life of 75 minutes, more than what was needed for the electrical work. Once this was done, the electrician put the main power back on, and... the UPS shutdown !
Since all servers were stopped already we didn't lose anything, but we had to put the UPS in bypass mode for a while, then back on, and now we hope for the best waiting for the UPS to be repaired, crossing most of our fingers because of the holidays...
In summary : testing that the UPS can handle the power coming back is as important as testing for it to be able to handle the power shutting down.
He forgot UPS-triggered shutdown (Score:5, Insightful)
If you're not at the machine, or don't know how to shutdown without a CRT, the disk can get messed up when the UPS runs out of power. Unless you only have a desktop machine with no network applications writing to disk (no BitTorrent); then you might be OK if you just walk away from your keyboard and let the system become quiescent before it loses power.
I dont get it (Score:3, Insightful)
1) You build a RAID5 array
2) You backup
3) You test your backups
4) You plug your server DIRECTLY INTO THE WALL?!?!
Ummm DUH! Of course you need a UPS - what kind of yutz does 1-3 and then powers the server off of unconditioned wall power?
What I did (Score:3)
It went from what was essentially a closet in a basement with a single AC unit and individual UPS's on each server.
So I decided redundancy was key. We had redundant AC, but the best part was power.
All servers (70 of them at last reckoning) are attached to an APC Symmetra that nominally gives 40 minutes of battery power. The Symmetra in turn is backed up by a 125kW natural-gas fired generator that spools up within 10 seconds.
It was decided we could suffer a brief AC outage so that was simply attached to the generator. There were two 2 ton AC units in place.
Even had the foresight to extend a tendril out to the MDF in the building so that our telecom and ISP could plug their UPS into the generator circuit.
And what was the fly in the ointment? Our DNS services were provided by an outside entity. So one day we had a power failure that hit a very large swath of the city and included us and the entity that provided DNS services.
So while everything in our shop was running, nobody from the outside could see our public services, and nobody inside could get out.
We actually got hold of the DNS zone and had our own after that.
Wrong, I think. (Score:3, Informative)
The hard drives and DMA controller however, will run a bit longer; so if data is being written to disk, the DMA controller will keep reading data from memory, but it has no idea that this data is corrupted.
Pretty sure that's wrong. It used to be (20 years ago) that hard drives losing power in this way had a chance of the heads crashing against the platters (the fabled "hard drive crash"). To solve this, modern drives are very sensitive to the power input. As soon as power fails the drives extract power from the spinning platters to move the heads over to the parked position. Regardless of what the DMA controller thinks it should be doing, the hard drive is busy parking the heads.
Re:Not me! (Score:5, Funny)
If there's clouds in your server room, your server's probably been slashdotted and is on fire!