Slashdot Log In
Why Mirroring Is Not a Backup Solution
Posted by
kdawson
on Fri Jan 02, 2009 12:25 PM
from the pointed-lesson dept.
from the pointed-lesson dept.
Craig writes "Journalspace.com has fallen and can't get up. The post on their site describes how their entire database was overwritten through either some inconceivable OS or application bug, or more likely a malicious act. Regardless of how the data was lost, their undoing appears to have been that they treated drive mirroring as a backup and have now paid the ultimate price for not having point-in-time backups of the data that was their business." The site had been in business since 2002 and had an Alexa page rank of 106,881. Quantcast said they had 14,000 monthly visitors recently. No word on how many thousands of bloggers' entire output has evaporated.
Related Stories
Submission: Mirroring is Not a Backup Solution by Anonymous Coward
This discussion has been archived.
No new comments can be posted.
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
Full
Abbreviated
Hidden
Loading... please wait.
DUH! (Score:5, Insightful)
DUH!
Re:DUH! (Score:5, Funny)
Parent
Re:DUH! (Score:5, Funny)
Journalspace CTO: We don't need an expensive off-site backup solution b/c we mirror all of our data real-time. It's genius!
-entire database gets overwritten-
Journalspace CTO: Ohhhhhh...now I get it.
Parent
Re:DUH! (Score:5, Funny)
Parent
Double Duh! (Score:5, Interesting)
Parent
Re:Double Duh! (Score:5, Informative)
Not quite. Backing up a live database can be a bit tricky. By the time you finish copying part of the database, the first bit can change again. So you have to create a snapshot of some kind. And that has to be supported in the database setup (at the application or server level) in order for the backup to be in a consistent state. And you don't want your backup process to degrade site performance, either. So a simple file copy is totally inadequate.
A common solution is replication. Backup is then performed by creating a replication point on the slave database machine then taking a snapshot and copying that while while master database machine continues serving at full speed. Replication can then catch up when the backup is complete. Another advantage to having replication is duplication on the machine level -- if the master fails, go live to the slave with minimal to no downtime. Set both machines up in a master-master configuration and you can swap back and forth as needed, allowing live maintenance and backup with no performance degredation.
Parent
Re:Double Duh! (Score:5, Informative)
*BZZZZT*
The GP was 100% correct. If you had kept reading, you'd see that the suggestion was to use replication so you can lock the DB into a consistent state while backing up. When the backup is done, the box starts replicating again. If you didn't have the backup box, you'd have to lock the production database while your backup was going on.
He was suggesting replication purely as a way to avoid having to pause the application during backup, not as the backup it's self.
Parent
Re:DUH! (Score:5, Funny)
What about archive.org?
Ah, apparently not... [archive.org] :-D
Parent
Re:DUH! (Score:5, Funny)
Parent
Again a frost post to a red story (Score:5, Funny)
While this mirrors previous comments, it's not really a backup solution.
When is backing up *not* an option? (Score:5, Interesting)
Mirroring, RAID, grid, whatever. At some point, you want your data safe and secure on something not physically attached to any power source.
Re:When is backing up *not* an option? (Score:5, Insightful)
This is at a minimum people. Come on!
Parent
Re:When is backing up *not* an option? (Score:5, Insightful)
Even accepting your price that's a cost of about 12.7 cents per gigabyte and you can get 800GB native LTO-4 tapes for about $50, which comes out to about 6.3 cents per gigabyte.
But quoting costs for desktop grade SATA drives severely understates the true cost. For any non-trivial site installation you're talking near-line rated drives, drive caddies, storage shelves and additional SAN fabric. Then price out the additional power, cooling and rack space. Then price offsite shipping and storage for the bulkier, heavier and more delicate disk option.
Mirroring has its place. Snapshotting has its place. And backups to stable media still has its place too.
Parent
Re:When is backing up *not* an option? (Score:5, Insightful)
Fine. Get the cartridges, but what about the capital cost minus depreciation of the drive? What about random access?
Random access is why snapshots also have their place. :) Archival backups and nearline backups solve different sets of problems.
Now weigh those against an inexpensive jbod frame with a 2gb FC backplane.
What kind of capacity are we talking. For a small site you can pick up a little 2U unit that'll store 6.4TB uncompressed for under $5k. Or if you're running a larger site you can snag a 4U unit with two drives for about $15k that'll handle 30.4TB with optional expansion to 60.8TB native.
What's the write speed of LT vs a tasty little GB SAS drive?
120MB/sec per drive without compression. And now that you've talking about SAS drives your per TB cost is hopelessly optimistic. Even OEM packaged terabyte SAS drives are going to run you about a quarter a gigabyte, which is now four times the media cost of an LTO-4 solution.
Rackspace? You can put a dozen into about 4U.
So about 12TB in 4U compared to the 30TB unit I mention above.
Cooling? Although I'll grant you green cost, the random accessibility out-classes the seek time and tape insertion by a human cost dramatically.
Have you never heard of a tape library?
Stable media? Tape? Sometimes.
Properly handled tape is incredibly stable.
Shelf space?
If you're doing off-site storage, that's going to be an issue regardless of what media you're using. And as I pointed out, tape is far more compact and far lighter than disks.
No need to use tape anymore. Get out of the reality distortion field, but do the right thing by testing what you have and doing drills to ensure that whatever you have, works and is a procedure understood by all.
I'm not the one dismissing an entire class of technology while demonstrating ignorance of its costs and benefits.
Parent
Re:When is backing up *not* an option? (Score:5, Insightful)
That's not my company's policy, that's *my* policy. I can take a 3-month hit to my personal data. AND YET MY LAX PERSONAL POLICY WOULD HAVE SAVED JOURNALSPACE.
My *company's* policy is daily offsiting. Expensive, but very many of our locations could become a smoking hole in the ground and we'd still be able to restore and operate.
Parent
To many shops think HA==DR (Score:5, Informative)
It's more an issue that some people think that HA == DR.. which obviously this story reminds us that it is not the same thing.
Mirroring / RAID == HA.. if one of your HDDs let the smoke out, you still don't incur downtime. If you have a hot-spare, you're even better.. all it does it let you have alittle time to correct the
issue (ie: "It can wait until morning").
Also, one other very important thing.. mirroring doesn't prevent/restore data corruption. If you're mirroring your rm -rf (as pointed out by Corsec67 below), your RAID will happy do what it does.. and span your command to all your disks.... Congrats, you just successfully gave yourself HA to your disk erasing! :]
Backups are DR.. If your RAID croaks.. your SOL if you don't off-machine backups. If you accidently nuke your disks with an rm or something, you can still go back and restore data.. sure you'll likely loose -some- data, but -some- is better then all in this case.
Parent
Re:To many shops think HA==DR (Score:5, Informative)
DR is Disaster Recovery
HA is High Availability
Parent
Re:To many shops think HA==DR (Score:5, Funny)
I tried Googling, but the only results I got were a medical office in Chinatown.....
Parent
Dear Every Corporate Tool in the Universe: (Score:5, Insightful)
Re:Dear Every Corporate Tool in the Universe: (Score:5, Insightful)
And that's why your IT department actually needs funding. Sleep tight.
They've had the site live for 6 years.
This wasn't a lack of funding, it was just sheer stupidity.
6 years and nobody ever thought it'd be a good idea to back everything up to dvd or an external hard drive. HTML compresses really well in case they didn't know.
Parent
Re:Dear Every Corporate Tool in the Universe: (Score:5, Insightful)
Parent
Re:Dear Every Corporate Tool in the Universe: (Score:5, Insightful)
Being too stupid to recognize your own shortcomings is also a form of stupidity. Or hubris, whichever is more appropriate.
Parent
Re:Dear Every Corporate Tool in the Universe: (Score:5, Funny)
Parent
Re:Dear Every Corporate Tool in the Universe: (Score:5, Funny)
Parent
Re:Dear Every Corporate Tool in the Universe: (Score:5, Insightful)
Hell, they could have spent $50 on a USB hard drive (i.e., half-assed it) and been better off!
Parent
Re:Dear Every Corporate Tool in the Universe: (Score:5, Funny)
Screw that!! IT Departments are cost centers and have absolutely no benefit to the bottom line of a company... none at all... nope.
Parent
rm -rf / (Score:5, Informative)
That is one reason why mirroring isn't a backup, and why backups should ideally be off-line.
Re:rm -rf / (Score:5, Funny)
C:\>rm -rf /
'rm' is not recognized as an internal or external command,
operable program or batch file.
Everything's still running here...
Parent
Ouch (Score:4, Informative)
Re:Ouch (Score:5, Insightful)
Or even one, stale, backup.
Parent
You need more than backups ... (Score:5, Insightful)
Parent
Excellent! (Score:5, Funny)
Re:Excellent! (Score:5, Funny)
Ironically, it's more useful than the entire collection of blogs that they stored.
Parent
That's what backups are for (Score:5, Interesting)
It's really unfortunate that this happened. If they had simply had a backup snapshot of the DB they could have restored it. RAID only saves you from disk failures. It doesn't work on OS/user failures.
Unfortunately this is the kind of thing you tend to learn from experience (either yours or someone else). It's very easy to think "RAID 1 = disks are safe".
Just like a database cluster wouldn't have saved them. A clustering database can save you from load, or you can swap servers if a disk goes bad. But when someone issues "DELETE * FROM..." the other cluster nodes start to happily run the same thing and now you have 2 (or 3 or 10 or...) empty database boxes.
I hope those bloggers had a backup of some sort of their own.
Re:That's what backups are for (Score:5, Insightful)
My guess (and this is a guess, I'd never heard of the site before yesterday) is that this is some guy who started his own little site and it got bigger and bigger. Basically he never designed the backup, the system was just slowly pieced bigger and bigger until it got to it's current state.
The comments in the messages from the site's operator about the cost of the drive recover and thinking both drives just died at once indicate to me that this site was basically a hobby for him and he isn't experienced as an admin.
Parent
El Oh El (Score:4, Insightful)
That's all I can say at this. I'm really surprised that with all the users they had, they are so quick to say "everything is gone and we're giving up" instead of just starting over and maybe implementing protocol that would make sure this doesn't happen again.
Re:El Oh El (Score:5, Insightful)
Considering how complete and unrecoverable the loss is, they have no idea who their users are. The accounts would have to be recreated from scratch, but who would try? Their users have no reason to ever trust them again. Journalspace would have a difficult time wooing back their original users, and no new user would seriously consider using them.
Bowing out is the only recourse, but I'm glad they're considering releasing their source code.
Parent
How hard is it to remember: (Score:5, Insightful)
Mirroring: High availability
Backups: High reliability
The rules of backups (Score:5, Informative)
The rules of backups:
1. Backup all your data
2. Backup frequently
3. Take some backups off-site
4. Keep some old backups
5. Test your backups
6. Secure your backups
7. Perform integrity checking
To the HR department (Score:5, Funny)
A lesson for admins, and users too (Score:5, Insightful)
No doubt this incident is the result of the admin's fault. He's been confusing mirroring and backup and carried on the mistake until it's too late, as pointed out in other comments.
Now what about a user's angle? The morale is you can never think your data is safer when it's "in the cloud". If you value your blog and your readers, you *should* save a copy of your work as well as the readers' info, *locally*, somewhere you have control over.
There's no place like $HOME.
Re:A lesson for admins, and users too (Score:5, Insightful)
And a corollary to the parent's good advice: if you can't easily get a complete copy of your work, find another host. Manual one-by-one downloads don't cut it.
Parent
No Archive.org either (Score:5, Informative)
They also purposely blocked archive.org via a robots.txt exclusion, so the bloggers can't use that to try and recover some of their blogs.
Google cache diving (Score:5, Informative)
Looks like at least some content is still in Google's cache [google.com], those looking to salvage their journals should act quickly.
You can limit google's search results to a particular site by using the "site:domainname.com" search term (example [google.com]) and then click the "Cached" [209.85.173.132] links of each result to see Google's copy.
There's also a Greasemonkey script [userscripts.org] for Firefox that can automatically add Google Cache links next to page links, so you can navigate from one cached page to another easier.
Parent
There is a denial going on (Score:5, Insightful)
In today's world where primary storage and protection storage are well-defined, and where entire industry grew around it (examples: NetApp, Data Domain), one is hard-pressed to understand the reason for such a debacle. The reading of the note referred to in the article [journalspace.com] leads me to believe, unfortunately, that Journalspace's IT department did not understand the difference.
It is sometimes considered a bad form to say something bad about fellow techies. We prefer to look for 'outside' causes. Still, to learn and avoid the same problems in the future, one has to admit his mistakes first. This paragraph from the Journalspace's page:
The value of such a setup is that if one drive fails, the server keeps running, using the remaining drive. Since the remaining drive has a copy of the data on the other drive, the data is intact. The administrator simply replaces the drive that's gone bad, and the server is back to operating with two redundant drives.
makes me believe there is a denial going on.
Mirroring (Score:5, Insightful)
Re:Mirroring (Score:5, Funny)
There's a major flaw in your analogy. See, if I stick a fork in my right eye, the mirror image will stick a fork in his left eye. Between the two of us, however, we still have one good left AND right eye. So ipso fatso, I have a complete backup.
Parent
Personal backups of online data (Score:5, Insightful)
Do the big kahunas of the "Web 2.0" world give users that option? Gmail, Myspace, Facebook, Twitter etcetera ad nauseam?
Re:stunned silence (Score:5, Funny)
I am experiencing a strange phenomenon. The jaw-drop reflex has been popping my mouth open for several minutes and won't stop. If I focus I can close it, but then it pops open again. wow.
Parent
Re:Noobs. No, really. (Score:5, Informative)
Even the greenest IT employee knows that mirroring is to protect against hard drive failure and not software corruption.
I only wish that were true. I've given up arguing with friends about this, who insist that their mirrors are good enough backups. I just stare at colleagues who think such, especially those who SHOULD know better. And I *know* coworkers are doing this @ work, too, and I'm just waiting for about 50TB of data to suddenly go missing...
Parent