Why Mirroring Is Not a Backup Solution 711
Craig writes "Journalspace.com has fallen and can't get up. The post on their site describes how their entire database was overwritten through either some inconceivable OS or application bug, or more likely a malicious act. Regardless of how the data was lost, their undoing appears to have been that they treated drive mirroring as a backup and have now paid the ultimate price for not having point-in-time backups of the data that was their business." The site had been in business since 2002 and had an Alexa page rank of 106,881. Quantcast said they had 14,000 monthly visitors recently. No word on how many thousands of bloggers' entire output has evaporated.
When is backing up *not* an option? (Score:5, Interesting)
Mirroring, RAID, grid, whatever. At some point, you want your data safe and secure on something not physically attached to any power source.
That's what backups are for (Score:5, Interesting)
It's really unfortunate that this happened. If they had simply had a backup snapshot of the DB they could have restored it. RAID only saves you from disk failures. It doesn't work on OS/user failures.
Unfortunately this is the kind of thing you tend to learn from experience (either yours or someone else). It's very easy to think "RAID 1 = disks are safe".
Just like a database cluster wouldn't have saved them. A clustering database can save you from load, or you can swap servers if a disk goes bad. But when someone issues "DELETE * FROM..." the other cluster nodes start to happily run the same thing and now you have 2 (or 3 or 10 or...) empty database boxes.
I hope those bloggers had a backup of some sort of their own.
Re:That's what backups are for (Score:2, Interesting)
Even a moment of thought would have made it abundantly clear that this was not a backup situation, and it certainly should not require loss to pound it into someone's head.
They were clearly way over their heads if they thought this protected them from anything other than a single drive failure. More likely they were entirely aware of the risk, but decided to wing it anyways.
Re:No Archive.org either-Compound Foolishness (Score:3, Interesting)
This is just compound foolishness. I gather they did it in an attempt to control bandwidth costs since it's hard to imagine any other reason.
Re:When is backing up *not* an option? (Score:4, Interesting)
Nope.
Mirrors are fine, just snapshot them and store them offsite regularly. Do delta backups as needed but close-in for fast restoration.
There is no rational justification for tape anymore, what with the cost per TB stored on hard disks now under $130, total $$. Random accessibility unless you're stalling a subpoena, is just mandatory on backup media.
Yes, it is a backup solution (Score:2, Interesting)
Only wimps use tape backup: _real_ men just upload their important stuff on ftp, and let the rest of the world mirror it ;)
Double Duh! (Score:5, Interesting)
Re:Ouch (Score:3, Interesting)
Working at several hosting places I would say,you are correct. Never trust a hosting service backup. I always told our customers to never trust our backup. Sometimes backups just never happened. They are not high on the list of things to keep working.
Re:Google Cache to the rescue? (Score:3, Interesting)
Re:That's what backups are for (Score:4, Interesting)
Ah, it totally depends on the type of database cluster. For example, with Oracle, if you're using Oracle DataGuard, even in synchronous replication mode you can define an "apply delay" - basically, "Don't acknowledge this commit until it is written locally, and copied and acknowledged on the remote side, but don't actually apply the transaction for two hours"
That way, if someone does a delete * from blogs;, it will be reflected immediately on the production, but you've got a nice window to sort it out.
Plus, if you've got database flashback turned on, you can simply say, "Flash my database back to what it looked like before someone was an idiot", and all your data comes back.
These features are expensive in Oracle, but they can be very useful when you actually need them.
Re:El Oh El (Score:2, Interesting)
Their post said that only the task-specific server for data was hosed. If Journalspace offered paid services, then their billing system should still have all their customer's details.
Re:You need more than backups ... (Score:3, Interesting)
The best way I have found to test the backup is to nuke the data and restore.
Seriously, if you know what files store the data (and that you are backing up), just stop services and rename a directory or two so the data is "gone". Then, restore from backup, start the service, and see how things look. Another good way is to restore the data to a VM that runs the same software as the production server. You can sandbox a simulation of the entire Internet inside a few VMs if you want, and test what happens.
I just did something similar when I upgraded the OS on a VM that runs a MySQL server:
Basically, if things had gone poorly, I could just stop the new VM and revert back to the old one.
Re:When is backing up *not* an option? (Score:3, Interesting)
that's where trixter needs to zfs send/recv the snapshots to an offsite location (and probably roll snapshots more frequently)
OS X Server (Score:3, Interesting)
The site was run on OS X Server... I think this may be indicative of the level of IT effort with the company. Look, *I* run an OS X Server... but *I* am a Biology major that knows approximately dick about the UNIX command line, and use it to run a server that I probably wouldn't be able to run any other way. I also have it backup nightly to a cheap NAS, archiving old backups, and I've tested a restore to make sure it works.
This is probably just a couple guys who ran a website in their spare time... not a huge IT effort that failed.
Re:Just give up? (Score:3, Interesting)
Thats bullshit, and has been for decades.
Its a myth. Just learn about it. Even if we use our newest AFM, or XMCD microscopy, you wont see an overwritten byte in any drive of the last 5 years. And even the last decade is very doubtful (basically, since GMR drives are around).
There IS NO SPACE between tracks anymore. Bits are right next to each other. If you overwrite, nothing above the superparamagnetic limit is left.
Not even the NSA could get anything useful out of a single overwrite with zeros (well, except relocation sectors and other specialities that might compromise security, but doesnt help with a backup)
Re:Dear Every Corporate Tool in the Universe: (Score:2, Interesting)
Even better if the drive goes home with the admin at night.
Would the admin be tempted to look at other people's data?
Re:Dear Every Corporate Tool in the Universe: (Score:3, Interesting)
Never underestimate the beancounter's desire to save every cent possible.
That's contrary to my experience. Other expenses have been skimped on occasionally, but just mention the word "backup" and the funding was there.
I remember Karl Rove had a JournalSpace.... (Score:2, Interesting)
Re:Double Duh! (Score:3, Interesting)
did they ever fix their multithreading issues, where the performance went to shit as soon as you got more than 4 threads going at once?
THAT is what puts me off os x server, not teh pretties
Re:When is backing up *not* an option? (Score:3, Interesting)
Disks are still crap for archival storage and for most of us they are still crap for daily offsite backups. It would be nice to have a second site and a really fast, cheap way to get terabytes of data to it - but for me the expense of that is a lot more than taking a box of tapes to a storage shed. Eventually those daily tapes become archival tapes when projects finish. To cap it all off, most of the incoming data where I work comes in on tape. DVDs are utter crap when you want to be sure a lot of data can actually be read at the endpoint and only hold 4.5GB anyway. Portable USB drives are replacing tapes for transport but they are just as slow to read and fragile to transport, and even then you get the time consuming crap of spending twenty minutes going around the building to find someone with the same sort of USB drive because it came without cables.
The final point is I really do not want to have to maintain the systems to have a couple of hundred terabytes of storage when the working set is well under 5 terabytes and there would be a lot of repition in that few hundred terabytes of recovered tapes. It then also raises security problems to ensure client data is not easily accessable by other clients. Disks are not the best for archival storage. One of the reasons I was using tapes from 1982 is that nobody was interested in the area the data came from between that data and this year. Disk space is still too expensive to keep two and a bit copies of something big for more than twenty years :)
Tapes are often annoying and I sopmetimes wish we could be rid of them but there are many circumstances where they are useful - paticularly archives, transport and backups. While I've been happily using rsync to take daily snapshots of important stuff for years and distributed it to machines in different buildings it still cannot always replace tape. I've had too many dead hard drives or unreadable DVDs (plus they are too small) to seriously consider them as archival storage. As for transport - why are USB hard drives so horribly slow? The ease of random access is irrelevant when you want a lot of files from the disk so LTO and SDLT still win there.
Re:Double Duh! (Score:3, Interesting)
"I think I can reveal this much: an EX IT guy didn't do something he was supposed to have done. This wasn't discovered until AFTER the disks crashed. So, there were probably other reasons for this guy being an EX, too. Anyway, the crash happened, the mistake was discovered but now too late to fix."
From: http://dorrie.de/F1/viewtopic.php?t=194&start=375&sid=a65b1d9b0fbcc3c3e8df874fc167d495 [dorrie.de]