UK ISP PlusNet Accidentally Deletes 700GB of Email 282
steste writes "A tale of email woe for PlusNET ISP. According to this announcement they have spent the last month attempting to recover 700GB of accidentally deleted emails. By their estimates, up to 12GB of these had yet to be read by their recipients. Despite the efforts of a data recovery specialist, they have now given up on recovering any of the deleted data.
Well that's one way to deal with spam." Spam is one thing; I just wonder how inevitable losses like this one square with the EU-wide data retention laws.
Conspiracy (Score:5, Funny)
Yeah, right!
Re:Conspiracy (Score:2)
Only the other week they emailed their Force9 customers their customer database [theregister.co.uk], for all to see.
I used to be with Plusnet a few years ago (about the time they kept installing new "fat pipes" and "redbacks"), and they were awful then. The service was horrible and their web setup was laughable. That said, they're not the worst out there, which says something...
Re:Conspiracy (Score:5, Funny)
Re:Conspiracy (Score:5, Insightful)
On the plus side.... (Score:3, Funny)
Re:On the plus side.... (Score:5, Insightful)
Shall I repeat it again to increase my karma?
X (puzzled)
I guess if I look at my email (Score:5, Funny)
Re:I guess if I look at my email (Score:5, Interesting)
Re:I guess if I look at my email (Score:5, Interesting)
Re:I guess if I look at my email (Score:2)
If your back-up system isn't regularly tested and found to be restorable, then you don't have any, no matter how many tapes/spare hard drives/CD-Rs/whatever you use.
I do wonder at what stage this sort of cock-up becomes outright negligence. I haven't checked my terms of use recently, but I doubt disclaimers will hold up for long after this sort of mess anyway.
Re:I guess if I look at my email (Score:2)
1) my ISP blocks my smtp server
2) my spam filters aren't anywhere near as good as those i am provided with gmail or even my ISP's servers..
Re:I guess if I look at my email (Score:4, Insightful)
2) Use Spamassassin, and tune it according to the WIDELY available docs. SA even runs on Windows.
Re:I guess if I look at my email (Score:4, Interesting)
Well, check around. I found most ISP's will offer a business acccount. It is a few dollars a month more than regular, but, like with mine from Cox Cable...I have:
Check into the 'business account' that many if not most ISP's provide. It may cost a little more, but, that freedom to do and host what you please is quite worth it...to me at least.
Re:I guess if I look at my email (Score:3, Funny)
The guy that actually 0wnz your home computer would.
Re:I guess if I look at my email (Score:5, Interesting)
Then one day, I lost several hard drives in one go (storm). Last week backups were corrupted, had to roll back to 2 weeks backups. I lost 400 mails, my gf lost about 10 mails.
It's been six months, now I forward everything to several gmail/yahoo accounts, do a local backup of thunderbird's mboxes, and scp the backups to two different machines.
To this day, I still can hear the screams from my One And Deareast Female User when I cry myself to sleep.
Bottom line:
- If you are single, yeah, host your server, it's fun, you learn a lot of stuff.
- If you are not, paranoïa should be part of your base skillset.
Re:I guess if I look at my email (Score:4, Interesting)
Running your own mail server provides a number of advantages :
On the other hand, it does require a little knowhow (with the number of HOWTOs floating around it's really not that hard), a dedicated connexion (although nowadays with widespread xDSL or cable this is much less of a problem than it used to be just a few years ago) and some kind of backup system. Some ISPs won't let you run a SMTP server though, in which case there really isn't much you can do (especially if it's the only ISP in your area as seems to be frequently the case in the US).
For backups I use a little NAT box with 300GB of RAID 1 storage to dump the essential stuff on my network, including the mail directories. They are also copied to a secondary drive inside the mail server.
Of course as the story illustrates, there is no absolute security. Your house may burn down, an asteroid may obliterate your region, or you might just type a space in the wrong place in your "rm" command
Apart from that, I've run my own mail server for ages now and I know I wouldn't want it any other way.
Re:I guess if I look at my email (Score:2)
Single or Not, Paranoia should be part of you skill set. Paranoia I'm sure has stopped some problems on my part. And while it's hard to quantify problems you don't have, I remember the problems I did have before I gained a certain level of paranoia.
Now, I must prepare for the people who are watching me.
Re:I guess if I look at my email (Score:2)
Re:I guess if I look at my email (Score:4, Insightful)
Spam became such a nuisance that I recently migrated to Google's free Gmail for your domain [google.com] hosting service. It's webmail and POP3 client complient and the spam filter is a friggin marvel. It intercepts at least 199 of every 200 spams. I highly recommend their service! Free access gets you 25 addresses with 2GB each!
Re:I guess if I look at my email (Score:2)
Clicked on the Read More link just now (Score:5, Funny)
404: The requested URL (hardware/06/08/03/1319220.shtml) was not found.
Looks like PlusNet aren't the only ones losing things!
I'm here all night, try the veal.
mudda mudda (Score:2, Funny)
Re:mudda mudda (Score:2)
Hmm. Good thing they're British... (Score:4, Funny)
I imagine that if this sort of thing were to happen in the U.S., the government would get involved real quick. Not, you know, because some subscribers got hurt, but because all of those precious, precious terrorist communications that were lost forever, dooming the Fort Worth Convention Center to premature destruction at the hands of an angry Palestian truck driver.
Or maybe this can't happen in the U.S. at all. Maybe there's some quiet deal where large ISPs can simply back their data up on blade servers in Langley...
Re:Hmm. Good thing they're British... (Score:2)
More likely the gov't intercepts and archives them independently, and doesn't grant even the ISPs access. If it happened in the U.S., some gov't employee would be reading those 12GB of unread emails right now and saying to himself "oh man, john137@hotmail.com is going to be realllly pissed when he finds out they shipped his Viagra to the wrong address!"
Re:Hmm. Good thing they're British... (Score:3, Funny)
That's what they WANT you to think.
That's not what the laws are for (Score:3, Informative)
Re:That's not what the laws are for (Score:4, Interesting)
Re:That's not what the laws are for (Score:2)
Re:That's not what the laws are for (Score:2)
Re:That's not what the laws are for (Score:2)
Bad news for those who use email as a file cabinet (Score:5, Interesting)
But I know people who use email accounts as a repository for their online lives. Gmail is encouraging this attitude, of course. Now I think Google is probably a little more responsible, but it does give one pause.
Now, for that unread email, that just sucks eggs for those poor people...
Re:Bad news for those who use email as a file cabi (Score:2)
I store every "insecure" e-mail I get on GMail and I'm happy to delete it from my server and my mobile device knowing that it's likely going to be available via the web.
For "secure" and/or "important" e-mails, they get stored locally or on my mobile device and possibly even printed out and locked a
what do you mean by "secure"? (Score:4, Insightful)
What do you mean by "secure"? Surely you wouldn't trust anything that is a security concern with SMTP and possibly also POP3, two protocols where everything is sent plaintext.
Re:what do you mean by "secure"? (Score:2, Insightful)
Four words for you:
----- BEGIN PGP SIGNED MESSAGE -----
Re:Bad news for those who use email as a file cabi (Score:4, Interesting)
I have every e-mail I have ever sent or received except for spam. I can't count how many times this has been useful. I don't want to waste time trying to figure out what I should keep or not, I keep it all. I do keep all of them local on my own hardware though. This allows me to protect and backup my own data.
This is just a case of a poor backup strategy causing data loss that should not have happened.
Data Recovery Specialist (Score:3, Interesting)
Re:Data Recovery Specialist (Score:3, Interesting)
700GB was definitely over a couple of disks, probably a RAID box. If you lose a RAID controller in SOME(not most or all) brands of RAID boxes, it's very difficult to rebuild the map. I saw an NStor RAID box lose a controller, and it was impossible to recover.
I wonder of how much was recovered was from tape? 12GB of unread email probably means "we didn't have a backup of the 12GB", though that's a L
Re:Data Recovery Specialist (Score:2)
No: "Accidentally deleted" means that a technician was working on a backup storage system, but also had a window open to the primary, and reformatted the wrong one.
http://www.theregister.co.uk/2006/07/11/plusnet_em ail_fiasco/ [theregister.co.uk]
Re:Data Recovery Specialist (Score:3, Funny)
How very apt
Re:Data Recovery Specialist (Score:2)
Re:Data Recovery Specialist (Score:5, Funny)
"Accidentally deleted" probably means "we had a hardware failure and we're too cheap to recover everything".
Actually, "Accidentally deleted" means "wiped the live disk array instead of the new disk array we were going to migrate on to." The Register has a brief writeup [theregister.co.uk].
System Administrator Lesson 14: Shared consoles and remote administration are convenient, especially for wiping the wrong system. Check system ID before hitting enter!"
Re:Data Recovery Specialist (Score:3)
Re:Data Recovery Specialist (Score:2)
Re:Data Recovery Specialist (Score:3, Funny)
Re:Data Recovery Specialist (Score:3, Insightful)
Spam wonderful spam (Score:2, Funny)
Shh! You insensitive clod! Some of us need that spam...how else are we supposed to maintain powerful erections with that "s3xy b@by"!
Deleted emails? It obviously didn't run linux...ah no wait, hang on.
A quote from the ISP (Score:5, Funny)
Re:A quote from the ISP (Score:2)
Re:A quote from the ISP (Score:3, Funny)
Accidentally (Score:2)
I just wonder how inevitable losses like this one square with the EU-wide data retention laws.
Well, since it was an accident... wait, or was it?
Spam is heavy (Score:3, Informative)
Re:Spam is heavy (Score:2)
Do yourself a favor: put a *nix box (or 12) in front of your exchange box running an MTA with clamav / spamassassin on it. If *nix isn't your gig, buy an appliance (there are many) or outsource your MX to a service. Frankly, IMHO, Exchange should NEVER EVER be directly connected to the internet (for many reasons. Search google.)
I setup a simple (near zero admini
Re:Spam is heavy (Score:2)
I currently block over 11 thousand spams a day.. and that's for a small company with 4 employees. I'd hat
Re:Spam is heavy (Score:2)
Re:Spam is heavy (Score:2)
12GBs yet to be read? (Score:5, Funny)
Here's the break down:
Re:12GBs yet to be read? (Score:3, Informative)
Re:12GBs yet to be read? (Score:2)
Pardon me, I completely forgot that spam filters are 100% foolproof [slashdot.org]
Re:12GBs yet to be read? (Score:2)
Technical details (Score:5, Informative)
Re:Technical details (Score:2)
It's still an interesting story though, with other aspects making things even harder for
Re:Technical details (Score:3, Informative)
dd if=/dev/buggered_disk of=image
And do all your work on the image file!
IDIOTS!
So...where were the backups? (Score:3, Interesting)
Or am I missing the point and actually 700gb of mail was just one day?
I'm a PlusNet user - but not for long (Score:5, Interesting)
I've been with PlusNet for years, and they were a pretty good ISP until a few months ago. Since then, we've had a string of problems, of which this is only the latest.
I've had my broadband connection out more than on for weeks at a time, for a start. This in itself is inexcusable. What's even more inexcusable is telling me I had to accept a significant penalty charge if they escalated the fault to my telephone service provider (BT) and they found no fault -- which doesn't sound unreasonable, until you know that the fault was evident using nothing but PlusNet-supplied hardware plugged into a BT-installed phone socket, with no complications whatsoever, and that PlusNet had already indicated that they themselves couldn't diagnose a fault. This was a total loss of service for hours at a time, several days a week, remember.
On top of that, they decided to forcibly upgrade everyone to "up to 8MB" broadband recently. The ethics of using that term are dubious at best: it's only for downloading; the highest recorded speeds off-peak are more like 5-6; and at peak times you'll be lucky to get more than 1-2. Moreover, they acknowledged ahead of time that there would be significant disruption (for weeks, not hours) to each customer after the upgrade, they said they wouldn't confirm when any given customer was being upgraded (so no idea whether the problems I had were to do with this or some more general issue, then) and they said some customers' performance would actually drop but they wouldn't revert the change if this happened. They had so many problems with this that they have now suspended/abandoned the process, and sent a grovelling e-mail message to their customers.
Their tech support people have also been completely over-run, partly due to inadequate resources and partly due to their own incompetence (e.g., they totally failed to read a note I'd helpfully left on their system for them clarifying a question they always ask, and asked the question in boilerplate form anyway). To add insult to injury, they've changed their phone system in ways that have repeatedly broken, and now mean you go through several layers of automated menus before talking to a real person. Yes, they really did tell me at one stage that if I was experiencing broadband connectivity faults, I could find more information on their web site.
And now, of course, we have the e-mail fiasco. It's not the first big e-mail problem: I've recently had legitimate and important messages from the sysadmins of another service I use being bounced because they "contained a virus". (Not according to the other service, whose admins I know and trust, nor according to one well-respected intermediate service that was involved in forwarding the mail.) Moreover, this occurred even when I disabled virus checking for incoming e-mail; they were blocking incoming messages to me against my explicit instructions. Oh, and their new webmail system is poor in functionality and so bug-ridden that you can actually lose data. Some of this, in particular an arbitrary time-out for composing mails using webmail, was regarded as a feature when I asked the support staff about it!
I don't know what's happened to PlusNet. Perhaps they have simply been victims of their own success, after getting very positive comments for years (they were widely regarded as one of the best ISPs in the UK for a while) and a consequent boost in custom? In any case, the mighty have well and truly fallen, and I (along with many other people I know) am currently investigating alternatives as a matter of urgency.
Re:I'm a PlusNet user - but not for long (Score:4, Interesting)
Their customer service has never been particularly superb, it's true: they messed me around a bit when I was initially getting broadband installed, for example. Still, until maybe the start of this year, the tech support guys always seemed to know their stuff, and any problems I did have usually resulted in a fairly quick solution after a fairly short wait to speak to a real and knowledgable person on the phone. I've noticed a really obvious drop in their service levels over the past six months or so, though.
Re:I'm a PlusNet user - but not for long (Score:2)
Yes, the BT callout charge is exactly what I was objecting to as well. They refused to budge, repeatedly stating that I had to accept responsibility for the charge before they would escalate the fault report. Eventually, I got one of their old-timer support guys to say categorically that the policy was just because so many people don't bother to check obvious things like hardware being properly connected up before reporting a fault, and if I had done everything on their checklist and found no fault on my si
These Things Happen. (Score:3, Interesting)
It just goes to show, for PlusNet's customers, that electronic systems cannot be fully trusted, even when and if multiple instances of the data exist. We can approach an approximation of 100% reliability, but we can't ever fully eliminate the possibility of data loss, especially when human error is involved.
Another time in an incident that is mostly unrelated to the topic at hand but makes for a good story, I had a customer who lost their array in a PC server. The machine had an array of full height HDDs that would get so hot within five minutes of power up that you couldn't touch them without gloves. To top it off, the tape drive mechanism that was supposedly backing up the system was sitting directly above the arrary, with a backup job that had been running over and over nightly on the same cartridge for over two years (so you can be certain the tape was useless).
It took about three weeks, but I was eventually able to recover all the data on the array (so far as we were able to determine at the time).
NSA has 'em (Score:5, Funny)
Heck, the NSA could turn this into a side business. If they spin it right, maybe they can convince the general public that they're not spying, they're just providing a cutting-edge data backup service!
Re:NSA has 'em (Score:3, Funny)
Do they support rsync over ssh?
Only 700 Megs? (Score:2)
Quality Company (Score:4, Insightful)
Re:Quality Company (Score:2)
They provide free static IPs so I run my own mailserver.
Never press the shiny candy like red button!!! (Score:3, Funny)
*click*
poor intern (Score:5, Funny)
(I'm so nervous seeing that on my screen I'm afraid to hit the "Submit" button)
Re:poor intern (Score:3, Funny)
No, it shouldn't take half an hour. You obviously need to get a faster computer, faster disk, and configure the DMA correctly :-)
Oh, and I hid some files in your root directory for you to practice on.
"Daddy..." (Score:4, Funny)
-NOOOOOO!
Am I missing something? (Score:2, Insightful)
If they've got the e-mails why not just re-queue it? Surely the "To" field is a bit of a give-away.
Spam Filter (Score:3, Funny)
Filter with 0% Tolerance
Are you absolutely sure?
[ok] [cancel]
I don't understand how this is possible... (Score:5, Funny)
All eggs in one basket? (Score:3, Funny)
They had all their eggs in one basket? Where's the live replica machine? Where are the redundant copies? Oh wait, this is a for-profit business. Never mind.
Oh, woe... (Score:3, Funny)
I may be out of date on this... (Score:2, Interesting)
difficult to create a universal standard (Score:3, Insightful)
Yes.
Does anyone know how to reach the sysadmin? (Score:2)
Spam, data retention laws (Score:2)
Re:Welcome to three weeks ago (Score:5, Informative)
If you read you post, they were calling in the recovery speciallists to try and get it back.
They failed, its game over for recovering anything.
your objection is SO last week (Score:4, Insightful)
Besides, it always mystifies me that people who feel that their time is wasted by duplicate or outdated stories have no problem wasting more of their time, not to mention server space and the time of all the readers, posting "this has already been covered." Do you get karmic cool points for ranting (again) about (another) dupe? What's the payoff? Does it make you happy? I'm not the most fanatically efficient person out there, but it seems petty and, well, stupid to not only dwell on, but to go to the point to complain in writing about the dupe or outdated story, which actually raises the net energy and time spent on this problem that you ostensibly found so vexing. No, I'm not complaining about you, only wondering what the hell you find so moving about the whole issue. Is it just the principle? A matter of pride? Does it bode ill for humanity? What gives?
Re:your objection is SO last week (Score:2)
hear hear (Score:4, Funny)
Do the abilities required for such a venture require
a) A willingness to try and realise there will be the odd piece of news that is repeated.
b)Being a Whining Biatch.
Well, if it's (b), then the whiners are for teh win. Somehow I have my doubts...
Re:Welcome to three weeks ago (Score:4, Insightful)
From the article:
Anyone who has inadvertently typed an 'rm -rf' should now feel a bit better.
I do wonder whether this will cause people (and companies) to re-evaluate the growing popularity and hence reliance on web-based email. Myself, I don't go near it. Leaving the reliability concerns, and ignoring the historically bad reputation of services such as Hotmail, the spammy footers and similarly badly formatted garbage that users of web-based email end up sending everyone else, I can't fathom why it's so difficult for someone simply to log in remotely to a server that their company manages, or their own box at home. I hear you can even use those same tubes to do it.
This incident makes for a good argument, but my guess is that people will want to continue use their browsers for everything and similarly continue to rely on companies they think they know.
Re:Welcome to three weeks ago (Score:3, Insightful)
I wouldn't rely on a webmail much if it's from some random local ISP, but if it's about a company having a major part as its company profile to provide webmail, like Microsoft, Google, or Yahoo!, I think you'd be pretty safe. I can't imagine these lacking some healthy amounts of redundancy as it would be devastating if e.g. Gmail suddenly crashed
Re:old news (Score:2)
Re:And on and on and on... (Score:4, Interesting)
Re:And on and on and on... (Score:2)
Re:And on and on and on... (Score:3, Informative)
They give you a static IP and don't mind if you run a mail/web server on it - my local website was Slashdotted, maxing out my upstream bandwidth for a couple of days and they didn't complain.
But then, I was wi
Re:And on and on and on... (Score:2)
I asked about alternatives on a local forum recently. Some of the responses were a bit surprising. Our local cable firm is NTL, for example, and despite having a less than stellar reputation generally, it sounds like their ISP wing is doing its job fine (at least, according to a few locals who use it).
I've also heard good reports about some of the smaller but significantly more expensive ISPs. These are backed up by various ISP comparison web sites, where the smaller outfits consistently get way higher fe
Re:And on and on and on... (Score:2)
In any case, I have no problem with usage quotas as long as they're transparent. As someone who doesn't spend all day illegally ripping things over P2P, and whose monthly bandwidth only gets anywhere near the current 3GB threshold I have on the rare occasions that I download several very large installers in one month, I'm quite happy not subsidising the aforementioned P2Pers.
Now, signing someone up for an "unlimited" package and then imposing usage quotas without a corresponding drop in price, that's a di
Re:Googlymail (Score:2)
Re:Compensation for Customers? (Score:2, Funny)
Yeah, that would be pretty terrible. So to spare you that fate... here is exactly what you would be missing out on:
Hell, I could probably recover