UK ISP PlusNet Accidentally Deletes 700GB of Email

steste writes "A tale of email woe for PlusNET ISP. According to this announcement they have spent the last month attempting to recover 700GB of accidentally deleted emails. By their estimates, up to 12GB of these had yet to be read by their recipients. Despite the efforts of a data recovery specialist, they have now given up on recovering any of the deleted data. Well that's one way to deal with spam." Spam is one thing; I just wonder how inevitable losses like this one square with the EU-wide data retention laws.

Conspiracy

"Accidentally deleted", "despite of best efforts"....

Yeah, right!

Re:Conspiracy

You guys have fat pipes? We're still stuck with tubes. They're getting clogged up with internets, too.

Re:Conspiracy

The real question is, was the admin fired before, or after the deletion?

On the plus side....

On the plus side, that's 699GB of spams nobody will have to worry about now.

Re:On the plus side....

How the h*ll does a repeat of a joke that was already made in the article itself get modded up as +5 Funny? Instant replay?

Shall I repeat it again to increase my karma?

X (puzzled)

I guess if I look at my email

My inbox will say, nothing to see here. Move along.

Re:I guess if I look at my email

That's why I like to run my OWN email server...I can make sure my backups work, and if it fails, I have only myself to blame.

Re:I guess if I look at my email

I lost about a thousand emails (I had read most of them, thankfully, only really lost-lost about 20-30, many non-critical) a few months ago because back-ups also failed. I guess that's what I get for doing webmail instead of downloading to an actual client. Oh well, wasn't really my fault. And to be honest I'm not sure I trust the report about back-ups failing. I suspect they didn't have any. At least now I'm back to using a client, where I can do my own back-ups locally.

Re:I guess if I look at my email

1) Use your ISP's servers (or other email service) for outbound mail. In 99.9% of "blocking" cases, port 587 (the MSA port) is still open so you CAN use some other service. Inbound is generally not an issue (unless your ISP is on Planet Stupid. In this case, get a real ISP and not one that delivers "damaged" service.)
2) Use Spamassassin, and tune it according to the WIDELY available docs. SA even runs on Windows.

Re:I guess if I look at my email

"my ISP blocks my smtp server...my spam filters aren't anywhere near as good as those i am provided with gmail or even my ISP's servers..."

Well, check around. I found most ISP's will offer a business acccount. It is a few dollars a month more than regular, but, like with mine from Cox Cable...I have:

• Static IP
• No bandwidth limits
• No Ports blocked
• Basic SLA

It isn't that much more money, and I can run any servers I want, hell, even make a little money hosting small site for people too...

Check into the 'business account' that many if not most ISP's provide. It may cost a little more, but, that freedom to do and host what you please is quite worth it...to me at least.

Re:I guess if I look at my email

I ran my own local imap store + fetchmail from my ISP + a very nice backup scheme (scp=>powerbook)

Then one day, I lost several hard drives in one go (storm). Last week backups were corrupted, had to roll back to 2 weeks backups. I lost 400 mails, my gf lost about 10 mails.

It's been six months, now I forward everything to several gmail/yahoo accounts, do a local backup of thunderbird's mboxes, and scp the backups to two different machines.

To this day, I still can hear the screams from my One And Deareast Female User when I cry myself to sleep.

Bottom line:
  - If you are single, yeah, host your server, it's fun, you learn a lot of stuff.
  - If you are not, paranoïa should be part of your base skillset.

Re:I guess if I look at my email

If you have aerial power lines power (common worldwide in the countryside) surges in a storm are indeed a potential problem so a secure power supply for your server is a must. In most cities, this isn't a widespread problem.

Running your own mail server provides a number of advantages :

• it's client agnostic (run an IMAP server)
  
• you can setup filters easily
  
• you can create as many addresses as you like and mangle them as much as you want
  

On the other hand, it does require a little knowhow (with the number of HOWTOs floating around it's really not that hard), a dedicated connexion (although nowadays with widespread xDSL or cable this is much less of a problem than it used to be just a few years ago) and some kind of backup system. Some ISPs won't let you run a SMTP server though, in which case there really isn't much you can do (especially if it's the only ISP in your area as seems to be frequently the case in the US).

For backups I use a little NAT box with 300GB of RAID 1 storage to dump the essential stuff on my network, including the mail directories. They are also copied to a secondary drive inside the mail server.

Of course as the story illustrates, there is no absolute security. Your house may burn down, an asteroid may obliterate your region, or you might just type a space in the wrong place in your "rm" command ;)

Apart from that, I've run my own mail server for ages now and I know I wouldn't want it any other way.

Re:I guess if I look at my email

I ran a POP3/SMTP server for many years. It was a great experience and I learned volumes. Especially about open relays. One day I investigated why no email was being sent or recieved, and found an 800MB cache file clogging up the works thanks to 40,000 spams being sent from China. That took a little while to mop up.

Spam became such a nuisance that I recently migrated to Google's free Gmail for your domain [google.com] hosting service. It's webmail and POP3 client complient and the spam filter is a friggin marvel. It intercepts at least 199 of every 200 spams. I highly recommend their service! Free access gets you 25 addresses with 2GB each!

Clicked on the Read More link just now

and got

404: The requested URL (hardware/06/08/03/1319220.shtml) was not found.

Looks like PlusNet aren't the only ones losing things!

I'm here all night, try the veal.

Hmm. Good thing they're British...

I imagine that if this sort of thing were to happen in the U.S., the government would get involved real quick. Not, you know, because some subscribers got hurt, but because all of those precious, precious terrorist communications that were lost forever, dooming the Fort Worth Convention Center to premature destruction at the hands of an angry Palestian truck driver.

Or maybe this can't happen in the U.S. at all. Maybe there's some quiet deal where large ISPs can simply back their data up on blade servers in Langley...

That's not what the laws are for

Data retention laws do not facilitate or even mandate data retention, nor are they designed to. All they say is that when (not even if) you lose, misplace or destroy data, the government will come and kick your butt into next Sunday. Which is what shall now unfold.

Re:That's not what the laws are for

I'm having trouble distinguishing between the two. How is a mandate supposed to be enforced if it isn't a statute that says, "If you don't do x and y, we will punish you?" Isn't that what a mandate is? Please clarify.

Bad news for those who use email as a file cabinet

Speaking for myself, a previously read email is trash. If there's something I want to save, I save it somewhere else, usually in a couple of places.

But I know people who use email accounts as a repository for their online lives. Gmail is encouraging this attitude, of course. Now I think Google is probably a little more responsible, but it does give one pause.

Now, for that unread email, that just sucks eggs for those poor people...

what do you mean by "secure"?

For "secure" and/or "important" e-mails, they get stored locally or on my mobile device and possibly even printed out and locked away for later retrieval. "Important" e-mails will be archived on GMail but "secure" ones never are.

What do you mean by "secure"? Surely you wouldn't trust anything that is a security concern with SMTP and possibly also POP3, two protocols where everything is sent plaintext.

Re:Bad news for those who use email as a file cabi

Eh? Using it as a file cabinet is how it should be used!

I have every e-mail I have ever sent or received except for spam. I can't count how many times this has been useful. I don't want to waste time trying to figure out what I should keep or not, I keep it all. I do keep all of them local on my own hardware though. This allows me to protect and backup my own data.

This is just a case of a poor backup strategy causing data loss that should not have happened.

Re:And on and on and on...

Indeed. Plusnet never accept any responsibility for their screw-ups (at least not if it will cost them anything). They're currently refusing to accept responsibility for their subcontractors' incompetience cauign lots of their customers to lose broadband for over a month.

Data Recovery Specialist

If the files were only accidentaly deleted and this was noticed in a timely manner, why would it be so damn diffacult to recover the files. I've seen data recovered from a hard drive that was on fire! (not while they were recovering the data :P )

Re:Data Recovery Specialist

"Accidentally deleted" probably means "we had a hardware failure and we're too cheap to recover everything".

Actually, "Accidentally deleted" means "wiped the live disk array instead of the new disk array we were going to migrate on to." The Register has a brief writeup [theregister.co.uk].

System Administrator Lesson 14: Shared consoles and remote administration are convenient, especially for wiping the wrong system. Check system ID before hitting enter!"

A quote from the ISP

"My bad."

Spam is heavy

I administer an Exchange email server for a small company. On average 60% of all our mail is spam and it adds up to several MB of spam per user per week. If users don't make a daily effort to delete spam, it does fill the email storage. Spam is more than annoying, it costs money in storage and processing. You may laugh at the ISP's problem but I have had to manually delete email from user's accounts when they would process their spam. Yes, we have a professional server spam filter, and it works for 99.8% of the time.

12GBs yet to be read?

By their estimates, up to 12GB of these had yet to be read by their recipients.

Here's the break down:

• 3GB: Viagra
• 3GB: Manhood Enhancements
• 2GB: Lonely Housewives in your area
• 3.9GB: Loans
• 100MB: Various GIFs of the Zidane headbutt from the World Cup

Technical details

According to this posting at ADSLGuide (which might be the text found at one of the links in the announcement linked to above), the initial problem was exacerbated by the technician trying to create a new volume of the same size as the one he had just deleted. This left a load of orphaned i-nodes on the second and third volumes. http://bbs.adslguide.org.uk/showthreaded.php?Cat=& Board=plusnet&Number=2600008 [adslguide.org.uk]

So...where were the backups?

The company I work for offers email services to our clients. We've had hardware failures of course, but then we also have our backups so that in the worst case you're going to lose a day (or whatever it is) of mail. So where were the backups in this case? Sure, it costs money to backup mail... but having seen the reaction from businesses when a day of mail is lost, I'd hate to be in the firing line if we lost more than that.

Or am I missing the point and actually 700gb of mail was just one day?

I'm a PlusNet user - but not for long

I've been with PlusNet for years, and they were a pretty good ISP until a few months ago. Since then, we've had a string of problems, of which this is only the latest.

I've had my broadband connection out more than on for weeks at a time, for a start. This in itself is inexcusable. What's even more inexcusable is telling me I had to accept a significant penalty charge if they escalated the fault to my telephone service provider (BT) and they found no fault -- which doesn't sound unreasonable, until you know that the fault was evident using nothing but PlusNet-supplied hardware plugged into a BT-installed phone socket, with no complications whatsoever, and that PlusNet had already indicated that they themselves couldn't diagnose a fault. This was a total loss of service for hours at a time, several days a week, remember.

On top of that, they decided to forcibly upgrade everyone to "up to 8MB" broadband recently. The ethics of using that term are dubious at best: it's only for downloading; the highest recorded speeds off-peak are more like 5-6; and at peak times you'll be lucky to get more than 1-2. Moreover, they acknowledged ahead of time that there would be significant disruption (for weeks, not hours) to each customer after the upgrade, they said they wouldn't confirm when any given customer was being upgraded (so no idea whether the problems I had were to do with this or some more general issue, then) and they said some customers' performance would actually drop but they wouldn't revert the change if this happened. They had so many problems with this that they have now suspended/abandoned the process, and sent a grovelling e-mail message to their customers.

Their tech support people have also been completely over-run, partly due to inadequate resources and partly due to their own incompetence (e.g., they totally failed to read a note I'd helpfully left on their system for them clarifying a question they always ask, and asked the question in boilerplate form anyway). To add insult to injury, they've changed their phone system in ways that have repeatedly broken, and now mean you go through several layers of automated menus before talking to a real person. Yes, they really did tell me at one stage that if I was experiencing broadband connectivity faults, I could find more information on their web site.

And now, of course, we have the e-mail fiasco. It's not the first big e-mail problem: I've recently had legitimate and important messages from the sysadmins of another service I use being bounced because they "contained a virus". (Not according to the other service, whose admins I know and trust, nor according to one well-respected intermediate service that was involved in forwarding the mail.) Moreover, this occurred even when I disabled virus checking for incoming e-mail; they were blocking incoming messages to me against my explicit instructions. Oh, and their new webmail system is poor in functionality and so bug-ridden that you can actually lose data. Some of this, in particular an arbitrary time-out for composing mails using webmail, was regarded as a feature when I asked the support staff about it!

I don't know what's happened to PlusNet. Perhaps they have simply been victims of their own success, after getting very positive comments for years (they were widely regarded as one of the best ISPs in the UK for a while) and a consequent boost in custom? In any case, the mighty have well and truly fallen, and I (along with many other people I know) am currently investigating alternatives as a matter of urgency.

Re:I'm a PlusNet user - but not for long

Their customer service has never been particularly superb, it's true: they messed me around a bit when I was initially getting broadband installed, for example. Still, until maybe the start of this year, the tech support guys always seemed to know their stuff, and any problems I did have usually resulted in a fairly quick solution after a fairly short wait to speak to a real and knowledgable person on the phone. I've noticed a really obvious drop in their service levels over the past six months or so, though.

These Things Happen.

I feel sorry for the guy who accidentally erased the array, and equally sorry for the people whose data was lost, but these things will happen. Just a couple of months ago, I myself had to dig through a few years worth of backups, because when I transferred abpout 100GB of files from one array to another under Mac OS X Server, I forgot to use the "ditto -rsrc" command, rather than "cp -Rp". Oops. All of the metadata for the files was lost. Not an unrecoverable situation, but it still cost me thousands of dollars in unbillable hours to correct the problem. You can be sure I won't be making the same mistake again, as I am sure the person who fracked the PlusNet system will more than likely never make an equivalent error.

It just goes to show, for PlusNet's customers, that electronic systems cannot be fully trusted, even when and if multiple instances of the data exist. We can approach an approximation of 100% reliability, but we can't ever fully eliminate the possibility of data loss, especially when human error is involved.

Another time in an incident that is mostly unrelated to the topic at hand but makes for a good story, I had a customer who lost their array in a PC server. The machine had an array of full height HDDs that would get so hot within five minutes of power up that you couldn't touch them without gloves. To top it off, the tape drive mechanism that was supposedly backing up the system was sitting directly above the arrary, with a backup job that had been running over and over nightly on the same cartridge for over two years (so you can be certain the tape was useless).

It took about three weeks, but I was eventually able to recover all the data on the array (so far as we were able to determine at the time).

NSA has 'em

They should just ask the NSA to send their copies of the emails to PlusNet.

Heck, the NSA could turn this into a side business. If they spin it right, maybe they can convince the general public that they're not spying, they're just providing a cutting-edge data backup service!

Quality Company

I've been a PlusNet user for several years now and have nothing but praise for them. Reliable service, competitive pricing and excellent support. However, I've always used Yahoo for my email...

Never press the shiny candy like red button!!!

"Hey what's this big red *Press ONLY in case of federal investigation!* button for?"

*click*

poor intern

"Hey, there were a bunch of hidden files in my home directory, so I typed rm -rf .* to get rid of them. That shouldn't take half an hour, should it?"

(I'm so nervous seeing that on my screen I'm afraid to hit the "Submit" button)

"Daddy..."

"what does this button do?"

-NOOOOOO!

Spam Filter

Spam Filter

Filter with 0% Tolerance
Are you absolutely sure?

[ok] [cancel]

I don't understand how this is possible...

...I don't recall working for PlusNet.

All eggs in one basket?

They had all their eggs in one basket? Where's the live replica machine? Where are the redundant copies? Oh wait, this is a for-profit business. Never mind.

Oh, woe...

I guess some Nigerians will have trouble making the rent this month now.

Re:Welcome to three weeks ago

This is the final update, they have NOT been able to recover the data.
If you read you post, they were calling in the recovery speciallists to try and get it back.

They failed, its game over for recovering anything.

your objection is SO last week

You do realize, I suppose, that all stories have, by virtue of describing events in the past, already happened? That's not to say we should cover the release of the 386sx processor or anything that archaic, but stories that highlight the unreliability of systems we heavily rely on do raise questions and get the ol' brain-cells moving about. It's a discussion site, you know, and even if the events discussed are more than 3 days old, sometimes interesting points are still made. That would be what we call the point of discussion sites.

Besides, it always mystifies me that people who feel that their time is wasted by duplicate or outdated stories have no problem wasting more of their time, not to mention server space and the time of all the readers, posting "this has already been covered." Do you get karmic cool points for ranting (again) about (another) dupe? What's the payoff? Does it make you happy? I'm not the most fanatically efficient person out there, but it seems petty and, well, stupid to not only dwell on, but to go to the point to complain in writing about the dupe or outdated story, which actually raises the net energy and time spent on this problem that you ostensibly found so vexing. No, I'm not complaining about you, only wondering what the hell you find so moving about the whole issue. Is it just the principle? A matter of pride? Does it bode ill for humanity? What gives?

hear hear

My response is 'if you don't like a thing, make a better one'. Can the complainers make a better slashdot? Hmm, let me see.

Do the abilities required for such a venture require

a) A willingness to try and realise there will be the odd piece of news that is repeated.

b)Being a Whining Biatch.

Well, if it's (b), then the whiners are for teh win. Somehow I have my doubts...

Re:Welcome to three weeks ago

It could be that people don't submit articles from El Reg because they can't figure out what a punter is. At any rate ...

From the article:

"At the time of making this change the engineer had two management console sessions open - one to the backup storage system and one to live storage. These both have the same interface ... the engineer made an incorrect presumption that the window he was working in was the back-up rather than the live server. Subsequently the command to reconfigure the disk pack and remove all data therein was made to the wrong server."

Anyone who has inadvertently typed an 'rm -rf' should now feel a bit better.

I do wonder whether this will cause people (and companies) to re-evaluate the growing popularity and hence reliance on web-based email. Myself, I don't go near it. Leaving the reliability concerns, and ignoring the historically bad reputation of services such as Hotmail, the spammy footers and similarly badly formatted garbage that users of web-based email end up sending everyone else, I can't fathom why it's so difficult for someone simply to log in remotely to a server that their company manages, or their own box at home. I hear you can even use those same tubes to do it.

This incident makes for a good argument, but my guess is that people will want to continue use their browsers for everything and similarly continue to rely on companies they think they know.