MS Says All Sidekick Data Recovered, But Damage Done 279
nandemoari writes "T-Mobile is taking a huge financial hit in the fallout over the Sidekick data loss. But Microsoft, which bears at least part of the responsibility for the mistake, is paying the price with its reputation. As reported earlier this week, the phone network had to admit that some users' data had been permanently lost due to a problem with a server run by Microsoft-owned company Danger. The handset works by storing data such as contacts and appointments on a remote computer rather than on the phone itself. BBC news reports today that Microsoft has in fact recovered all data, but a minority are still affected (out of 1 million subscribers). Amidst this, Microsoft appears not to have suffered any financial damage. However, it seems certain that its relationship with T-Mobile will have taken a major knock. The software giant is also the target of some very bad publicity as critics question how on earth it failed to put in place adequate back-ups of the data. That could seriously damage the potential success of the firm's other 'cloud computing' plans, such as web-only editions of Office."
Don't blame t-mobile for Danger's failure (Score:5, Insightful)
Backups are unimportant; restore is everything. (Score:5, Insightful)
Worth repeating every time. Nobody cares if you back up your data. Take a blank server; take whatever it is that you store offsite. If you can turn the blank server into your production system then you are fine. If you can't then your strategy is failing. If you never try it then you are an amateur.
This incompetence is something far beyond serious for MS. T-mobile is a much bigger customer than almost anyone short of vodafone can ever hope to be. MS have been moving strategically into hosting servers such as exchange for many customers. If you're a CEO you should be calling your CIO in and asking him when he plans to be free of MS services. If you are a CIO you want to be able to answer "there's nothing business critical relying on MS services" by the time that meeting comes.
Re:This is why you have press people (Score:4, Insightful)
Re:Huh? (Score:5, Insightful)
What's up with all the editorializing in the summary? Danger was bought by MS only 18 months ago. What the heck has this got to with Office and cloud computing except wishful thinking by the submitter?
So... in a year and a half they shouldn't have toured their new acquisition and checked for basic things like:
1) Updated server software
2) Firewalls
3) Backups
And other "yer an idjit if you don't do this" kinda stuff?
For *any* kind of hosted service, having backups measures just slightly below "is it turned on" in terms of importance. And for a year and a half, NONE WERE DONE? Further, they did a major update to a SAN and didn't backup first?
This isn't about bashing Microsoft - highly successful businesses have had to close shop forever due to glaring, horrid oversights like this. This is gross incompetence.
Re:Huh? (Score:3, Insightful)
Er... because it is a form of cloud computing which failed? When a failure like this occurs, it rightfully raises doubt as to the reliability of other cloud computing services, one of which happens to involve office.
Critics only *NOW* questioning MS's competence?!?! (Score:4, Insightful)
Years of BSODS.
Years of viruses.
Years of trojans.
Yet THIS "damages Microsoft's reputation"?!?!?!
Re:Huh? (Score:5, Insightful)
Well, it's a bit of a non sequitur, to be sure. But the whole incident spells out in stark detail the dangers of "cloud computing", or as us folks who actually have worked with computers for more than than ten minutes call it; the client-server model. When explained as what it really is, it's a matter of ensuring adequate and timely backups. When described in some pathetic marketing term, it sounds like some magical new way of computing, no longer constrained by those old-fashioned good practices.
Quite frankly, I would never ever ever put any mission critical data or apps on a system that I couldn't back end the data on my own out of. If I can't move my data out of the app, then my data never gets there in the first place.
Re:Don't blame t-mobile for Danger's failure (Score:3, Insightful)
But Johnny SidekickUser can't contract directly with Danger, he has to deal with T-Mobile. T-Mobile has some responsibility for making sure the service they're reselling operates as advertised. This shouldn't be a "best-effort" service.
Re:Don't blame t-mobile for Danger's failure (Score:5, Insightful)
T-Mobile and Danger were partners long before Microsoft ate Danger up. It's not like Microsoft had a history of failed backups and horrible transitions.
Re:Huh? (Score:5, Insightful)
You really don't see the connection?
Yesterday, you put all your cell phone contacts and calendar data up in the "cloud".
Today, your data is lost.
Tomorrow, the same companies responsible for losing your cell phone data now want to take over all your Office documents.
Well, since this is /., you take your car in for a routine oil change. The mechanic botches the job.
Are you going to go back to the same mechanic for a transmission rebuild?
Re:Cloud computer (Score:5, Insightful)
Perhaps for people who don't care about their data... Privacy, security, accountability and reliability cannot be ensured by a third party. I'll keep my data in-house thank you.
One thing and another (Score:5, Insightful)
What we see here is a small device storing it's data remotely and I wonder why.
Considering how cheap a couple of GB of memory are and how precious wireless bandwidth is this can mean only one thing, having and thus exploiting that data is worth more than the cost of the bandwidth.
Re:Backups are unimportant; restore is everything. (Score:5, Insightful)
No, he's NOT overstating his point. Unless your data is a bunch of flat text files or Word documents or whatever the restore is a critically difficult process.
Enterprise data like this often has never been in a flat or "dead" state since the original implementation. Complex applications frequently have delicate interactions between the live application and the contents of the database at any particular moment. Having a bunch of database tables on a tape somewhere doesn't do you much good if the application can't actually start from the state contained on the tapes, and it's a two-week manual process to clean up the issues.
If you can afford a "slow and sketchy" restore process, or your application is just not that complicated, then by all means, don't test your restore, and don't create a department with responsibility for backups and nothing else. It's still amateur work.
Microsoft? No. (Score:5, Insightful)
I don't believe that customers care if your services providers have problems. They have an agreement with you, not your providers.
Re:Not likely (Score:4, Insightful)
Except that people make decisions and don't really care if something is just "affiliated".
Microsoft and Google bid for the "cloud computing" "office" contract at some company. Do you really think Google isn't going to mention, with a bunch of references, this screw up?
With quotes from press releases like:
in big bold blocks.
Re:Huh? (Score:5, Insightful)
They have not had this problem in their first 8 years. Then, 18 months after Microsoft acquires them, they have a critical failure. You think that's all coincidence?
I suppose it's possible for one company to buy another and leave the company alone, but Microsoft certainly didn't do this. They moved most of the developers to Project Pink (and most of them have left MS entirely by now). I think it's pretty clear that the new MS was responsible. They managed the company. The data was stored at Microsoft's data centers.
Meanwhile, Microsoft is trying to sell people on the idea that their data should be hosted at Microsoft data centers. Am I not supposed to be skeptical about this now?
Re:Trusting in Microsoft's servers? Hah! (Score:2, Insightful)
Last I checked, Hotmail still ran on FreeBSD
Which was what? 8 years ago?
Re:Trusting in Microsoft's servers? Hah! (Score:2, Insightful)
Re:said it before and will say it again (Score:5, Insightful)
Man, what fantasyland are your utilities located in? I wanna move there! In my experience, utilities *are* "giant rapacious corporation uncaring of human concerns".
Re:Backups are unimportant; restore is everything. (Score:3, Insightful)
Your admin team or hosting company should be able to tell you what is involved to get from your backup to a fully functioning production system (a truly well thought-out backup scheme will have a step-by-step recovery checklist), and they should be able to provide a worst-case data loss estimate based on your backup scheme.
This isn't a failure of "cloud computing" or any other buzzword-of-the-day but rather a failure of basic competence in information management: an unforeseen event coupled with broken, inadequate or nonexistent backups lead to a catastrophic data loss that should never have happened.
Re:Cloud computer (Score:3, Insightful)
Perhaps for people who don't care about their data... Privacy, security, accountability and reliability cannot be ensured by a third party. I'll keep my data in-house thank you.
Dude, organizations use third party data centers (or data centers that they physically own but are managed by a 3rd party) all the time w/o a glitch. Unless you are a software giant (like ebay or amazon) that can build your own data center, or are a minor/midsize operation (or are just a guy with a home computer), you will inevitably have a large part of your stuff either running on someone else's infrastructure or having it operate on someone else's watch.
It is done all the time, by many, for years now. Almost no glitches that can be directly attributed by the fact that a 3rd party was involved. In order to have a meaningful opinion on IT operations, you need to differentiate problems that occur because things are not run by you (things that are inevitable in computing) vs problems that occur because of lack of safeguards or wrong procedures (which can and will happen under your watch or someone else's.)
Re:Huh? (Score:1, Insightful)
i dunno.... this data loss and subsequent PR fallout is one way.. it's nearly all aimed smack at tmobile..
microsoft will come out of this unscathed, and PR for them will shift to back to the feel-good fluff pieces surrounding the release of win7.
Re:Cloud computer (Score:4, Insightful)
If you can setup offline synchronization and data encryption, there is no reason to not use cloud computing.
All a local backup will give me is reliability.
If I can't encrypt my data on their servers I don't really have privacy or security.
The touch of Microsoft. (Score:1, Insightful)
Microsoft sure seams to have a wicked spell of utter incompetence cast upon them. Anything they tuch turns to crap.
Nobody in their right mind will put anything even remotely important in a cloud ran by Microsoft.
Re:Don't blame t-mobile for Danger's failure (Score:2, Insightful)
Bullshit. If T-Mobile is supplying you with this phone and this service (i.e., you pay T-Mobile every month for the priviledge of using this service, do you not?), then why shouldn't they also be responsible for failures and outages? As a customer, I shouldn't need to care what they use as a back-end solution, and I certainly wouldn't accept "it's somebody else's screwup" as an excuse if something went wrong. The fact that the failure happened in some back-end service provider's network and not in the T-Mobile network itself doesn't matter. Presumably it was T-Mobil that chose this back end solution in the first place, and they (presumably) did the due diligence to make sure that their selection was up to their standards. If they fucked up their due diligence, or didn't do due diligence at all, then they are as much responsible for the failure as MS/Danger is. If they had done a better job picking their back-end provider, or at specifying the requirements of the system (i.e. specifying that a robust backup solution be used (and tested!) on all customer data), then this wouldn't have happened. In that regard, T-Mobile definitely could have prevented this issue.
Re:Huh? (Score:3, Insightful)
If you really want a conspiracy theory, toss in that another factor Microsoft considered was that Danger uses Unix servers, Oracle RAC, Java apps, and Hitachi SAN software. No sign of any significant Windows technology. So, they purposefully destroy the data. That not only hits T-Mobile, per your proposed conspiracy theory, but also hits Oracle and Unix and Java, and it shakes confidence in the whose Cloud idea.
Google and Amazon are ahead of MS right now in Cloud stuff, so if Microsoft can throw a delay into that sector, it hurts Google and Amazon more than it hurts Microsoft. By the time people get over the fright and are ready to jump back in, Microsoft will have its cloud offering out, AND they can point out that all major cloud failures have been on Unix or Linux, and with non-MS databases and app servers--and argue that if you want to get back into the cloud, go with MS on Windows servers, MSSQL databases, and .NET apps.
The problem with proposing fun conspiracy theories like you and I are doing, though, is that the real conspiracy theorists are already there. I've already seen several of the tin-foil hat crowd saying it was on purpose.
Re:Huh? (Score:3, Insightful)
I seem to remember Microsoft buying Hotmail back in the 90s, and royally screwing up its operations in much less than 18 months. They tried to move to Windows servers very quickly, and it was a disaster, and they were forced to go back to their FreeBSD infrastructure for a while.
Maybe something similar happened here.
Re:Huh? (Score:4, Insightful)
I bet that most companies, when buying other companies, don't check a lot of basic things before buying them. As for "mickey-mouse outfit", in my experience, most corporations fit that definition well. The people running them really aren't that smart, and make all kinds of dumb mistakes.
Re:Hire more Americans (Score:3, Insightful)
Microsoft has been employing and run by Americans since it started, and they've produced nothing but buggy crap.
Meanwhile, the Mars rovers have been a tremendous success, built by American engineers using American-made software I believe (I'm pretty sure they use vxWorks). This is the epitome of software reliability I think.
I don't think nationality has much to do with this one.
Wrong wrong wrong! (Score:5, Insightful)
"But Microsoft, which bears at least part of the responsibility for the mistake, is paying the price with its reputation."
Microsoft bears ALL THE RESPONSIBILITY FOR THE MISTAKE!
They own Danger and they run the data center that stores the data!
It was their fault 100%.
Re:As Rob Pegoraro of The WaPo points out (Score:3, Insightful)
Re:Microsoft's reputation (Score:4, Insightful)
The "Pink" project was a Microsoft creation based on their technology, NOT a Danger product. It was the brainchild of Microsoft's Roz Ho. Microsoft may have bought a terribly run company, but that happens all the time in the real world. After a year and a half under the leadership of Microsoft, problems can no longer be blamed on the previous company's leadership. Most of those people don't even work there anymore. It's all on Microsoft's head.
The problem is not that the Danger division is run like a separate company. The problem is that every little division of Microsoft is run like a separate company. That's their biggest flaw, and they really need to get an effective leader (as in replace Steve Ballmer) who isn't afraid to fire anyone who is more concerned about protecting his/her own empire than with the good of the company. That pretty much means replacing large swaths of the management hierarchy. That's the only thing that will save Microsoft from eventual total failure. That or a huge government bailout in twenty years for being "too big to fail".
Re:Huh? (Score:3, Insightful)
You really don't see the connection?
Yesterday, you put all your cell phone contacts and calendar data up in the "cloud".
Today, your data is lost.
Tomorrow, the same companies responsible for losing your cell phone data now want to take over all your Office documents.
The phrasing of this sounds chilling until one realizes that the main point here is that you still want to keep your own local copy. The T-Mobile phones should have done that. You should do that when creating documents on-line.
This is such a silly reason to vilify 'the cloud'.
From where I sit, the problem started when some guy wearing a tie said "and the phones use the server exclusively to house the data!" Dumb. The 'cloud' shouldn't even be part of this discussion.
Re:Cloud computer (Score:3, Insightful)
Looms = non cloud computing.
That is a bit of a stretch.
I think ppl can use OSS & Linux and use VPN's like Open VPN and
say that whatever crap comes out of M$ is not worth using.
There are already DNS issues, and other Internet Infrastructure issues
and relying on a system that is burdened with massive spam, torrent,
and email data is a recipe for disaster.
Do not talk to me about Telcos taking care of that.
Their track records are WELL known.
http://www.tispa.org/node/14 [tispa.org]
$200 billion in tax payer money pissed away on TOTAL lies.
The cloud is a spider web built on greed and broken
promises and lies just like our banking and a good
portion of our government.
Re:Don't blame t-mobile for Danger's failure (Score:3, Insightful)
Could very well be, but then what of the following, taken from David Brooks' writeup describing the HotMail conversion (emphasis added)
Source: http://www.securityoffice.net/mssecrets/hotmail.html [securityoffice.net]
The word, however, wasn't the point in referencing the post. The intended take-away was Microsoft's prior dodgy track record when doing infrastructure swap-outs on some of their newly-acquired products, swap-outs driven by factors which clearly included strong doses of NotInventedHere.
Given the background and consequence similarities between this and past episodes like HotMail, its a valid line of speculation with some historical precedent. Not that it will ever be anything more than speculation, of course.
Re:Cloud computer (Score:2, Insightful)
And vista works so dreamily....
Riiiiiight....
Windows 7 is Vista SP2 and it is semi-decent.
The best bet for true corporate computing is blade servers running Linux and using virtualization
of a M$ OS so they do not have to shell out for a bunch of licenses.
M$ hates that by the way, and is trying to much up VMware for that very reason.
Sounds like JournalSpace (Score:3, Insightful)
> "The outage was caused by a system failure that created data loss in the core database and the back up,"
> [Microsoft Corporate Vice President Roz Ho] wrote in an open letter to customers.
It sounds like their "backup" was a replica on another connected server.
No actual offline backups at all.
When JournalSpace was destroyed, one SlashDot thread was "Why Mirroring Is Not a Backup Solution".
My favorite comment was by JoelKatz:
>> The whole point of a backup is that it is *stable*. Neither copy is stable, so there is no ... if the active copy of the data is corrupted, there is no backup.
>> "backup on the hardware level". There are two active systems.
>>
>> If you cannot restore an accidentally-deleted file from it, it's not a backup.
>>
Re:Cloud computer (Score:4, Insightful)
I think you missed the point:
- With software you own, you can ignore Microsoft's mistakes (Office2007, Vista) and continue using their older products (Office2003 or 97, XP).
- With software you rent off the internet (cloud), the bad ideas are shoved upon you whether you like them or not.
Re:One thing and another (Score:4, Insightful)
Correction: locking the data into a closed service from which you cannot easily retrieve it, making you permanently dependent on their service for your contact information is worth more than the cost of the bandwidth. It's the Facebook of the cell phone world.
Reputation? (Score:3, Insightful)
"But Microsoft, which bears at least part of the responsibility for the mistake, is paying the price with its reputation"
Just out of curiosity, what reputation might that be? :-)