Slashdot Log In
RIM Releases Reason for Blackberry Outage
Posted by
Zonk
on Fri Apr 20, 2007 11:24 AM
from the isn't-testing-a-requirement dept.
from the isn't-testing-a-requirement dept.
An anonymous reader writes "According to BBC News, RIM has announced that the cause of this week's network failure for the Blackberry wireless e-mail device was an insufficiently tested software upgrade. Blackberry said in a statement that the failure was trigged by 'the introduction of a new, non-critical system routine' designed to increase the system's e-mail holding space. The network disruption comes as RIM faces a formal probe by the US financial watchdog, the Securities and Exchange Commission, over its stock options."
Related Stories
[+]
IT: RIM Offers BlackBerry Service Without the BlackBerry 80 comments
TheCybernator writes "RIM has announced that they're essentially planning to offer BlackBerry service ... without the BlackBerry. The company plans an app suite that will turn its push e-mail technology into a platform for Windows Mobile 6 devices. Less than a week after a network outage crippled BlackBerry users across North America, Research In Motion announced an application pack for Windows Mobile 6 devices that Canadian software developers said will intensify the competition for push e-mail. The firm has said that the BlackBerry Application suite will appear as an icon on the screen of the Mobile Windows device and load BlackBerry applications such as e-mail, phone, calendar, address book, tasks, memos, browser, and instant messaging. RIM said users will easily be able toggle between the two platforms, one of which would have a BlackBerry-style interface."
This discussion has been archived.
No new comments can be posted.
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
Full
Abbreviated
Hidden
Loading... please wait.
perhaps (Score:5, Interesting)
What really happened... (Score:5, Funny)
Non-critical? (Score:5, Funny)
bkd
Buying time (Score:5, Funny)
Ah ha! (Score:5, Funny)
Is this really so bad? (Score:4, Insightful)
Yeah, they've got areas to tighten up their QA and patch processes, but on the whole they got it all back up and running faster than most enterprises get their email functioning after a worm.
Re: (Score:3, Funny)
Yes it is. They've put themselves in a critical... (Score:5, Insightful)
Several hours of email downtime is "OKish" if you are talking about a medium sized company that only has a handful of servers and a few IT guys. This is not the same at all.
Prior to this, I never realized that the RIM system was THIS centralized. It's kind of concerning really. And I don't quite understand why so many US gov't users are allowed to route their email through a NOC in Canada (disclosure: I'm Canadian).
Parent
RIM's biggest failure (Score:5, Interesting)
Re: (Score:3, Funny)
Pop quiz! (Score:3, Insightful)
A) The fact one piece of software took down their environment.
B) Their failover plan didn't work.
C) All of the above.
D) None of the above.
Personally, I vote for "B". Face it, s**h happens. But when you plan for s**t happening and the plan doesn't work, that's a VERY bad thing.
Testing of Complex Systems (Score:4, Insightful)
And a bunch of suits will want the heads of the technicians responsible.
I feel for them, I really do.
A few years ago I put in a minor maintenance change that made headlines for my employer.
This is a natural result of the budgetary constraints we have to live with in the real world. Testing and certification is expensive, and the more complex the environment, the more expensive it gets. It is difficult to justify a full blown certification test for minor, routine maintenance, unless you are talking about health and safety systems. So a worst-case event occurred, RIM suffers some corporate embarrassment, some low-level techs will get yelled at, and possibly fired, and a bunch of people had to suffer crackberry withdrawal.
Nobody died. No planes crashed. No reactors melted down.
RIM will work up some new and improved testing standards, and tighten the screws on system maintenance so much that productivity will suffer, they may even spend a bunch of money on the equipment needed to do full-production-parallel certification testing. And then in a year or so cut the budget to upgrade the certification environment as 'needless expense', and come up with work-arounds to reduce the time it takes to get trivial changes and bugfixes rolled out.
I wish them luck. Especially to the poor sods who did the implementation.
At least when I did my 'headline-making-minor-maintenance' it only made the local papers for a couple of days.
the REAL reason.... (Score:4, Funny)
:
>The network disruption comes as RIM faces a formal probe by the US financial watchdog, the Securities and Exchange Commission, over its stock options.
Hmmm... so when they wiped the incriminating e-mails from the system (which would certainly create more space), they took the rest of the system down (which prevented anyone else from grabbing copies).
I'm reading WAY too many conspiracy novels these days
(Not that I think this actually happened - but it makes for a great plotline).
More details (Score:4, Informative)
Of course he would not elaborate more on what it is.
This Computer World article [computerworld.com] has more detail.
PR to IT translation results (Score:3, Funny)
Re:I'd hate to be their QA manager right now! (Score:5, Insightful)
It's quite likely the development group listed this as a risk, with a good backout plan, and upper management simply didn't want to pay for the cost of having a quick backout.
If that's the case, you can be pretty sure upper management WON'T take the blame.
Parent
Re:I'd hate to be their QA manager right now! (Score:5, Insightful)
Parent
Re:I'd hate to be their QA manager right now! (Score:5, Insightful)
Blasphemer!
Parent
Re:I'd hate to be their QA manager right now! (Score:5, Insightful)
Parent
Re:I'd hate to be their QA manager right now! (Score:4, Insightful)
Parent
Re:I'd hate to be their QA manager right now! (Score:5, Insightful)
Whatever it is, the production problems are due to bad process, which is what management is supposed to control. They are not even responsible for coming up with the technicalities of the process, they are responsible for making sure that there is a sufficient process (sufficient in terms that it is agreed by all parties, DEVs, QAs, BAs, client that it is good enough.) They are responsible to make sure that the process is followed.
Over a year ago now in Toronto, ON, Canada, the Royal Bank of Canada had a similar problem, but of course with a bank it is much more dangerous it is lots of money of lots of people. Heads rolled at the management level only.
Parent
Re:I'd hate to be their QA manager right now! (Score:5, Insightful)
Because that's not how change should happen in large/business critical applications.
What should happen is that the update is thoroughly tested, a change control request is raised and at the next change control meeting the change request is discussed.
The change request should include at the very least a benefit analysis (what's the benefit in making this change), risk analysis (what could happen if it goes wrong) and a rollback plan (what we do if it goes wrong). None of these should necessarily be vastly complicated - but if the risk analysis is "our entire network falls apart horribly" and the rollback plan is "er... we haven't got one. Suppose we'll have to go back to backups. We have tested those, haven't we?" then the change request should be denied.
As much as anything else, this process forces the person who's going to be making the change to think about what they're going to be doing in a clear way and make sure they've got a plan B. It also serves as a means to notify the management that a change is going to be taking place, and that a risk is attached to it.
And if a change is made but hasn't been approved through that process, then it's a disciplinary issue.
Of course, it's entirely possible that such a process was in place and someone did put a change through without approval. In which case, I don't envy their next job interview.... "Why did you leave your last job?"
Parent
Re:I'd hate to be their QA manager right now! (Score:4, Insightful)
How many people here have checked in buggy code that neither management nor QA knew was buggy? (crickets)
How many people here have been on projects where management shoved the code out the door despite major bugs that they knew about? (thunderous applause)
How many people here have tried to get time on The Schedule to do something The Right Way, only to be told by management to do it half-assed, because that's all there's time/resources for? (applause, hooting)
There you go.
Parent
Re: (Score:3, Funny)
~kicks guy into a bottomless pit~
Re:testing departments (Score:5, Informative)
Increasing storage capacity (when current capacity not close to exhaustion)? Non-critical.
Fixing the shut-down system that resulted from the upgrade? Critical.
Watching the sales reps in my office apoplectically try to figure out how to get in touch with their clients? Priceless.
Parent