RIM Releases Reason for Blackberry Outage

An anonymous reader writes "According to BBC News, RIM has announced that the cause of this week's network failure for the Blackberry wireless e-mail device was an insufficiently tested software upgrade. Blackberry said in a statement that the failure was trigged by 'the introduction of a new, non-critical system routine' designed to increase the system's e-mail holding space. The network disruption comes as RIM faces a formal probe by the US financial watchdog, the Securities and Exchange Commission, over its stock options."

Re:testing departments

[Red Flayer]

Someone needs to redefine what non-critical actually is.

A non-critical upgrade is one that isn't critical that it be performed.

Increasing storage capacity (when current capacity not close to exhaustion)? Non-critical.

Fixing the shut-down system that resulted from the upgrade? Critical.

Watching the sales reps in my office apoplectically try to figure out how to get in touch with their clients? Priceless.

More details

[kbahey]

I live in Waterloo, and have friends and acquaintances who work at RIM. Talking to one of them who got called that night, he says that it started with a vendor issue, and then RIM's software did not react well to that issue.

Of course he would not elaborate more on what it is.

This Computer World article [computerworld.com] has more detail.

The outage lasted about 12 hours overnight Tuesday for BlackBerry users mainly in North America, RIM and users reported.

RIM said a fail-over system designed to stop the impact of such a problem did not work as expected, either. The company apologized to its 8 million users. RIM added that security and capacity issues were not the cause of the outage.

"RIM has determined that the incident was triggered by the introduction of a new, noncritical system routine that was designed to provide better optimization of the system's cache," RIM officials said in a statement.

"The system routine was expected to be nonimpacting with respect to the real-time operation of the BlackBerry infrastructure, but the pretesting of the system routine proved to be insufficient," the statement said.

The new system routine "produced an unexpected impact and triggered a compounding series of interaction errors between the system's operational database and cache," according to the statement. "After isolating the resulting database problem and unsuccessfully attempting to correct it, RIM began it's fail-over process to a backup system."

RIM described the backup system inadequacies this way: "Although the backup system and fail-over process had been repeatedly and successfully tested previously, the fail-over process did not fully perform to RIM's expectations in this situation and therefore caused further delay in restoring service and processing the resulting message queue."