BlackBerry suffered a system-wide network failure Tuesday night into Wednesday morning, stopping mobile email in its tracks for millions of BlackBerry users.
BlackBerry maker Research In Motion Ltd. (RIM) said little of the outage, aside from a recorded message at its customer support hotline that said: "Please be advised that we are currently experiencing service interruptions that are causing delays in sending and receiving messages. We apologize for the inconvenience and will provide updates as soon as they become available."
As of press time, RIM did not return phone calls seeking comment, and the company's Web site had no information on the outage. The cause of the problem was still unknown.
Around 10:30 a.m. EST, BlackBerry issued a statement via email that said, "A service interruption occurred Tuesday night that affected BlackBerry in North America. Email delivery was delayed or intermittent during the service interruption. Phone service on BlackBerry handsets was unaffected. Root cause is currently under review, but service for most customers was restored overnight and RIM is closely monitoring systems in order to maintain normal service levels."
Worldwide, there are roughly 8 million BlackBerry users. It was unclear yesterday morning just how many were affected by the outage, but most of the trouble appeared to be confined to North America and other portions of the Western Hemisphere. The outage prevented the sending and receiving of emails on BlackBerry devices.
One user described the outage as sudden and scary on a major BlackBerry user forum message board. The user wrote: "It was frightening. My BB [BlackBerry] just shut down earlier this evening -- no reason. Tried not to panic. I took the battery out and then couldn't get the EDGE technology. Then I tried again. I did get EDGE, however emails have been significantly delayed … hopefully this will be taken care of quickly."
Gilbert Villaroman, BlackBerry administrator for Ross Stores, who manages a BlackBerry deployment of more than 300 devices, said he noticed some message failures last night, but didn't know the true extent of the outage until Wednesday morning. Instinctively, Villaroman checked the BlackBerry server and found no problems.
Villaroman was contacted by at least one of Ross Stores' BlackBerry users who was experiencing spotty or on-and-off messaging, but he said it may still be too early for others to realize they're without service. He said that because the outage happened after business hours on Tuesday, many BlackBerry users might not have even realized there was an outage until they went to retrieve messages in the morning.
"It would be good for RIM to send out a notification or something," he said. "If there was some sort of issue in the evening, I rely on BlackBerry to receive those messages, especially if it's a critical message that needs to be disseminated to other users."
Kathryn Weldon, an analyst with Current Analysis, said around 9:30 a.m. Wednesday that the problem was "already making [her] day more difficult."
Weldon said a quick fix is necessary to prevent the BlackBerry blackout from becoming a more major problem.
"Carriers have had significant outages throughout the years with varying repercussions," she said. "In some cases, they led to the carrier having to provide stricter, more standardized SLAs so that customers could get some money back at the end of the month in the case of network problems. In this case, since it's not carrier-specific but RIM specific, there is an extra layer of complexity because data goes through RIM's NOC."
BlackBerry has often advised users to have a backup plan in case of a widespread outage, but Weldon said she doesn't feel the most recent outage will prompt too many users to give BlackBerry the boot altogether.
"I wouldn't assume a massive defection of mobile email users – if it lasts a day it will be annoying and frustrating," Weldon said. "If it lasts a week, I would say there may be more lasting effects."
Current Analysis analyst Avi Greengart agreed.
Greengart said the NOC-centric nature of RIM's solution has always been a target for its competitors, and NOC service outages are a "huge gift to competitors' marketing departments, because service outages are real. End users feel those. If a BlackBerry solution can be positioned as unreliable, it gives Microsoft, SEVEN, Motorola, Nokia, etc. a big leg-up when courting accounts who do not currently have a mobile email solution."
According to Daniel Taylor, managing director for the Mobile Enterprise Alliance, the BlackBerry outage could have two significant implications.
"The first is that RIM – as a company – is terrible at handling crisis communications," he said. "I mean, where's the information? Where's the corporate statement? Why isn't someone from RIM appearing on major business news television shows? Why isn't there a statement on the company's Web site?"
Taylor continued: "The second is that a major criticism of RIM's architecture has been demonstrated to millions of users today. An outage like this is good for marketing teams at Nokia, Motorola and Microsoft."
Taylor said that if he was managing mobile email for a company that plans to use BlackBerry for emergency communications, similar to those needed during the shootings this week at Virginia Tech or during other crisis situations, he'd be looking for something with better uptime, mirroring and crisis support.
"Either way, I'd look long and hard at solutions that IT departments can manage without depending on a third-party NOC," he said. "It's one thing if BES [BlackBerry Enterprise Server] goes down in a corporate data center. IT departments can plan for situations like that. But with the way RIM manages mobile email, there's absolutely nothing an IT manager can do when RIM's NOC goes down. It's embarrassing."