Nextel's soap opera (act 3, scene 1)

Rob Rosenberger, Vmyths co-founder
Friday, 15 December 2000 ONCE IS NOT enough for Nextel. We can learn things the easy way by watching them learn things the hard way multiple times.

An employee read a company-wide email and hit "Reply to All" by accident. Then he tried to "recall" his email — and it generated tens of millions of failure messages!

Nextel's latest woes began with a company-wide email. It announced an employee discount for new-car buyers — but one worker at a branch in California didn't care for the limited selection. (Californians have better taste in automobiles, you know.) He clicked the reply button and said something to the effect of "let me know when you cut a deal with Porche or BMW." Whoops! He clicked "Reply to All" by accident. His whiny response went to everyone at Nextel. Let's remember something important here. If an employee can do something like this, then a virus can do it, too. Nextel got swamped by ILoveYou for exactly this reason. You'd think their email administrator would've learned a lesson, but you'd be wrong. The average Nextel employee can still fire off company-wide emails on a whim. Judging by the company's size, I'd say the reply went to roughly 15,000 employees. All because of an errant mouseclick. Wait, it gets better. Mr. Bad Eye/Hand Coordination soon realized his mistake. (I'll bet a soda he realized it when he got his own reply.) So he tried to "recall" the email he sent. "Ouch." Did I mention Nextel's email servers use Microsoft Exchange?

"Email aware" viruses? Big deal. It's only a matter of time before we see a "recall aware" virus...

When an Exchange server attempts a "recall," it may encounter a recipient who already deleted it. Exchange sends a "recall failure" notice to all of the recipients for each recipient who did so. If this guy deleted his own reply before recalling it, then he alone generated 15,000 failure notices. My source at Nextel believes thousands of employees deleted his message before the recall went out. If we multiply just 3,000 cubicle dwellers by 15,000 recipients ... um, carry the one ... we're talking in the range of 45 million failure notices! Nextel has people all over the country; each branch office has a finite amount of network bandwidth. Given the company's size, we can assume they run ten or more dedicated email servers. Those babies consumed most (if not all) of the available network bandwidth while trying to deliver failure notices. It's like pushing a bowling ball through a drinking straw. The "disaster clock" started when the employee tried to recall his message. Two minutes can pass before an automated distress signal goes out to a pager. Give the administrator 2-5 minutes to hightail it over to a climate-controlled network room. Let's say it takes another 4-5 minutes to log in and determine the root cause of the problem. Give him one minute of stressful decision making, followed by 5-10 minutes of keyboard banging. Nextel's email servers probably generated "recall failure" notices for 14-23 minutes before their administrator gained the upper hand. "Ouch." I'll bet some of the servers crashed due to an overload, and I'll bet some of the branch offices lost network connectivity for a few hours. (You can't max out bandwidth at a large firm without some repercussions.) It'd take another few hours to fully clean up the resulting mess.

Why didn't Nextel learn this lesson when Melissa struck in 1999? Why didn't they learn it when ILoveYou swamped them earlier this year?

All this, just because one employee clicked "Reply to All" by accident.
THIS EVENT DIDN'T involve a virus, but it can teach us a valuable lesson about "email aware" viruses. Look at Melissa, for example. It didn't really attack users — rather, it attacked the underlying email infrastructure. A Nextel employee did the same thing with an errant mouseclick. It's only a matter of time before we see a "recall aware" virus, you know. Email administrators shouldn't wait for it to come along before they do something about it. We've learned another valuable lesson at Nextel's expense. Let's hope Nextel finally learned it, too.