Truth About Computer Security Hysteria

GRC.com's worst enemy is ... Steve Gibson?!?

• Attacked: 15 times (total)
  • Abandoned his duty during an attack: 01 times
  • Considered asking his ISP to assist an attacker: 01 times
• SELF-inflicted: 13 times
• Service provider attack: 06 times
• Media-induced popularity DDoS: 02 times
• Normal wear & tear: 18 times (total)
  • Routing problem: 04 times
  • Hardware problem: 06 times
  • OS problem: 02 times
  • Software problem: 03 times
  • Programming error: 03 times
• Accident: 02 times
• Unknown reason: 01 times

• 28 Apr 04: SELF-inflicted. "Since I didn't really have anything else going on today, I decided that it would be a good time to completely re-wire our networking with Verio. So that 30-minute network outage we just suffered was the re-routing of our traffic, as planned for months..."

  "Since I didn't really have any­thing else going on to­day, I de­cided that it would be a good time to com­pletely re-wire our net­working with Verio. So that 30-minute net­work outage we just suffered..."

• 2 Apr 04: planned service provider attack. "The Verio guy said that, given readiness at the new location, the total connectivity outage time should be on the order of 15 minutes at the most ... but I'm betting that it will be much shorter than that, on the order of a minute or two..."
• 5 Jan 04: hardware problem. "Last night the second Linksys hub we've had fail, failed. This resulted in a chunk of our network being cut off from access to the external Internet..."

• 29 Nov 03: unidentified routing problem. "I rebooted the massively stateful equipment which watches and manages our border ... and my cable modem is again able to see GRC..."
• 25 Nov 03: hardware problem. "At about 2 AM PST this morning, until I discovered it fifteen minutes ago, GRC was completely off the air. I was initially puzzled until I saw that all of the lights on our central network hub were dark. A quick swap of the dead hub with a spare ... brought us right back online..."
• 28 Sep 03: unidentified accident. "I don't know for sure WHAT that was. (But it wasn't a DoS attack.) Our bandwidth went quite dead for about five hours. It may have been scheduled Verio maintenance, or I may have pissed off some upstream Verio router due to all of the resets I was causing on our interfaces..."
• 24 Sep 03: planned SELF-inflicted. "September 25th, you may experience repeated loss of persistent connections to our news server... I will be performing the first extensive reconfiguration of our network border systems..."
• 14 Sep 03: unidentified accident. "Whoops, Sorry about that folks..."
• 12 Sep 03: software problem. "News Server Log Filled Up The Drive!!"

  "I don't know for sure WHAT that was. (But it wasn't a DoS attack.) Our band­width went quite dead for about five hours..."

• 10 Sep 03: OS problem. "I've been working on tracking down the annoying server kernel leak which requires me to reboot the main grc.com server every three days..."
• 12 Aug 03: OS problem. "Everything collapsed again. So we're now running at 16x without any trouble, and our two T1's are totally saturated and running at capacity."
• 30 Jul 03: planned SELF-inflicted. "The next piece of work I have planned for this evening will take both our main grc.com web server and the R&D nanoprobe server offline for some time — hopefully less than an hour..."
• 27 Jun 03: hardware problem. "The experimental/developmental nanoprobe.grc.com server ... just crashed and rebooted itself. It has been running in the LAST of the flaky little desktop boxes that have been dying one-by-one, presumably due to the motherboard electrolytic capacitor problem..."
• 13 Jun 03: attacked. "Just an FYI in case, or when, you're able to receive this. Our bandwidth has been under a rather sizeable attack for the past four or five hours..."
• 5 Jun 03: programming error. "Yesterday I tweaked the server's code a bit and briefly broke the system's article cancel validation system..."
• 29 May 03: service provider attack. "I'm sorry not to have given everyone advance notice of tonight's extended network outage. I had completely forgotten about it until we dropped completely off the Net..."
• 26 May 03: hardware problem. "Yesterday afternoon the news server crashed as the Motherboard's capacitors began causing the same problems we've seen with the other systems I was using as "scratch" machines. I started watching it more closely, and today when it began really having trouble I took it offline..."

  "I'm sorry not to have given every­one ad­vance notice of to­night's ex­tended net­work out­age. I had com­pletely for­got­ten about it un­til we dropped com­pletely off the Net..."

• 12 Mar 03: service provider attack. "I have verified that the source of the connection trouble people are experiencing is Verio's DNS servers..."
• 31 Jan 03: planned SELF-inflicted. "Since troubles are continuing after the news server returned from its near-death experience, and since I think there is more that I can do, I am going to take it down for a few hours Saturday evening and see about re-rebuilding the various databases..."
• 26 Jan 03: hardware problem. "it has apparently been crashing and rebooting itself quietly. I know that I am quite guilty of neglecting it, since it virtually never complains or needs any attention — for years now... there *WAS* some extensive corruption of the news server's database files..."

• 17 Nov 02: hardware problem. "it wasn't any sort of attack. Apparently something's still being a bit wonky with one of our UPS systems..."
• 2 Nov 02: abandoned his duty during an attack. "We have been down until now because the attack hit just as I was leaving for dinner with friends (and a guy has to eat! :) and then because I have been learning a LOT about the performance of my system when under fire..."
• 1 Nov 02: SELF-inflicted. "I *may* have figured out why the new server was spewing packets from a private 10.* network, and what's been causing the disconnections ... that reset at about 2:00 PST was me..."
• 1 Nov 02: attacked. "If you are able to read this ... that's good news ... and it means that I am gradually managing to get things back on line..."
• 14 Oct 02: attacked. "it took down Verio's routers several stages upstream — it was BIG..."
• 11 Oct 02: programming error. "I was tweaking some code in the news server again, and introduced a subtle bug in the article processor..."

  "The attack hit just as I was leaving for dinner with friends (and a guy has to eat! :)"

• 4 Oct 02: planned SELF-inflicted. "GRC.COM will experience a (hopefully) brief whole-network outage of my own making later tonight..."
• 2 Oct 02: planned service provider attack. "Verio has planned an outage on the router to which you connect, and you may experience some interruption of service while we work on the router. The details are below: Outage Window: From 10/04/2002 at 0900 to 10/04/2002 at 1200..."
• 4 Sep 02: software problem. "While I was updating some scripts on the news server just now, there was a brief period where someone connecting during a small window of time would have snagged a copy of the scripts which wouldn't properly post. It appears that about fifteen to twenty posts were lost on their way to the news database..."
• 3 Sep 02: programming error. "I was getting cocky and not checking my Perl syntax before putting changes on line ... thus some news server code I have been writing was keeping people from connecting..."
• 27 Aug 02: attacked. "This guy's new Bot fleet is growing in strength. That hour and forty minute outage/slowdown just now was the result of a new tool that's been under development for some time..."
• 8 Aug 02: attacked. "The DoS (random Syn flood) attack is the same one we've been 'tested' with four times in the past 24 hours. Each of those previous times took us down for less than ten minutes, so it wasn't worth bothering Verio. But about three hours ago, at 9:30 PST, the attack started and this time didn't let up..."

  "I was getting cocky and not checking my Perl syntax before putting changes on line..."

• 26 Jun 02: SELF-inflicted. "A quick comment about our brief absence this evening: The whole vicinity lost power earlier tonight. Our beefy UPS's kept things running for quite a while ... and I could have fired up the generator to keep us online beyond that, but everything needed a good rebooting anyway, since nothing has had a moment's rest for years. So I finally performed a graceful shutdown to await the power's return."
• 17 Jun 02: attacked. "A quick note about this afternoon's network outage: It was a DDoS attack, but one which our existing defenses SHOULD have blocked so that we would never have even felt it... Some changes I made to our filters several weeks ago (to block another new type of attack) overrode the rules I had in place..."
• 6 May 02: routing problem. "We're back after a three hour (approx) network outage... A power-cycle/reboot of the router immediately restored us to full normal operation. So, near as I can tell ... our Cisco router just "hung" for no apparent reason. Unfortunately ... I'm told that such behavior is not all that uncommon for such routers..."
• 26 Feb 02: SELF-inflicted. "I am currently pushing out a mailing to our 665,000+ mailing list subscribers. This is the first mailing I've done since the "Wicked" attack report from early last July. As always happens, the mass mailing is driving a LOT of traffic to the site ... and nearly saturating our bandwidth..."
• 22 Feb 02: possibly attacked. "About ten minutes ago, we suffered a 100% outage for about two minutes. This virtually never happens, so it might well have been a 'ranging shot' in preparation for an all out attack..."

  The whole vicinity lost power earlier to­night... I could have fired up the gener­ator to keep us on­line beyond that, but every­thing needed a good re­booting anyway...

• 13 Jan 02: considered asking his ISP to assist an attacker. "This afternoon I asked Verio to check on the router to see whether the DDoS attack was still waging on the other side of it. If so, I was going to have them briefly drop our defenses — putting us back into immediate Denial of Service — so I could use the cool new packet capture utility (see my previous posting here) to get a comprehensive snapshot of the attacking server base..."
• 11 Jan 02: attacked. "We were under a new sort of comprehensive TCP flooding attack that I haven't experienced before, but which I've been hearing a lot about recently. I waited until 4:00 AM before bothering Verio since it was the middle of the night and I hoped that the attack would end shortly. A short and similar feeling attack had hit us the night before, but it was over before I had the chance to capture, examine and characterize it. We are still under attack at the moment..."
• 8 Jan 02: media-induced popularity DDoS. "At about 9:10 Pacific time the grc.com web site traffic jumped significantly... We were just — at exactly the moment the traffic hit, featured in the EMAZING newsletter..."

• 20 Aug 01: attacked. "That three-hour outage just now was an 18-machine UDP flood, almost certainly launched from security compromised Windows PC's..."
• 30 Jun 01: routing problem. "I finally tracked the problem ... to our firewall. I don't yet know exactly what's going on, but I'll figure it out..."
• 22 Jun 01: routing problem. "I have been experiencing some odd news server connection difficulties today..."
• 18 Jun 01: unknown reason. "The Cecil-ID secure cancellation system is back online."

  "I asked Verio to check on the router to see whether the DDoS attack was still waging on the other side of it. If so, I was going to have them briefly drop our de­fenses — putting us back into im­me­diate De­nial of Ser­vice — so I could use the cool new packet cap­ture utility..."

• 16 Jun 01: SELF-inflicted. "I rebuilt the UNIX Kernel [for the news server]..."
• 8 Jun 01: SELF-inflicted. "With the news server cruising along and our network connection blessedly quiet today ... I have decided to bring the web interface back online..."
• 1 Jun 01: SELF-inflicted. "the news server being down was deliberate..."
• 31 May 01: media-induced popularity DDoS. "I have NEVER — and I mean NEVER NEVER — seen the server and our T1's so busy — *this*time* with 100% valid traffic!"
• 27 May 01: software problem. "I think these slow-downs are being caused by Microsoft's really pathetic news server implementation..."
• 27 May 01: service provider attack. "I think it was just another of those occasional routing-loop problems Verio seems to have from time to time..."
• 16 May 01: attacked. "Well, after six and a half hours the attack which lasted from 5:00 PM PST to 11:30 PST has subsided and we're back on the Net..."
• 14 May 01: attacked.
• 13 May 01: attacked. "As far as I know, Verio is still clueless. I've pretty much decided that they are simply lying about having 24/7 support. They have 24/7 telephone bodies, but I see no sign of them being able to take any action at night..."
• 12 May 01: service provider attack. "I fully expect our service to perhaps be unreliable for the next few hours. Verio [Gibson's service provider] jumped right on this instantly, have escalated it to the top level, and have asked for and received my permission to take us down for up to four hours, if required, to really nail the problem."

  "I think it was just another of those oc­ca­sional routing-loop prob­lems Verio seems to have from time to time..."

• 7 May 01: SELF-inflicted. "Something went wonky with the news server, and with the discussion groups web interface (which I hate) and then with the web server management interface ... so I did a full system shutdown and restart."
• 6 May 01: SELF-inflicted. "That wasn't an attack just now ... that was me... It seems that my 'kludge fix' for the ShieldsUP! second NIC problem upsets my eMail server ... which has been unable to send any eMail since Friday evening..."
• 5 May 01: attacked. "GRC.COM was knocked 'off the Net' for 17 hours due to a classic Distributed Denial of Service attack. The attack is still underway — and in fact it has been escalating throughout the day — but I was finally able to tell Verio what rules to place in their router in order to block the attack and we popped right back onto the Net..."