What we didn't know then was that a vendor had done something really naughty with a network switch and essentially created a bridge between the BYOC, the tournament network and the hotel's Internet. This caused a loop and very serious DNS lookup problems, forcing us to scrap games for the rest of the day in order to route out the source. To be clear, the NOC staff hadn't done anything incorrectly as a few of the comments in the main QuakeCon ESR post suggest. Rather, the network was being thwarted by someone doing something they never should have done in the first place.

It also meant that our plans to stream a few interviews that were complete to QLTV had to be put on hold as we were asked to pull the plug on all of our equipment in the booth to assist with troubleshooting. This also included the BYOC broadcasting rigs.

The Network Operations Center crew at QuakeCon stayed up all night working on the problem and I believe they essentially rebuilt the network from scratch to work around the loop that was still lurking and causing problems. The good news was that our streaming bandwidth was given its own gated community with guards and watchdogs with the hope that it would be immune to further shenanigans.

QuakeCon Volunteers and Staff Save QCon