Event: QC'10
"Do you smell that?", Slava asked me. "Yeah... that's...", I was interrupted by the entire tournament stage going dark. The power had been severed. "There's a fire under the stage!", I heard someone say. “Looks like I picked the wrong week to quit sniffing glue,” I thought as my memory replayed the famous clip of the late Lloyd Bridges in Airplane. ---
The morning had been spent doing equipment tests and working out a serious problem we had detected with sound synchronization on the stream. We had been handed some computers that were more powerful than our own and planned to use them as the streaming rigs to avoid our equipment being any sort of bottleneck. For those that are unaware, streams through Flash Media Live Encoder and use of a software video mixer will absolutely max any processor but an i7. When the processor maxes out, serious frame drops occur and the stream quality is degraded. Using fast hardware was a necessity, especially considering we were going to be pumping out a simultaneous HQ and LQ feed.
The night before, we had discovered an audio sync problem and stayed up into the wee hours trying to remedy it. Even without Internet, we were able to troubleshoot that the streaming box itself, one of the loaned machines, was introducing a video delay. After trying completely new hardware, swapping out video cards, pulling software components out of the pipeline, “borrowing” one of the Alienware rigs off of the stage for “testing” only to discover it didn’t have a PCI slot, refreshing drivers and just about everything else we could consider, we found a solution after replacing the newer, loaned machine with one of our own. The major difference? The operating system. One OS downgrade later, we had a proper stream ready to go. "One crisis averted.", I thought. None of this mattered, of course, because Internet connectivity was still unavailable. Still, we could preview our stream with the projectors at the tournament stage. These would display our content on large screens hanging high above the stage. On our end, at least, things were ready.
We had finally gotten a connection and were literally seconds from testing the stream on Day 1 when the power was cut due to the fire mentioned above. Fortunately, the fire (actually just smoke and that lovely “electrical burning” smell, no real flames) was caught early and no one was hurt. The start of the smoke was traced back to one of the hotel's pieces of equipment. While QuakeCon has been blamed for knocking out the power in previous years, it was innocent this time.
Now that Hits the GameSpot
With the power out, no real testing could be done. We had thankfully performed some streaming stress tests the night before from our booth. The results weren't as promising as the ones I had done from home. A few select people that tuned into our test page reported serious buffering and framerate issues and, of course, the sound and video sync problem that we had spent the morning fixing. When the power and net connection finally returned, we began further testing. The audio sync problem was gone but the buffering and framerate problems remained. We checked and double-checked our end of things. We were pumping out 30 frames steadily. We tweaked the bitrates and found that even a mere 500kbps was troublesome for most viewers on the QuakeLive.com stream. A few people reported things were fine on their end, but the ratio was unacceptable. We had to keep working to improve whatever we could.
Enter GameSpot. id Software and Bethesda had worked out a deal with GameSpot to syndicate our content. Ryan, GameSpot's rep at QuakeCon, was a Quake veteran. He loved the game and knew what was up. He understood the importance of QuakeCon. He was also a one-man tech and A/V army. His equipment patched into ours flawlessly and, when he began streaming, we saw a smooth, non-buffering stream on his page. "Thank gawd!", I thought. At the very least, our stuff was working properly. We turned to the QuakeLive.com streaming provider to see if there was anything he could do on his end, as we could now verify our setup was solid.
A DNS issue prevented Internet access later in the day and the tournament network still wasn't working by the end of the day. We sadly posted to twitter and ESR that coverage would have to truly begin the next day.
Day 2: Stream at Last, Stream at Last!