When I'm playing I get a lag spike when other players connect to the server.

Any one else experienced this? And found a solution?

Specs:

Intel Core i5-3570k
Geforce GTX 670
8 GB RAM
SSD harddisk
64 bit Windows 7
20/2 ADSL connection

Tried:
Reinstalling Quake Live
Running it through other browsers (IE, Firefox, Chrome) - I use QLPrism
Running a default config