The root cause, I am told by our hosting provider was, two other customers of theirs that reside on the same physical hardware/server as ours generating huge volumes of traffic for a co-ordinated DoS (genial of service) attack at some server on the internet. As the name suggests, its to generate enough traffic/activity to overwhelm the target, rendering it inoperable and unavailable to normal users, hence the name "Denial of Service".
The DoS was not directed us, but because our traffic transits the same internet routers, server blades, and other common infrastructure, we were just collateral damage.
Our initial problem hit us on Jan 8 - which at the time seemed we were a very popular site on the Sunday evening.
Most users ever online was 96 on 08 Jan 2012 09:40pm
Since then we've had intermittent bursts when they've launched their attacks and was more prevalent during the last week and a bit. As you can see below from the graphs I've generated, the spikes are bascially when the server is getting around to servicing the backlog of clubcj member traffic. The server gets overwhelmed in dealing with the backlog and eventually grinds to a halt, the only options were to wait it out, until it digested all the requests or just kick the server in the guts. Middle of week 2 I, instituted a difference servicing scheme on the web server to alleviate the issue as then the hosting provider has given their infrastructure a clean bill of health. Of course that was short lived...
Anyway, let me know if you notice any major slows, I'm hoping this was it as the hosting provider has told me.
