Downtime today - what happened?

vauge · 05-23-08, 07:07 PM

Well, we had a hacker - he/she was able to reboot the box. Looks like the ****er did it 3 times.

So far, nothing else seems out of whack - we are keeping a close eye on it. I've turned on more verbose logs and have contacted the authorities with the IP address that was used.

Site will be slow for a bit for the cache system to catch up.

Thanks for your patience.

-vauge

vauge · 05-23-08, 07:24 PM

DOH!

Well I just found out what happened... it makes more sense why a person was able to hack in so easily. I found zero indication of a break-in (normally they try to cover their tracks) but there was an IP address outside our hosts domain that did not resolve was sending the reboot commands and in places they shouldn't have been.

Obviously, we take hackers very seriously. Anywho - gotta call them and tell them to ignore the report but I am sure they know that already.

I'm still not happy camper - they should have given notice at least via email. I happened on this when I was just about to send another ticket.

Quote:

Originally Posted by our hosting service

At approximately 8 A.M. EST a segment of our data center experienced a power outage. Shortly after that we realized the city had experienced a power outage as well which attributed to the loss we saw in the NOC. We immediately went into generator power and we are still currently running on generator.

What happens when there is power loss is that we immediately go into UPS for a few minutes meanwhile the generator powers up. The UPS ensures no service disruption meanwhile the generator turns on. This happened normally but approximately 40% of the servers tripped during this process. Power loss and going into UPS followed by generator power has happened a few times in the past. But we have never experienced what has happened today - the power surge during this outage was extremely high.

We are meeting with our electrical engineers as we speak to analyze the situation to learn exactly what happened. We still have the data center running on generator as the power in the city could still be unstable and we do not want to risk any further outages. As many of you know these kind of issues do not happen often and we are taking this incident very seriously. The data center is equipped with a high level of UPS and generators are in place to ensure maximum uptime. We apologize for the inconveniences as we know this has effected a lot of you and your clients. We will be compensating per our SLA and will ensure we find out in detail what happened, resolve it completely and put measures in place so it doesn't repeat again.

All effected servers should be operating normally now and we sincerely thank you for your patience.

winston53660 · 05-23-08, 07:29 PM

Thanxs for all your effort

cnredd · 05-23-08, 08:06 PM

Al Gore just called and blamed global warming...

Hmmm...Where you suggest people go when stuff like this happens vauge?...

independent_thinker2002 · 05-24-08, 12:13 AM

Quote:

Originally Posted by cnredd

Al Gore just called and blamed global warming...

Hmmm...Where you suggest people go when stuff like this happens vauge?...

John McCain just agreed with Gore!

independent_thinker2002 · 05-24-08, 12:15 AM

vauge, I spent today wondering if it was TOT. It seems that these problems happen right after he has a sock puppet barrage. How was he able to take out the power grid?

Trinity · 05-24-08, 04:59 AM

It isn't that crazy to consider a mirror site, is it? I really missed this place when it was down many times today. Last forum I was in, the host did have a mirror site and it was something backup at least. Just for emergencies like this. It was always pretty sick when he got ddosed.