[ADMIN]

For approx 26hrs HGS and related websites were down due to a catastrophic failure at the datacenter that houses our servers. This is one of the largest (if not the largest) datacenters in the US (and a very large company).

It was basically due to the error of a low level tech at the datacenter (not an employee of mine, but of the datacenter). They were supposed to be adding a hard drive to the server and accidentally formatted/wiped the WRONG drive. It took getting the CEO of the company directly involved, calling in top recovery experts and my staying up most of the 48 surrounding hours (2.5 hrs sleep).

We are back and with 0 data loss (keeping fingers crossed!).

Going to sleep now.

-Victor/Admin

[Super Step]

Thank You Victor!
Now get some sleep.

[ADMIN]

(Woke up, will probably nap again in a bit)

FYI even if the entire server was destroyed in a fire/flood/earthquake/etc we have regular off-site backups of everything at a location hundreds of miles away. This occurs at least every 3 days (soon to change to every day). So worst case scenario ever, we could lose a few days of posts and be down for a day or two switching to another server. We actually have redundant hardware on the server so even if a drive fails the server keeps running fine. This was a freakish human error (and yes they are in trouuuubbbblllle).

-Victor/Admin

[Bailey-Boots]

Oh wow Victor, that sounds like a pretty manic time.

I guess someone somewhere is writing out "I must not type format C:\ ... I must not type format C:\...I must not type format C:\"

I hope you did get some sleep in the end

[Super Step]

Originally Posted by Bailey-Boots

Oh wow Victor, that sounds like a pretty manic time.

I guess someone somewhere is writing out "I must not type format C:\ ... I must not type format C:\...I must not type format C:\"

I hope you did get some sleep in the end

Or someone, somewhere is applying for a new job, and not listing the most recent employer.