TGIF my butt. Okay I had made it through the day and was out to dinner with my lovely wife when... My cell phone rings...
7:30 PM EST...
Security company calling, you have a power outage at your office... CRAP!!! I am third in line on the call list. This means that the owner and his son have been called at home and on cell phones and could not be reached. If they got me on the cell then they have called me at the condo already. My server will be toast before I get there. And as I hang up the cell the the food arrives... The Owner's son-in-law is put on the task of sorting out the power problem as he was an electrician before getting married and working for us... He calls the power company to check things out.. Hours pass... I call to see what is happening...
9:00 PM EST..
nothing from the power company...
10:00 PM EST
nothing from the power company...
11:00 PM
he calls back the power company and they say they where out at 9:30 but forgot to call him back. Everything to up to the power meter checks out, so the trouble must be inside...
11:30 PM...
in the car on the way to the office... My room is full of UPS units bleeping the tones of death. most of the servers had already crashed before the first phone got to me. I start shutting down the UPSs.
12:00
The son-in-law arrives.. We start checking out all the obvious causes... Nothing. Flip the main breaker off and on.. Nothing. Some outlets have full power, some no power, and worse, some have half power. The light glow an eerie low orange. Finally outside to the meter... Its digital and does not have a reading... Call the power company.. No this is not normal, the driver we sent out should have checked all the way to the meter, we'll send out another truck.
1:15 AM EST - Saturday morning....
the truck arrives the guy looks at the meter and then fires up the bucket truck and starts inspecting the transformer on the power.. Then the fuse links on serveral powers, then drives down the road using a spotlight to trace the wires... And finds the problem.. A palm tree two blocks away shorted one of the phases coming into the building., we are getting only two of three phases, hence the strange power in the building. He replaces the fuses and the power fully comes back on...
2:00 AM...
spin all the servers back up one by one. DC first and the the RAID servers, all 7 of them. Next the exchange server... And it will not boot. It has a config error and can only be fixed by going into the BIOS... But is will not see the KVM. Pull it from tea rack and fix. This time is boots and we have email again...
2:30 AM...
go home and die...

and that's how we have an IT night on the town...