Bodies in the dark
No, not a halloween party. I received a call from Allen at 4 pm on Saturday saying the power was down at Mail Order and judging from the signals out at the intersection it was an entire area outage. I was at home getting ready for a halloween party - popped open my laptop, verified no connectivity to Mail Order and so I packed my bag and headed out.
I made it to Mail Order by 4:20 or so and stumbled into the office section to the plaintive bleeping of the UPS boxes. I blindly stumbled to the door to the dotcom office, answering calls from people in the bookings office as to who I was, and brushed up against some warm body on the way. Once in the server room I got one monitor on for light and found all the machines had run out of juice except for two. One I was able to shutdown easily through the kvm monitor on the left but the other was on the right and the UPS driving the monitor had died. I ran to the car, grabbed my camping head lamp out of the glove box and a minimag flashlight. Getting the monitor and kvm switched over to a UPS with juice left I was able to shutdown gracefully the last machine. Craig called and I updated him on the situation.
Word from Kathi was that DTE estimated power back by 11 pm. We closed up the building and I headed over to Creamery and Bakehouse to imform them that internet and mail were down. I called Ron and he offered to call some other partners and let them know. All were very understanding about the outage. Though Grace surprisingly heard about the planned outage at Detroit St for Sunday for the first time. Don’t know how that happened. I called Joe, since he lives nearby and brought him up to speed and he offered to come in if needed.
I headed home to eat and think about further actions. Maybe rent a generator and get the internet connection back up? I gathered some extension cords and more flashlights and headed to Deli where I picked up some UPS 1000 boxes and the tool bags and talked to Grace. They did not have a strong need for internet connection that night so I scrapped the generator idea but instead decided to take advantage of the power outage to rearrange all the servers and their UPS boxes.
I called Craig and updated him as to my plan and he offered to join me later. Once at Mail Order I ran an extension cord from my Prius (and AC converter) to the server room and plugged in a lamp to see by. Craig arrived and we got down to business with a goal of each server having it’s own UPS, the big important ones on 1000s and the others on 700s. We also rearranged machines to get the new web servers on the shelf. Once done with the heavy lifting and power cord juggling we shifted our attention to the network equipment rack and straightened that out. It took a little over two hours but felt great to have it done. Craig left and as I was packing up my tools the power came back on (at about 9:45).
Checking all the machines I found them all to be in good shape except the two, arguably, most important ones. The mail server and web server. The web server wasn’t accessing it’s database server and mail wouldn’t load services. As I worked on those Billie stopped by for a bit to check in and provide moral support. After working on the mail server for at least an hour I finally called Craig to come in and help. I also called Tom and filled him in and learned that the web server was now working, though the database server was not providing a console. Good enough - focus turned to the mail server and with Craig on that I took a break and grabbed some food. Eventually Craig was able to determine what the problem was with the mail server and brought it around to his way of thinking. We checked a few more things then closed up. Made it home by 1am.