Craig and I had a good meeting with our new rep from our network security vendor. We’ve had a relationship with the vendor for over 6 years and occasionally get a new rep who we bring up to speed. It’s always a two-way learning. Our new rep has a solid background in the field with lots of local connections in the industry. We talked about WAN changes and our goal of optimizing for speed/reliability/cost and some things we’re experimenting with right now and looking forward a couple of years. We also had some good chat time on where the network security industry is going with ongoing consolidation of vendors. And the ongoing topic of IP telephony came up and we shared various scenarios on how that might work for us. Every year we get closer to overwhelming compelling reasons for this dramatic change.

This morning from 3:30 to 6:15 our firewall decided to play hide-and-seek with Tom and I. It hid pretty well - blocking all incoming and outgoing traffic. It was a sense of deja-vu from our NYC trip but a different piece of equipment this time. I’ve never felt great about our brand of firewall but as Tom pointed out, once we’d fixed it, “Firewalls are just black magic and you’re a good wizard. Stands to reason you won’t get along.”

The house finally moved today! At around 10:30am, the cable people started mobilizing to take down the wires, and Brian and I proceeded to shut down the servers and workstations. Then we went outside. It was a thrilling sight. We (Craig and Roan were there too) stood at the corner of Detroit and Fifth and took pictures as it inched its way up the street. It was interesting to watch the different companies take down their respective cables. The trucks pulled in and out, cherry pickers swung up and down, lines fell in sequence. The house was about to roll on through when one of the men noticed a thin wire running between the ATS and Kerrytown buliding. They quickly sent someone up to cut the wire down. The house passed safely on by; the only thing preventing it from barreling down the street were two shovel trucks. Power came back on almost an hour later. ATS came back up okay. The loft is still down.

Update 12:30 pm:
Apparently, Comcast jumped ahead and began reinstalling all their lines. DTE is pissed because they’re supposed to go first, being the power guys and all. So they have to wait till the cable boys finish before they can get power back up to the Loft.

Update 1:30 pm:
Loft is back up! The Deli is packed with customers - everything is back to normal. That is, if you ignore the mess of wire and leaves left in the wake of the house up Fifth Street.

Sunday. A historic house is planned to be moved down 5th, right by our building and we’ve been informed the power will go out for a ’short duration’ between 7am to 10pm. We were unable to verify whether that included the Deli or just ATS and we have no way to learn whether it will happen or at what time. Sort of inconvenient.

So we’ve scheduled all IT people across the 15 hours to be here when (if) the power goes down. I’ve made a checklist of machines to shutdown and issues to look for. Joe has 7 to 10, Elph 10 to 1, Brian 1 to 4, Craig 4 to 7, and Billie 7 to 10.

Update Monday, Oct 30:

The house did not move on Sunday, but we got some stuff moved around in the office. According to the AANews it is scheduled for this coming Sunday again. We still don’t know if the power outage will affect the Deli or not but are trying to find out. This Sunday Brian is on duty for the IT7 holiday project, along with Billie, until 5 pm.

Update Tuesday, Oct 31:

The Deli is not part of the planned power outage. Just the ATS. That’s a huge relief.

No, not a halloween party. I received a call from Allen at 4 pm on Saturday saying the power was down at Mail Order and judging from the signals out at the intersection it was an entire area outage. I was at home getting ready for a halloween party - popped open my laptop, verified no connectivity to Mail Order and so I packed my bag and headed out.

I made it to Mail Order by 4:20 or so and stumbled into the office section to the plaintive bleeping of the UPS boxes. I blindly stumbled to the door to the dotcom office, answering calls from people in the bookings office as to who I was, and brushed up against some warm body on the way. Once in the server room I got one monitor on for light and found all the machines had run out of juice except for two. One I was able to shutdown easily through the kvm monitor on the left but the other was on the right and the UPS driving the monitor had died. I ran to the car, grabbed my camping head lamp out of the glove box and a minimag flashlight. Getting the monitor and kvm switched over to a UPS with juice left I was able to shutdown gracefully the last machine. Craig called and I updated him on the situation.

Word from Kathi was that DTE estimated power back by 11 pm. We closed up the building and I headed over to Creamery and Bakehouse to imform them that internet and mail were down. I called Ron and he offered to call some other partners and let them know. All were very understanding about the outage. Though Grace surprisingly heard about the planned outage at Detroit St for Sunday for the first time. Don’t know how that happened. I called Joe, since he lives nearby and brought him up to speed and he offered to come in if needed.

I headed home to eat and think about further actions. Maybe rent a generator and get the internet connection back up? I gathered some extension cords and more flashlights and headed to Deli where I picked up some UPS 1000 boxes and the tool bags and talked to Grace. They did not have a strong need for internet connection that night so I scrapped the generator idea but instead decided to take advantage of the power outage to rearrange all the servers and their UPS boxes.

I called Craig and updated him as to my plan and he offered to join me later. Once at Mail Order I ran an extension cord from my Prius (and AC converter) to the server room and plugged in a lamp to see by. Craig arrived and we got down to business with a goal of each server having it’s own UPS, the big important ones on 1000s and the others on 700s. We also rearranged machines to get the new web servers on the shelf. Once done with the heavy lifting and power cord juggling we shifted our attention to the network equipment rack and straightened that out. It took a little over two hours but felt great to have it done. Craig left and as I was packing up my tools the power came back on (at about 9:45).

Checking all the machines I found them all to be in good shape except the two, arguably, most important ones. The mail server and web server. The web server wasn’t accessing it’s database server and mail wouldn’t load services. As I worked on those Billie stopped by for a bit to check in and provide moral support. After working on the mail server for at least an hour I finally called Craig to come in and help. I also called Tom and filled him in and learned that the web server was now working, though the database server was not providing a console. Good enough - focus turned to the mail server and with Craig on that I took a break and grabbed some food. Eventually Craig was able to determine what the problem was with the mail server and brought it around to his way of thinking. We checked a few more things then closed up. Made it home by 1am.

4/Onsite

Each IT staff member commits to spending at least 4 hours at a Zingerman’s business. We rotate every week among five locations - Deli & Catering, ATS & ZHouse (ZingTrain, ZingNet, DSE), Roadhouse, Bakehouse & Creamery, Mail Order & Coffee. While out there we make ourselves available, we attend huddles, inspect and clean the machines, and offer direct on-the-spot technical support to anyone in the area. We ask questions, build relationships and trust, and get to know the routines of the businesses.

Bringing IT to the People: We realize that many staff members are very busy and may not be able to immediately email or phone IT when they encounter technical problems. Now there’s always an IT person in the area, so even the tiniest concerns get addressed. Just being around lets ‘cobbers know we’re here for them.

Report/Measure: We report on our on-sites during our weekly huddles. We share what we learned and document problems we’ve encountered and their solutions.

Better Interactions: The 4 on-sites let us give a higher level of service and a personal touch to our interactions with our customers. We found that our customers appreciate our presence and the full attention we are able to provide.

12″ iBook G4, 12″ & 15″ PowerBook G4
https://depot.info.apple.com/batteryexchange/

15″ MacBook Pro
https://support.apple.com/macbookpro15/batteryexchange/

NYTimes article

Dell is recalling 4.1 million notebook computer batteries because they could erupt in flames, the company said today. This will be the largest safety recall in the history of the consumer electronics industry, the Consumer Product Safety Commission said.

Dell Battery Exchange Program

• GENERAL

Explanation of Outlook Today/Welcome message
- Outlook opens to the last folder viewed, rarely the Outlook Today window
- Not necessary to explain in the handout (?)
Different menu views and how to expand menus
- Covered in all classes
- Not necessary to explain in the handout (?)
Preferences for sorting contacts
- Alphabetical by default
- Further sorting an advanced feature
Keeping under quota
- We don’t enforce quota
Organizing Tool
- Assumes too much and can be difficult
- Advanced feature

• SENDING

Request A Read Receipt
- Rarely used
- Advanced feature

• CONTACTS

Adding Contacts From Scratch
- Redundant, covered in the next section “Creating A New Contact”

• TASKS

Using Tasks Pro
- Tasks Pro falling out of use

• CALENDAR

Inviting Members/Invitations to a Meeting
? Advanced feature

Well, the reboot of NUPU looked like it was going well last night… For the first three seconds. Then as I had remoted the screen started filling up with help windows. Quickly. As it reached a hundred of them I had gotten to Ctrl-Alt-Del and as that opened it started filling in the comment window with 11111111111111111. I stabbed at the restart button and waited while the count of help windows went from 197 down to 1 and then everything went black.

I started a ping session, noted when it stopped responding and then waited. 5 minutes. 10 minutes. 15 minutes. No NUPU.

The drive to Mail Order was uneventful.

When I got to Mail Order I found the server room to be what felt like 110 degrees, but after bringing in a thermometer saw that it was only 90.

Nupu was stuck in recognizing the harddrives and making this pathetic screeching sound, like a kitten under a boot making it’s last gasps. Okay, that may be a bit too vivid.

I pulled the front face plate off of NUPU and the screeching stopped momentarily - I hit the restart button and then replugged in the keyboard and mouse that had popped out of their sockets (being plugged into the front.) As it rebooted I headed out to look for cool. I cranked the A/C in the office to high but it didn’t feel very cold. I then went into the admin office and set the thermostat to FAN ON so that it would at least vent the air from the server room = there’s an air intake in the server room, but no air vent blowing in.

When I got back to NUPU the pathetic mewing had returned and it has stopped at harddrive checks. I pulled on the front faceplate again and the mewing stopped and the boot commenced one step, then the sound returned. Then in a flash of groggy heat-laden insight I yanked the keyboard and mouse plugs out of the front of the machine and the sound stopped and the booting commenced. I followed the cords back and found a backup tape cartridge sitting on the keyboard effectively pressing several keys. One quick swipe to rectify that and plugging the keyboard and mouse back in and the server came up from boot just fine.
The drive home was uneventful.