I came in to work this morning after about four hours of sleep to learn that yeah, it was going to be a ten hour shift! Voluntary, of course. Since it was an impromptu shift extension, the phone goons didn't have to stay past their scheduled end of shift (eight hours) if they didn't want to, but supervisors and staff got to stay as long as there were phone goons to watch! Whee!

It was about that time that we, in the form of Cute Geek Super (followed by the rest of us), discovered that one of the computers in the back room, Tombstone, was inaccessible. (Someone in the company evidently thought it would be cute, or perhaps sufficiently interesting yet bland enough for Corporate, to name not only rooms, but also pieces of the computer system, after local geography. So we've got the Grand Canyon phone room, the Lake Havasu phone room, and then the box Tombstone (among many others) in the back computer room (which is doubtless named something else cute and local).) Tombstone holds certain crucial files, such as the supervisor memos, many shared documentation files, and, especially, my precious check-in hours spreadsheets!

This was only a minor inconvenience, I decided. I could use Excel and create a new spreadsheet and keep half track of things, at least until IT got the thing functioning. There had already been a few pages sent off to IT; they'd be there to fix things any time. I headed off to get people seated in their booths with far too much energy. (Stressy College Chick Super may or may not have noticed the coincidence between the timing of Figment's arrival and my perking way up.) I was ready for a very good shift.

That was about when the Cute Geek Super discovered that actually, when one tried to log in to go on the phones, one encountered certain dreadful and fatal errors, and could not actually get anything done. It seemed that the interviewing system also ran through Tombstone, which was proving to be a fairly deadly obstacle in our day.

IT had been paged at 7:45 when Tombstone was discovered to be down, and again at 8:00. By 8:15, some of us were getting annoyed when no one from IT had so much as called in to see what we were making a fuss over. Stressy College Chick Shift Ops Super was nervous when they still hadn't responded to the fourth page at 8:30, and called a head honcho, and they determined that if IT hadn't answered the fifth page (8:45) by nine, that Stressy College Chick was calling not one of the intermediate or junior IT guys, but the lead IT guy, at home. The rationale was generally that the page should have summoned whatever IT guy who was on duty to call in, and when five pages did not summon an IT guy, the head IT guy should know about this. Fires were evidently lit under the appropriate tails.

There was reasonably little for me to do, once I had determined that yes, all sixty-five phone goons who were in out of the eighty-odd who were scheduled in were sitting cooling their heels and collecting downtime pay, so I went out on the floor and wandered and played riot control. (I realized about two hours before shift end that I could have actually done some paper distribution as well, because there were performance memos still to hand out. Oops.) Figment was a Good Little Phone Goon and managed to finagle permission to go through training surveys (that portion of the system was live enough to work); the Stressy College Chick said that anyone who wanted to do training surveys could do so, on the reasonable grounds that hey, nothing else was going on...

By 9:15, we were reasonably assured that someone from IT was on his way. "How far out does he live?" "I dunno." I started up a new Excel spreadsheet and amused myself by trying to calculate the person-hours we'd already accumulated on System Is Down downtime (at a minimum of $8.50+$0.50 per person-hour). The total was not nice.

The plan for strategic retreat went something like this: if IT failed to show up and/or fix the problem by 10-ish, Stressy College Chick would obtain permission to try Cute Geek Super's proposed work-around, through the old piece of software (Lucent PowerDialer) that we hadn't used in years (and 3/4 of the phone goons never were trained on). Cute Geek Super would do a trial call to see if it took. If it took in the system, we'd wind up spending serious amounts of time getting the phone center trained on the software and operational, but we'd finally wind up limping towards productivity. If that failed, then everyone would wind up going home, and there would be an utter hours panic Sunday and so on to the end of the month.

By 9:45, IT still wasn't there. I did my level best to keep those phone goons around me informed of the situation, and keep the noise level down to a dull roar. I poked at my spreadsheet.

IT, in the person of the 30-something young man with the sandy hair in the business haircut (the one who'd so famously knocked over his chair and huffed off with one of the computers tucked under his arm), was finally routed out of bed and showed up, insufficiently caffienated and with a rumpled iteration of one of his usual crisp button-down shirts (gaping at the neck instead of buttoned up all the way), around 10:10 or so. I wandered off the floor back into the bullpen to see what was up. IT took stock of the situation and disappeared into the back. I sat down at my desk and fiddled with the lame spreadsheet I'd concocted, to see if I could make it start adding things better, and keep track of the mounting tally of downtime, and took the internal phone call from the machine room at 10:14.

The system was back up.

IT re-appeared as the entire call center got itself set up, logged in, turned on, and dialing within what looked like a record five minutes. It seemed that Tombstone had unbooted itself in the process of processing an automatic update sometime Friday night, but had not rebooted for Saturday morning. "I keep telling them that having them do automatic updates Friday night is stupid," IT lamented. "They should do them Thursday night, that way we'll be here in the morning to deal with things if anything goes wrong, instead of having them do it on Friday when we won't be here Saturday morning."

$1161+ later, maybe they'll take his suggestion...