Azure Jane Lunatic (Azz) 🌺 (![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png)
azurelunatic) wrote2015-10-03 12:03 am
azurelunatic) wrote2015-10-03 12:03 am
A Friday: poodle, irc, server room, success.
Thursday night: poodle. He's malfunctioning slightly (being 10 by now) and is now on 4x daily pills.
In the middle of the night I woke up with sensory hell due to my hands itching, enough so that I went home to get a pair of gloves to deal with it.
Work was somewhat surreal. lb got the new IRC server set up under his desk, and a few of us poked around on it and made sure it was stable before inviting everyone over. He sent messages in all the channels about what was up, and prepared to do the swap after lunch.
Since I'd been in early, I opted for earlier lunch, and with lb's table. I advised Purple of my plans, and he thought that sounded like a good idea, and decided to join as well.
Mr. Wizard Beard had been at lunch also; afterwards, he offered to show Purple some hardware in his server room, and asked if I'd like to come see as well.
This was my first time inside a full-blown server room. I'd peeked into a room in which there were a few racks which housed some of archive.org before, but that was a room with a couple racks. This is a room with rows and aisles and at least three ginormous cabinet air conditioning units. Mr. Wizard Beard explained some of the provenance of the servers: this one is from that vendor, the other one is from the other one, and they wanted to do X but then plans changed, and now they're doing Y, and we had to do something else with X hardware...
The door has a sign on the outside: NO FOOD. NO LIQUIDS. Without being told, I set my waterbottle down on the coffee table of the conversation group near the outside. There are windows into the room, and Mr. Wizard Beard pointed out some salient features of the hardware on one stack of servers next to the window. (I did not think of the parallels to meeting my baby sister for the first time through an observation window at that time, but will hold it in reserve.) Then we went inside.
The door is card-locked, as all outside doors but few inside doors are, at my workplace. Regular cards do not open this door. As soon as the door came open, a roar of fans sounded. Think of the gentle whirring noise of one desktop computer. Multiplied by the hundreds and perhaps thousands, and much more powerful machines, it almost sounds like an airplane getting ready to take off. Immediately inside the door was a sticky welcome mat sort of thing, for the purpose of getting extraneous dust off of shoes. The majority of the footprints on the mat looked like Mr. Wizard Beard's sneakers. I stomped a bit, to make sure to get as much dust as I could leave off.
I've heard server rooms opened before. I've heard the noise. I've heard about how there's an incredible amount of cooling that goes into it. I've seen pictures of cables, and blinky lights. The thing I was not prepared for was the wind.
Imagine a 10-year-old car on the hottest day of summer, with the air conditioning running full blast, coldish air streaming out the vents and blowing your hair around, even though the windows are fully sealed. Imagine a narrow gym locker room full of vents like that, cold air blowing intermittently into every aisle.
Imagine that for every cold air vent, there is also a hair dryer set to warm-but-not-crispy, also blowing out into the narrow aisles of the locker room.
Walking down the aisles of this server room is the air equivalent of drifting through a hot tub in full on bubble mode, on a day when the water inside has been allowed to cool down and it's just now starting to warm up. There are jets of hot and cold, and even by standing still, you feel different temperatures on different parts of your body. Purple tucked his glorious mane inside his t-shirt. My hair was securely braided already.
Mr. Wizard Beard's server room is neither the sort of wiring nightmare that you see horror pictures of, nor the sort of painfully neatly organized set of cables where you just know that in another month when things change and it needs to be wired up slightly differently, that there will be a very angry data center lord because *the wiring is no longer in aesthetic balance*. There was a glorious mismatch of types and colors of cable, and an incredible hodgepodge of very very expensive hardware, all with tidy little printed sticky labels on the boxes and little yellow card tags with handwritten notes (in a firm but terrible hand) for each cable. This is the domain of a fiftyish professional who has had the time and experience to know what works, what does not work, and keeps close track of everything that goes on within.
As we left, I saw the bucket of disposable earplugs by the main door. This is not a place where people converse regularly. Mr. Wizard Beard showed us the Big Red Button, by the back door and completely without a mollyguard. He uses this button any time he feels he needs to, if someone's in danger of getting hurt in some sort of electrical situation, or if the power's unstable or something and the machines are in danger. These days, he explained, an unexpected power-off is generally survivable, as modern machines have capacitor-powered failsafes that lift the head clear of the disk so we don't get the vivid "well, imagine a jet liner coming to a stop, and your disk is the orange on the runway..." scenario. However, say your power has gone out once. Your capacitor-powered failsafes have dragged the heads clear of the disks. How long does it take the capacitors to recharge to do the same trick again? Is it before the next power cut? That is not a bet that Mr. Wizard Beard wants to take. Cleaning up after that mess is worse than cleaning up after the mess of an unexpected clean power down.
When I got back to my desk, lb had accomplished the IRC swapover. He'd arranged the DNS swap, waited out the full ttl (5 minutes), then did one of the things he wasn't going to do because it would crash the old server, and crashed the old server to force a disconnect on everyone. Everyone with automatic reconnect showed up all right and tight on the new server. Everyone breathed a sigh of relief. The permanent new server should be up next week sometime, depending on when Mr. Wizard Beard has the new box that we'll be borrowing a corner of set up in his server room.
Haystack, for his part, had the upgrade to our little tool already installed. I did a round of data entry and confirmed that some of the questions I had from the last iteration are still valid questions this round.
For my sins, I will be taking on the helldesk queue for IRC. I told the ticket triage person, who was deeply amused by both the situation and my phrasing. We are working out the details.
Later in the day, Purple asked after a printer in my general area. 90% of the printers visible on the print server for his building were in some error state or other. He came through to pick up his print job, give me a hug, and head off to commune with his hackers, it being the first Friday of the month.
I can really recommend against panic attacks. I was headed back homeward in good time, but then remembered I'd left my iPad at work. Ordinarily this would be no big deal. On a poodlesitting weekend, it's better than hauling a laptop around. I went back, got it, and then various things played katamari damacy in my brain and voila! Panic attack! I pulled over at the park and ride and got my brain's ducks back from rowing into actual rows, which took a bit. I really did not get enough sleep.
The poodle seemed happy to see me. Yay poodle. I shall have to wash all the things once I get home after this, since while poodles are supposed to be less of a problem for allergies than many dogs, I don't actually wish for my coat to give Purple hives by accident.
In the middle of the night I woke up with sensory hell due to my hands itching, enough so that I went home to get a pair of gloves to deal with it.
Work was somewhat surreal. lb got the new IRC server set up under his desk, and a few of us poked around on it and made sure it was stable before inviting everyone over. He sent messages in all the channels about what was up, and prepared to do the swap after lunch.
Since I'd been in early, I opted for earlier lunch, and with lb's table. I advised Purple of my plans, and he thought that sounded like a good idea, and decided to join as well.
Mr. Wizard Beard had been at lunch also; afterwards, he offered to show Purple some hardware in his server room, and asked if I'd like to come see as well.
This was my first time inside a full-blown server room. I'd peeked into a room in which there were a few racks which housed some of archive.org before, but that was a room with a couple racks. This is a room with rows and aisles and at least three ginormous cabinet air conditioning units. Mr. Wizard Beard explained some of the provenance of the servers: this one is from that vendor, the other one is from the other one, and they wanted to do X but then plans changed, and now they're doing Y, and we had to do something else with X hardware...
The door has a sign on the outside: NO FOOD. NO LIQUIDS. Without being told, I set my waterbottle down on the coffee table of the conversation group near the outside. There are windows into the room, and Mr. Wizard Beard pointed out some salient features of the hardware on one stack of servers next to the window. (I did not think of the parallels to meeting my baby sister for the first time through an observation window at that time, but will hold it in reserve.) Then we went inside.
The door is card-locked, as all outside doors but few inside doors are, at my workplace. Regular cards do not open this door. As soon as the door came open, a roar of fans sounded. Think of the gentle whirring noise of one desktop computer. Multiplied by the hundreds and perhaps thousands, and much more powerful machines, it almost sounds like an airplane getting ready to take off. Immediately inside the door was a sticky welcome mat sort of thing, for the purpose of getting extraneous dust off of shoes. The majority of the footprints on the mat looked like Mr. Wizard Beard's sneakers. I stomped a bit, to make sure to get as much dust as I could leave off.
I've heard server rooms opened before. I've heard the noise. I've heard about how there's an incredible amount of cooling that goes into it. I've seen pictures of cables, and blinky lights. The thing I was not prepared for was the wind.
Imagine a 10-year-old car on the hottest day of summer, with the air conditioning running full blast, coldish air streaming out the vents and blowing your hair around, even though the windows are fully sealed. Imagine a narrow gym locker room full of vents like that, cold air blowing intermittently into every aisle.
Imagine that for every cold air vent, there is also a hair dryer set to warm-but-not-crispy, also blowing out into the narrow aisles of the locker room.
Walking down the aisles of this server room is the air equivalent of drifting through a hot tub in full on bubble mode, on a day when the water inside has been allowed to cool down and it's just now starting to warm up. There are jets of hot and cold, and even by standing still, you feel different temperatures on different parts of your body. Purple tucked his glorious mane inside his t-shirt. My hair was securely braided already.
Mr. Wizard Beard's server room is neither the sort of wiring nightmare that you see horror pictures of, nor the sort of painfully neatly organized set of cables where you just know that in another month when things change and it needs to be wired up slightly differently, that there will be a very angry data center lord because *the wiring is no longer in aesthetic balance*. There was a glorious mismatch of types and colors of cable, and an incredible hodgepodge of very very expensive hardware, all with tidy little printed sticky labels on the boxes and little yellow card tags with handwritten notes (in a firm but terrible hand) for each cable. This is the domain of a fiftyish professional who has had the time and experience to know what works, what does not work, and keeps close track of everything that goes on within.
As we left, I saw the bucket of disposable earplugs by the main door. This is not a place where people converse regularly. Mr. Wizard Beard showed us the Big Red Button, by the back door and completely without a mollyguard. He uses this button any time he feels he needs to, if someone's in danger of getting hurt in some sort of electrical situation, or if the power's unstable or something and the machines are in danger. These days, he explained, an unexpected power-off is generally survivable, as modern machines have capacitor-powered failsafes that lift the head clear of the disk so we don't get the vivid "well, imagine a jet liner coming to a stop, and your disk is the orange on the runway..." scenario. However, say your power has gone out once. Your capacitor-powered failsafes have dragged the heads clear of the disks. How long does it take the capacitors to recharge to do the same trick again? Is it before the next power cut? That is not a bet that Mr. Wizard Beard wants to take. Cleaning up after that mess is worse than cleaning up after the mess of an unexpected clean power down.
When I got back to my desk, lb had accomplished the IRC swapover. He'd arranged the DNS swap, waited out the full ttl (5 minutes), then did one of the things he wasn't going to do because it would crash the old server, and crashed the old server to force a disconnect on everyone. Everyone with automatic reconnect showed up all right and tight on the new server. Everyone breathed a sigh of relief. The permanent new server should be up next week sometime, depending on when Mr. Wizard Beard has the new box that we'll be borrowing a corner of set up in his server room.
Haystack, for his part, had the upgrade to our little tool already installed. I did a round of data entry and confirmed that some of the questions I had from the last iteration are still valid questions this round.
For my sins, I will be taking on the helldesk queue for IRC. I told the ticket triage person, who was deeply amused by both the situation and my phrasing. We are working out the details.
Later in the day, Purple asked after a printer in my general area. 90% of the printers visible on the print server for his building were in some error state or other. He came through to pick up his print job, give me a hug, and head off to commune with his hackers, it being the first Friday of the month.
I can really recommend against panic attacks. I was headed back homeward in good time, but then remembered I'd left my iPad at work. Ordinarily this would be no big deal. On a poodlesitting weekend, it's better than hauling a laptop around. I went back, got it, and then various things played katamari damacy in my brain and voila! Panic attack! I pulled over at the park and ride and got my brain's ducks back from rowing into actual rows, which took a bit. I really did not get enough sleep.
The poodle seemed happy to see me. Yay poodle. I shall have to wash all the things once I get home after this, since while poodles are supposed to be less of a problem for allergies than many dogs, I don't actually wish for my coat to give Purple hives by accident.
no subject
no subject
It smells like hundreds of running computers as well.
no subject
Also thank you for the image of things playing katamari damacy in your brain. Shitty to experience, but a wonderful metaphor.
no subject
This one is a test lab for a lot of work-related stuff, which is why it's such a hodgepodge of hardware. I don't think anything in it is exposed to the greater internet.
A truly great metaphor may make it easier the next time my brain does this. We'll see.
no subject
A little picture of home.
no subject
The noises of data centres are all very different when you're there because something's exploded.
But having said that, I do miss proper hardware. If I'd sorted things out, I wanted to do a road trip to a German data centre to cheaply rack the 'cloud' beast I've been fiddling with, but alas, it didn't happen. These days you can be a perfectly effective sysadmin and never even touch a piece of actual server hardware. It's weird.
no subject
no subject
I won't talk about the time I accidentally knocked the power out of the switch in the 1/4 rack my boxes were in and my hand was too big to reach through the gap to plug it back in, because that was just embarrassing. I'd never get away with that in a normal commercial colocation setting: the guy running the colo was a uni friend of mine.