The penguins compel you.
So, there’s a job out there advertising for a Systems Administrator, who would work at a company in Vancouver that I’ve been interested in working with since they opened up their system’s public beta program. I’m really quite happy at the place I’m at, though there are three bosses and they all want TPS reports filled out daily….
- 5 years system administration experience with Linux and Apache (some network administration experience strongly preferred)
- Experience with both 32 and 64bit systems
- Experience with both hardware and software approaches for load balancing web serving and database traffic
- Experience in firewall administration and best practices for security
- Basic network design and administration
- Current knowledge of hardware systems (servers and networking gear)
- Prior experience running mid-sized systems (>30 servers)
- Expertise with MySQL, Squid and/or ImageMagick
- Experience in storage clustering
- General programming and scripting skills (especially php, perl, java)
- Systems architecture/planning experience
- Experience dealing with vendors (negotiation, support contracts, etc.)
- Lifting heavy servers into cages
- Good screwdriver skills (e.g., tighten, loosen)
- Know how to wear a pager … with style
- Management/team leadership experience
I have the requirements nailed down, and the “bonus skills” I’m running at 8/9.
My resume is available as a starting point, and I’ll reference it in the points below.
5 years system administration experience with Linux and Apache (some network administration experience strongly preferred)
I don’t count the two years with PCsupport / Voyus, but even then, I have 8 years of professional Linux sysadmin working for ISPs, web hosts, and web enabled companies. Also factoring in experience providing high uptime systems; 99.999% is not enough, and 100% is completely doable.
Experience with both 32 and 64bit systems
Most of my 64bit is in the Sun and Digital / Compaq world, and I’m only now starting in on systems with Opteron CPUs in them.
Experience with both hardware and software approaches for load balancing web serving and database traffic
I’ve run a number of software load balance systems, mostly all built inhouse around IP and MAC takeover or DNS tricks. Hardware such as BigIP / Arrowpoint has been used in most systems as well.
Experience in firewall administration and best practices for security
Deny All, allow some. I have a pretty well defined series of scripts I’ve built up to run my firewalls.
Basic network design and administration
From Superb Internet on up… where I didn’t design the networks, I typically re-designed them to fully segregate devel environments, establish proper firewalling between networks, etc.
Current knowledge of hardware systems (servers and networking gear)
I’m currently working mostly with server rackable 1U boxes from Tyan and SuperMicro, but fully aware of what’s happening in hardware currently. Outside of Cisco… they’re always doing something weird.
Prior experience running mid-sized systems (>30 servers)
Superb Internet consisted of a couple dozen Sun stations and a half dozen Windows and Linux machines, in addition to the office servers for the telephone system and office file stoage. Not a big homogenous network, but large. Globalmedia was the place where I had to deal with a decent sized system, and redesign some of it in the process. There were parallel systems for Development, staging, and rollout, each consisting of mirror copies hardware-wise, even down to the load balancing BigIP boxes. QA really liked this setup. In the NOC alone were 40 racked servers, in addition to the “live” system of a half dozen machines at our hosting facilities in Seattle and Toronto. There were also dozens of boxes “in the wild” acting as remote encoders, shipping audio and video streams back home over ISDN links.
Expertise with MySQL, Squid and/or ImageMagick
I run mysql for a number of online applications and use it in a couple of my contracts under heavy load situations. I’m no SQL god, but I can run the servers and maintain them. I leave it to DBAs to structure things, etc.
I’ve used squid to do silly things like block ad graphics from being requested, and to provide a cleaner appearing connection when running an entire office over two bonded 56K dialup modems.
I love mogrify and composite…
mogrify -fill red -gravity center -font @/usr/X11R6/lib/X11/fonts/TTF/Engraver-Light.ttf -pointsize 30 -draw "text 0,0 foo.ca" a*
Experience in storage clustering
This is the one I don’t have practical experience in. I’ve mostly been a centralised RAID tower mountable via NFS or SMB… nothing clustery.
General programming and scripting skills (especially php, perl, java)
I write perl and bash scripts for system maintenance purposes, and have been paid to code in PHP and Java (Rewardstream, Globalmedia, Superb Internet, and Alberta Supernet / Cyberion)
Systems architecture/planning experience
Rewardstream was my big go at this, though some of this was beginning to happen at Globalmedia. I was hired by Rewardstream to build out their IT systems. They had none, but they did have code written that they were going to deploy… I was the guy who planned the hardware end of things, planned for system redundancy, sourced vendors, setup, and installed the IT infrastructure.
Experience dealing with vendors (negotiation, support contracts, etc.)
My favorite skill that I’ve picked up is the ability to threaten a vendor if they don’t deliver. Not unreasonable things, just that if they don’t do what they promised, I’ll go elsewhere. Well, it’s not so much the threats that I’ve learned, it’s the following through. I maintain a couple vendor relationships now, and at any time one of them could go away.
Lifting heavy servers into cages
Rewardstream for this one. If you’ve ever had to lift a Sun e450 to fit in a cage that’s 3 feet off the ground, you will understand pain. The e250s weren’t lightweights either. The hazards of buying cheap cage space rather than rack space.
Good screwdriver skills (e.g., tighten, loosen)
Why limit yourself to two options? Screwdrivers are multipurpose tools… scrape, pry, puncture, parry, thrust…
Know how to wear a pager … with style
I have a sherpa who carries all of my electronics for me. I’m not sure if that’s stylish, but it’s practical – no unsightly bulges in my coat pockets.
Management/team leadership experience
Rewardstream I was lucky enough to be able to hire a half dozen people – recruit, interview, and hire. Some of them were direct reports, and some of them were hired as developers, etc. I did have to fire a couple people, which wasn’t ideal. At Globalmedia I was a team lead of 6-8 people in the network center. We were running the QA and test systems and the process by which code and updates went live daily.