Like probably almost anyone reading this blog, I'm a sysadmin. Specifically, I consider myself to be a pretty darned good one - years of experience, blah blah blah. One of the personality traits a "good sysadmin" has, is the drive and ability to learn new things. COMPLICATED things. No fear. Try out installing and setting up high-availability linux clusters without ever having done it before, try implementing SSL in apache, etc-- you just roll up your sleeves and go. You'll stumble, but you'll LEARN. And you become a rockin' sysadmin in the end because of it.

Lately, I have been wrestling with being able to do that any more. Specifically, the office in which I work has been reclassified as a "service center" of sorts. We have tons of campus customers, and we now charge back for our time. Which means *I* have to charge back for my time. It's a result of the current economic situation, everyone needs to do what they can to survive.

(Reference, yesterday's blog about moving servers from one row of racks to another):

OK, so I had several servers to move to a new rack in another row. As of this morning, they were ALL done, except one. The person who we wrote the application for on that server (the "owner"), had not responded to my emails or phone calls asking for permission to shut down the server. He probably had meetings, whatever. Yet, I was being pressured to finish my move.

So, I decided to do what any good sysadmin would do in this predicament: Move it anyway. Oh yeah, and with ZERO downtime.

First, this works only if the server has multiple power supples (it did). Unplugged power, and re-hooked new power fed from the other rack (using standard rack power cords, a few strung together). Repeated for second connection.

I saw on one of the news sites (Slashdot or OSNews, forget which exactly) a story about the following 10 Essential Rules for System Administrators. These are mostly pretty basic and many of them are not really SA specific. I'm not certain that "backup regularly" and "test your backups regularly" are deserving of being two separate rules; I consider that testing backups is part of the overall backup process.

One thing I thought was kind of interesting was the timing of this coming out with the number one rule being "Keep it Simple" and my first Black Belt System Administration topic "A punch is just a punch", which is not specifically about system design but about sticking with the basics.

The darcs revision control system has all but lost out to git within the past few months. A rather large part of the reason is a rushed and very poorly worded release announcement, following a rather long (I'm told 4 years; I haven't been aware of it that long, which itself is perhaps ominous) post-1.0 silence from the darcs developers:

Even people who are familiar with internal politics may not realize the extent to which technical decision-making can be a distant second, or worse, to political concerns. An extreme case is real politics:

I've recently had a number of projects at work that want to mix multiple volume managers on a single server, specifically ZFS and VxVM for SAN volumes (actually, three including SVM for internal boot disk mirroring). The projects generally are for database servers, and want to use VxVM for database volumes because ZFS currently has some serious limitation on database size (limited number of devices recommended in a single zpool) and performance (single threaded checksumming, for one). However, at the same time, they want to have access to some of ZFS's features (in particular, the ability to oversubscribe filesystems, dynamic resize, snapshots and rollback) for some of the other filesystems.

During the recent LoPSA live chat on #lopsa-live it was mentioned that there weren't enough topics for people to write about on the website and how it was harder for people to come up with an interesting topic than it was to actually write about that. That inspired me to finally draft a proposed series of topics based around the notion of "Scaling Up"; something that’s been bouncing around in my head for a few years.

Motivation

We spend a lot of effort building and describing tools and techniques - the craft of system administration. A precious few tomes reach beyond the command line to discuss the practice of system administration (I'm thinking specifically of the works of Tom Limoncelli & Christine Hogan.) What I'm searching for is a bridge between the two topics - what you want to eventually achieve and why (practice) and how you can actually achieve that (craft.)

Operations groups I suspect are going to have real problems soon, and not from the usual causes of automation. It is very common for operations groups to not only monitor the servers, but take on trivial tasks that need to be done out of hours, sometimes even during the business day in an effort to alleve the workload on the system administrator.

Here's the problem. Many more systems now store data that may be SOX impacting or some similar law that strictly regulates access. The system administrators themselves have a strong need to access the box, and usually are a fairly concrete and small team, but operations groups are much larger, maybe even offshored. I suspect that the access implications have not been fully thought out of giving these groups the access to do some of these root tasks.