Like probably almost anyone reading this blog, I'm a sysadmin. Specifically, I consider myself to be a pretty darned good one - years of experience, blah blah blah. One of the personality traits a "good sysadmin" has, is the drive and ability to learn new things. COMPLICATED things. No fear. Try out installing and setting up high-availability linux clusters without ever having done it before, try implementing SSL in apache, etc-- you just roll up your sleeves and go. You'll stumble, but you'll LEARN. And you become a rockin' sysadmin in the end because of it.

Lately, I have been wrestling with being able to do that any more. Specifically, the office in which I work has been reclassified as a "service center" of sorts. We have tons of campus customers, and we now charge back for our time. Which means *I* have to charge back for my time. It's a result of the current economic situation, everyone needs to do what they can to survive.

OK, not related to work, but worthy of a blog entry at least.

The other day I dove into VOIP for the first time. I admit, I'm a bit of a holdout with regards to my home phone service. I'm also old enough to recall the days when the phone in our house plugged into the wall with that huge 4-prong plug, and Ma Bell engineer(s) needed to do ANYTHING with regards to phone jacks or phones. If you so much as clipped a wire, out came the Bell-Police :-) Getting to the point of cutting my AT&T service is emotionally difficult. I've ALWAYS had AT&T.

Anyway, I finally decided to switch to VoIP and cut my land line. I'm not there yet, as I'm currently testing out the device first. I opted for an "ooma", which is essentially a product that's "VoIP in a box". You pay for the unit (about $200 US), and that's it. Everything else is free for as long as you own the unit. No monthly charges, no fees, no regulatory charges, nothing. Free local calls, free long distance. And they support porting your land-line number to the device when you're ready ($40 fee).

Like many of you, I like to have some of the latest technology at my fingertips. Specifically, I'm talking about my workstation. No, while I don't have über-fast sexy hardware, I like to at least have the latest OS flavor(s) installed so I can play with new features.

The other day I decided it was "time" for Fedora 12. Time to play with ext4.

Before I begin, a small word about my prior setup:

disk 1: Windows XP, NTFS partition. Old, crusty, not used anymore, as I have XP in a VM now.
disk 2: Fedora 11, Linux LVM partition. No real data of value, but it does house my XP vm.

I love it. Just when I start getting a little bored..., something comes along that basically makes me go, "WTF?" - and ends up giving me a chuckle in the end. There's stuff that's broken, and then there's stuff that's REALLY broken....

Very recently, we've been tasked with helping migrate key components of an old server that belonged to another department. We're going to migrate the important stuff to a new server of ours,and retire the rest. A lot of this has been my task, but it also involves our DBA and a handful of developers taking a little time on the side to hunt and migrate.

The cool thing about being a sysadmin is you're always learning - whether you want to or not...

Today I sent an email to a colleague from my corporate email account, and while we were IRC'ing about other stuff-- he told me that no, he didn't get the email. Huh. I double-checked my sent-items folder-- yes, the email address WAS correct. No, still no email.

About 2 minutes later, he eventually found it - in his 'Spam' folder.

While I was thrilled he got the email (something that took me a bit of time to compose), I was now immediately wondering- why the heck was my email, something that contained only technical text (no attachments), considered as SPAM? Time to check it out.

Today I was working with my DBA on a server migration-- the replacement server is configured, loaded, secured- and before we lit it up, she asked me to take one special final full cold backup. No problem!

...until I started monitoring the backup (to estimate when it would be done) - and was seeing well, absolutely horrible backup rates (averaging "2,645KB/S"). Terrible.

So, I started checking the logs to see what kind of tape device issues I was having. Huh- nothing. Backup configuration (parallelism, etc)? Set just right.

Then I remembered- this server was one of those servers that was very recently moved to another rack. I shipped out one of my favorite sysadmin tools, 'ethtool'. OUCH. Not only was my network connection at 100MB (instead of gig speed), but it was 100/half. With one command, I verified that not only was the server plugged into the wrong port in the switch, but that the promised "we'll fix that next week" response from the network team was never followed through (they were supposed to fix the broken autonegotiation on the 100mb ports weeks ago).

It's Friday afternoon, about 2pm. I have a console/management server that is having some difficulties with Splunk 4. Whether or not the RHEL version (4.8) is the cause-- I have finally decided it's time to upgrade to RHEL current (5.4). That way I can at least eliminate the OS as the cause of the issues I'm seeing, and so on. And, being Friday afternoon, it's quiet.

What could possibly go wrong?

I checked my ISO directories for the particular RHEL image I need (ver 5.4, x86). Yep! I've used it a lot on several VM's, but haven't needed to upgrade anything that's actual hardware yet. No problem, a few minutes later I had a hot & crispy DVD in my hand with 5.4 freshly burned on it. Off to the data center.

(Reference, yesterday's blog about moving servers from one row of racks to another):

OK, so I had several servers to move to a new rack in another row. As of this morning, they were ALL done, except one. The person who we wrote the application for on that server (the "owner"), had not responded to my emails or phone calls asking for permission to shut down the server. He probably had meetings, whatever. Yet, I was being pressured to finish my move.

So, I decided to do what any good sysadmin would do in this predicament: Move it anyway. Oh yeah, and with ZERO downtime.

First, this works only if the server has multiple power supples (it did). Unplugged power, and re-hooked new power fed from the other rack (using standard rack power cords, a few strung together). Repeated for second connection.

I don't know about how it works in YOUR organization, but where I work, as the Unix Systems Admin (job title, "IS Project Manager", go figure) - I do everything to manage the server from birth to death. Ordering, receiving, unpacking, installation, management, performance, uptime, retirement. And all the bits in-between.

Today I was moving a rack of servers-- from one rack to another. (Rack in the way of an incoming AC unit, that's a different story for later). As I sat on the floor, unscrewing screws, removing little square nuts. removing cable arms, unrouting (and re-routing) cables, (and doing the reverse on the other end) - I realized that many many people don't necessarily equate rack management with being a "hardware guy". While I cannot confirm it, I think when most people hear "hardware guy" for computers, they thing things like, replacing disk drives, installing PCI cards, etc. You know, the dude at Best Buy who charges $195 to replace a disk drive for you.

OK, a few minutes ago, I had written a really lengthy blog entry about LISA. It was awesome. And then-- I got interrupted by someone needing help, and well, an incorrect click basically wiped out my posting- before I had submitted it. (Did I mention my personal hard drive at home blew up last week? Yeah, it's been one of those seasons..).

Anyway, I can't re-create the posting except to summarize: LISA was awesome. If you are a sysadmin of a large group of servers- you need to come. Large-scale sysadmins are a special group of experts, and they all come together (well ok, a lot of them anyway) once a year - to share. it's an opportunity that should not be missed.