We have a large maintenance this weekend as the electricians cutover power to a several hundred AMP 480V switch panel in preparation for bringing a large 675KW UPS online. Also, at the same time, they lumped in some plumbing work to cutover to the 14" chilled water mains. So, in order to avoid any sort of inrush issues, we're shutting down the 588 machines in the cluster.

The current measurement units on the Starline 400A buses still read between 20A and 28A after shutdown, which means that this power is divided among 50 24 port Voltaire switches, 2 288 port Voltaire switches, 20 HP 10/100 ProCurve 2600 series switches, 22 Force 10 S50N switches, and 2 Force 10 S2410 switches, as well as whatever inefficiency exists in powering 30 Servertech 60A CDUs, 4 30A 0U PDUS, 10 30A 2U PDUs (all of which have monitoring hardware on board), and a Cyclades ACS 48 port console server. That's a fair bit more current than I would have intuited, and a fair percentage of a medium loaded cluster consuming about 160 A per phase at 208V (3 phase).

The premise is believable enough. There are still quite a few vendors out there that still distribute BIOS updates as a floppy image. Heck, they even make it 'easy' for you.. The program comes in a self-contained executable that has the image inside it. All you have to do is run the thing and put in the floppy disk.. and...
Oh.. wait a second..
Where's that floppy disk?
Hmmm.
Ok, so we'll use the CDROM image and...
ARGH.. this machine is a server bought in large quantities.. It doesn't have a CDROM.
Fine Fine.. We'll use one of these new tools that can turn a USB flash key into a bootable device. It's a modern machine. It supports that (Finally! Something modernish!).

Say you want to strace a script.. Well, you can't, right? Because strace expects ELF (or other object format) executables as a target, or a running process. So one trick is to put a sleep at the top of the process, start it, do a quick ps, then strace -p.. Ugly, but effective in many cases.

Another thing you can do is grab the shebang line from the top of the script and parse it out and run strace on that and lookup the proper argument for the command interpreter that you are executing and pass all of that to strace.

Since I have to do this so frequent, and I don't want to care about which arguments work best with which interpreters, I wrote a little C generalized wrapper that reexecs the program I'm interested.

NTP 4 introduces some interesting new things that few people seem to know about, are sparsely documented, and are difficult to setup correctly, however they can help with synchronization in the event of total external network failure (even if you don't have a reference time source).

Now, some reference time sources aren't expensive (others are), but sometimes you care more about node-to-node synchronization than you do about absolute time accuracy. One example might be a large computational cluster where, if the network is disconnected from the Internet for a while, or if the primary time source is down, you don't want the individual nodes to drift apart.

Luke Kanies blogs a rant on configuration management that claims that the entire state of system administration is broken. Maybe he's right, maybe he's wrong. The problem is, he describes a microcosm of system administration in configuration management and then goes on to conclude that all tools of all aspects of system administration suck and that the field is stuck in the stone ages. It's quite a logical leap. It would be like me complaining that all tools in all of medicine stuck because dentists still use pliers and knives to pull teeth. How large a percentage of your day to day sysadmin work is spent doing configuration management that would benefit from tools like lscfg, bcfg2, puppet, cfengine, or radmind? For me, it's less than 1% of my daily work.

Anthony Spina writes an interesting article on the Splunk blog (here). Using network databases like this can make distributed operations much easier. How do you tag your machines?

Some tips on publishing content

Most people don't know some of the shortcuts you can use to make content publishing easier and more useful. One of the first lessons is using interwiki to provide quick links to reference material.

For some tips on using this, see http://lopsa.org/interwiki/6.

Also, below the list of input filters when you are composing your blog entry you will find a link that says More information about formatting options. This link can also be accessed directly at http://lopsa.org/filter/tips.

Daniel Feenburg at NBER writes a good review comparing performance, reliability, price, and other aspects of common RAID NAS solutions. Among those covered are Netapp, DNF, Excel-Meridian, and Linux white-box with RedHat Linux and promise IDE controller.

What makes this version somewhat unique is it deals with the problem of secondary unrecoverable bit-errors that were previously undetected but manifest themselves when you go to rebuild a failed drive. This problem is more common than one would hope, and it is explained well.

Stephen Potter and I are meeting lots of new potential members here . At least one new member has already signed up. Since they don't have official networking, I'm logged in via GPRS. Many people haven't heard of LOPSA, so raising awareness is part of our target. The event looks to be a big success. Hundreds of people filter by the booth at every break. The projected attendance is 1700 people. Our own Beth Lynn Eicher is one of the conference organizers and doing a great job, here.

There's some interest in creating a sysadmin bookshelf, which I think is a great idea. We've basically got all the 'categories' we need, I think, in the Standards category taxonomy. It's nice to reuse things for multiple purposes. It gives a sense of coherency to site organization and allows you to browse through things in multiple ways regardless of content type. The next step would be to do something like is done for Tools.
I'm thinking of a Bookshelf content type that would make use of some of the amazon features we have (affiliates program for discounts), or other vendors like Pearson where we have even better discounts. Then, add the voting API to the nodes to allow scoring. Associating them into the taxonomy allows browsing (you could find all books and tools related to Network Visualization. That'd be pretty cool). If anybody would like to work on this (some knowledge of drupal is helpful, but not required), let me know. Seems like about 10 hours of work to get something at least beta quality.