doug's blog

musings on a maint

doug — Fri, 14 Mar 2008 22:28:16 -0700

We have a large maintenance this weekend as the electricians cutover power to a several hundred AMP 480V switch panel in preparation for bringing a large 675KW UPS online. Also, at the same time, they lumped in some plumbing work to cutover to the 14" chilled water mains. So, in order to avoid any sort of inrush issues, we're shutting down the 588 machines in the cluster.

The current measurement units on the Starline 400A buses still read between 20A and 28A after shutdown, which means that this power is divided among 50 24 port Voltaire switches, 2 288 port Voltaire switches, 20 HP 10/100 ProCurve 2600 series switches, 22 Force 10 S50N switches, and 2 Force 10 S2410 switches, as well as whatever inefficiency exists in powering 30 Servertech 60A CDUs, 4 30A 0U PDUS, 10 30A 2U PDUs (all of which have monitoring hardware on board), and a Cyclades ACS 48 port console server. That's a fair bit more current than I would have intuited, and a fair percentage of a medium loaded cluster consuming about 160 A per phase at 208V (3 phase).

So, you've got to do a BIOS upgrade on a modern computer and it only comes in DOS

doug — Wed, 31 Oct 2007 19:02:29 -0700

The premise is believable enough. There are still quite a few vendors out there that still distribute BIOS updates as a floppy image. Heck, they even make it 'easy' for you.. The program comes in a self-contained executable that has the image inside it. All you have to do is run the thing and put in the floppy disk.. and...
Oh.. wait a second..
Where's that floppy disk?
Hmmm.
Ok, so we'll use the CDROM image and...
ARGH.. this machine is a server bought in large quantities.. It doesn't have a CDROM.
Fine Fine.. We'll use one of these new tools that can turn a USB flash key into a bootable device. It's a modern machine. It supports that (Finally! Something modernish!).

a handy strace hack

doug — Wed, 08 Aug 2007 11:49:25 -0700

Say you want to strace a script.. Well, you can't, right? Because strace expects ELF (or other object format) executables as a target, or a running process. So one trick is to put a sleep at the top of the process, start it, do a quick ps, then strace -p.. Ugly, but effective in many cases.

Another thing you can do is grab the shebang line from the top of the script and parse it out and run strace on that and lookup the proper argument for the command interpreter that you are executing and pass all of that to strace.

Since I have to do this so frequent, and I don't want to care about which arguments work best with which interpreters, I wrote a little C generalized wrapper that reexecs the program I'm interested.

Towards a resilient NTP configuration in NTP4

doug — Thu, 12 Jul 2007 06:15:51 -0700

NTP 4 introduces some interesting new things that few people seem to know about, are sparsely documented, and are difficult to setup correctly, however they can help with synchronization in the event of total external network failure (even if you don't have a reference time source).

Now, some reference time sources aren't expensive (others are), but sometimes you care more about node-to-node synchronization than you do about absolute time accuracy. One example might be a large computational cluster where, if the network is disconnected from the Internet for a while, or if the primary time source is down, you don't want the individual nodes to drift apart.

Luke on Config Mgmt

doug — Fri, 09 Feb 2007 11:51:35 -0800

Luke Kanies blogs a rant on configuration management that claims that the entire state of system administration is broken. Maybe he's right, maybe he's wrong. The problem is, he describes a microcosm of system administration in configuration management and then goes on to conclude that all tools of all aspects of system administration suck and that the field is stuck in the stone ages. It's quite a logical leap. It would be like me complaining that all tools in all of medicine stuck because dentists still use pliers and knives to pull teeth. How large a percentage of your day to day sysadmin work is spent doing configuration management that would benefit from tools like lscfg, bcfg2, puppet, cfengine, or radmind? For me, it's less than 1% of my daily work.

Anthony Spina blogs on tagging

doug — Fri, 09 Feb 2007 11:39:57 -0800

Anthony Spina writes an interesting article on the Splunk blog (here). Using network databases like this can make distributed operations much easier. How do you tag your machines?

LOPSA blogging tips and shortcuts

doug — Sun, 29 Oct 2006 17:43:37 -0800

Some tips on publishing content

Most people don't know some of the shortcuts you can use to make content publishing easier and more useful. One of the first lessons is using interwiki to provide quick links to reference material.

For some tips on using this, see http://lopsa.org/interwiki/6.

Also, below the list of input filters when you are composing your blog entry you will find a link that says More information about formatting options. This link can also be accessed directly at http://lopsa.org/filter/tips.

NAS appliances comparisons and pitfalls

doug — Sun, 29 Oct 2006 17:23:03 -0800

Daniel Feenburg at NBER writes a good review comparing performance, reliability, price, and other aspects of common RAID NAS solutions. Among those covered are Netapp, DNF, Excel-Meridian, and Linux white-box with RedHat Linux and promise IDE controller.

What makes this version somewhat unique is it deals with the problem of secondary unrecoverable bit-errors that were previously undetected but manifest themselves when you go to rebuild a failed drive. This problem is more common than one would hope, and it is explained well.

Ohio Linux Conf

doug — Sat, 30 Sep 2006 08:07:08 -0700

Stephen Potter and I are meeting lots of new potential members here . At least one new member has already signed up. Since they don't have official networking, I'm logged in via GPRS. Many people haven't heard of LOPSA, so raising awareness is part of our target. The event looks to be a big success. Hundreds of people filter by the booth at every break. The projected attendance is 1700 people. Our own Beth Lynn Eicher is one of the conference organizers and doing a great job, here.

Sysadmin Bookshelf

doug — Fri, 04 Aug 2006 09:47:50 -0700

There's some interest in creating a sysadmin bookshelf, which I think is a great idea. We've basically got all the 'categories' we need, I think, in the Standards category taxonomy. It's nice to reuse things for multiple purposes. It gives a sense of coherency to site organization and allows you to browse through things in multiple ways regardless of content type. The next step would be to do something like is done for Tools.
I'm thinking of a Bookshelf content type that would make use of some of the amazon features we have (affiliates program for discounts), or other vendors like Pearson where we have even better discounts. Then, add the voting API to the nodes to allow scoring. Associating them into the taxonomy allows browsing (you could find all books and tools related to Network Visualization. That'd be pretty cool). If anybody would like to work on this (some knowledge of drupal is helpful, but not required), let me know. Seems like about 10 hours of work to get something at least beta quality.

Netflow tools I need to investigate

doug — Sat, 04 Mar 2006 14:20:18 -0800

Flowtools SiLK nfsen nfdump ntop ipacco (not really netflow)

Notes from SPAM guru session

doug — Tue, 13 Dec 2005 11:03:18 -0800

As promised, here are my notes on the [anti] SPAM guru session held Thursday Decmber 7 from 9AM to 10:30AM. This document is member editable in wiki format. Add stuff as needed. Put offsite hyper links in square brackets. Put onsite hyperlinks in square brackets. Any link starting with http: is automatically hyperlinked; in square brackets it's an offsite link.

examples:

[onsite link] [offsite link|http://offsite.link]

technology breakdown

whitelists
blacklists
reputation
bayesian
spf and other sender verification

Principles of volunteerism

doug — Fri, 02 Dec 2005 19:48:26 -0800

I've been thinking a bit about how to foster volunteerism and developing some principles to make it vibrant and healthy. Things like:

Make realistic time committments. Saying that you can commit only 2 hours a week and delivering is better than saying you can commit 8 and delivering 4. In order to effectively divide work the volunteer coordinators need to know accurate time committments to assign resources. Overcommittment can also lead to burnout; we value our volunteers.

2 sets of eyes are better than one. In our jobs, just running something by somebody else can result in quick resolution to problems. It helps to identify the important parts and clear our heads of the irrelevent stuff. We can apply the same principles towards volunteering and project development.

You can do that with a Netra?

doug — Fri, 02 Dec 2005 14:10:25 -0800

Most people don't realize the level of management information that you can get from a Netra (104 or 200 or v120). Sure, they are old and slow by modern standards, but they are extremely rugged and come in AC and DC varieties which is very handy when you are a telco shop or have telco components. The Lights out Management (LOM) functionality is still the best in breed of any vendor, and had capabilities 4 years ago that others are still trying to catch up to today. Rackable is next best and things go down hill from there.

(possibly incomplete) list of components:

Sun Netra SNMP Management Agent. Lots of documentation here. a few mib files too. If you take a look at the entity mib it will look suspiciously like IPMI. I think the IPMI WG used the sun entity mib as a reference model.

Keeping Drafts

doug — Fri, 18 Nov 2005 00:08:42 -0800

Are you borderline (or fully!?) ADD like me? Do you have trouble doing just one thing at a time? Do you write and then keep drafts for later? Well, I don't know about the first two, but I can help with the last one. I added a block on the left of the main blog screen 'My Drafts' that shows up if you have any unpublished documents.

So, let's say you want to come back to an entry later, you can uncheck the 'published' box at the top. It won't show up. Nobody will know it's there! (possibly including you). My Drafts will show you all the unpublished documents that you own so that you can click on them and finish editing and publish, or delete them.