Monitoring and analyzing performance is an important task for any sysadmin. Disk I/O bottlenecks can bring applications to a crawl. What are IOPS? Should I use SATA, SAS, or FC? How many spindles do I need? What RAID level should I use? Is my system read or write heavy? These are common questions for anyone embarking on an disk I/O analysis quest. Obligatory disclaimer: I do not consider myself an expert in storage or anything for that mater. This is just how I have done I/O analysis in the past. I welcome additions and corrections. I believe it’s also important to note that this analysis is geared toward random operations than sequential read/write workloads.

System performance analytics seems to be a frequent question on forums and mailing lists. Finding out why something is slow is generally nontrivial as there are many factors to consider. I have found the sysstat package to be an invaluable tool when looking at system performance. Specifically the command sar gives a wealth of information.

Read the full post Analyzing linux system performance and finding bottle necks

Last time....

In my last post I talked about some of the more common ways to get performance numbers out of your storage: hdparm, dd, and bonnie++.

These tools are pretty good at what they do: measure the performance of a single drive, single LUN, or single filesystem. For many sysadmins, that's all you need to care about. Now we'll look at some more advanced tools for measuring performance in different situations.

Why we need other tools

One project I've been working on recently is determining how much data we can push from a single host out to multiple shelves of disk storage.

Background

In my daily work, I've noticed that not a lot of attention is given to benchmarking various subsystems. When a new disk array comes in, there may be a run or two of bonnie++, but that's generally it. Network benchmarks? Generally I don't see people run them. Maybe you don't care; maybe there's an entire storage team that handles this at your site. However, if you do care, read on for my experiences.

Why benchmark at all?

The answer to question might seem obvious to some, but maybe not. I've often observed people at work getting requirements of X MB/s for a particular

Read an interesting discussion on the lkml list where Linus and friends talked about atime's performance impacts (http://kerneltrap.org/node/14148). Ingo phrased the problem best with:

' For every file that is read from the disk, lets do a ... write to
the disk! And, for every file that is already cached and which we
read from the cache ... do a write to the disk! '

Solutions are to mount your file systems with the noatime, nodiratime options. The only time this may cause a problem is if you have a local mail spool (the mailer will not know that new email arrived) or possibly with some backup software.

At work, every new technology we introduce must go through engineering and financial justification and be approved by a standards governance body. This is to help corral the proliferation of technology choices that we already have throughout the world.

My most recent major technology introduction is Solaris Containers for virtualization. Awhile back the Windows team introduced VMware for Windows virtualization (and our Linux team has never been happy since that Windows controls VMWare ESX, which runs on a Linux base). It is starting to gain traction and now there is some demand for Unix virtualization.