The Linux Sysadmin's Toolkit

If you're an admin for Linux servers that are going to be doing any real kind of work, you'll need to know how to make sure they're running right. You need to understand how the CPU, memory and disk get utilised by the OS, and to do that you need to know how to use a few essential tools and how to interpret the results.

I'll try and write this so admins coming from a Windows background can understand how Linux works compared to Windows.

CPU

There are 3 things you need to be concerned about with regards to how the system is performing from a processor point of view.

1. CPU utilisation percentage
2. CPU run queue (load)
3. CPU I/O wait

In Windows, you mostly are just concerned with CPU utilisation from just a single percentage figure with the maximum being 100%. This isn't really the whole story though.

In Linux, if you have 4 cores in total, the CPU utilisation will be shown as a percentage with the maximum 400%. That may seem strange to someone used to seeing 100% as the maximum but it actually makes more sense to add up the totals of each core and show you all the cores together.

The thing to understand about this is that CPU utilisation isn't actually how busy your system is. It's a part of it, but not the whole story. It's simply a representation of how long the CPU was seen as being busy over a time period. If the system looks at a CPU core for 10ms, and that core was busy for 2ms, it will be 20% busy. It will then sample the other 3 cores, and add those to the total. If they were also all busy for 2ms out of that 10, the total CPU utilisation of the system will be 80%, with the maximum being 400%.

We have a percentage of how busy the CPU is, why isn't that the whole story?

Well, if a CPU core is used by 1 process for 2ms out of 10ms, but for those 2ms there are also 5 other processes waiting to jump on that core and do stuff, a utilisation of 20% isn't really accurate is it? Because for those 2ms, the system is actually trying to do 5 times more than it actually can.

When you understand that both the CPU utilisation _and_ the CPU load are factors to be taken in conjunction with each other, you can interpret what the tools tell you.

top

top - 16:49:48 up 14 days,  6:18,  5 users,  load average: 2.75, 3.64, 3.87

Tasks: 315 total,   1 running, 314 sleeping,   0 stopped,   0 zombie

Cpu(s):  8.3%us,  1.6%sy,  0.0%ni, 86.4%id,  3.1%wa,  0.1%hi,  0.5%si,  0.0%st

Mem:  98871212k total, 81501412k used, 17369800k free,    50108k buffers

Swap:  9446212k total,    32700k used,  9413512k free,  7281528k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                                                                                                           

32031 mysql    -10   0 71.4g  69g 6132 S   97 74.2  13958:45 mysqld                                                                                                                                                                                                         

28358 root      20   0 44176  15m 1280 D   63  0.0 210:01.71 mysqlbackup                                                                                                                                                                                                       

19749 root      20   0 69624  18m 3160 S    7  0.0   3:02.40 iotop                                                                                                                                                                                                             

 6183 root      RT   0  161m  37m  17m S    5  0.0   1188:46 aisexec                                                                                                                                                                                                           

 5397 root      39  19     0    0    0 S    1  0.0 241:39.12 kipmi0                                                                                                                                                                                                         

 2971 root      15  -5     0    0    0 S    0  0.0  65:52.74 kjournald                                                                                                                                                                                                         

    1 root      20   0  1064  392  324 S    0  0.0   0:16.52 init 

top is the standard age-old tool for quickly looking at what's going on. The system above has 8 cores, which are hyperthreaded, so I know that it has 16 logical processors available (generally found out from cat /proc/cpuinfo). When I look at the processes, the mysqld process is taking 97%, but that's from a maximum of _around_ 1600%.

Then, as I said above, we can also look at the system load, which is represented as load average. In the output above, I can see that the first figure of 2.75 is the average over the last 1 minute, 3.64 over the last 5 minutes and 3.87 over the last 15 minutes.

What do these figures mean?

While the system was sampling how much was running on each usable CPU core, it also looked at how many processes were waiting to run. Out of 16 queues, around 3 were waiting at any time, 1 process has taken 97% of 1600%, and another 63%. Therefore, actually, what looked like the system was fairly busy, really has a lot of room to get busier. Until we're consistently filling almost all of the queues (16 on this system), and the CPU utilisation is getting nearer a total of 1600%, we don't need to worry.

The following is a Munin graph of the same system. We can see that the Max idle is 1600, and we're nowhere near it.

And this graph shows the load average

Again, it backs up what we saw from top. We don't have to worry about the load on this system, and we know this by combining the utilisation and load average to see what's really going on.

But what about IO?

A 3rd variable comes into the mix which complicates it a little further, which is IO wait. If a process is running on a CPU core, but you have a slow IO subsystem (e.g. a slow disk, or a saturated fibre channel host bus adapter), the process can be waiting for an IO request to complete. This in turn increases the CPU utilisation and the load average.

If you're seeing high CPU usage and need to find out why, you can see if it's IO wait by using vmstat.

These figures are from a web server. You can see that the io column has no blocks in and a few blocks out now and again. The blocks out are likely to be log files being written, and as it's a web server, everything is already in memory and doesn't need to be read in. No IO issues here.

procs -----------memory---------- ---swap-- -----io---- -system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 3  0    100 266688 302404 5135804    0    0     0     0 17822 24008 15  1 84  0  0
15  0    100 266532 302404 5135820    0    0     0   124 16510 24104 12  1 87  0  0
 0  0    100 265504 302404 5135848    0    0     0     0 18332 24488 17  2 82  0  0
 4  0    100 264312 302404 5135852    0    0     0     0 16986 23787 14  2 84  0  0
 6  0    100 265476 302404 5135864    0    0     0   344 16711 23948 15  1 83  1  0

This one is from a database server. You can see that the blocks in and blocks out (1 block is 1KB) is a lot larger, and as I ran this as vmstat 1 it's cycling every 1 second, so it was reading 30-50MB/s and writing 10-20MB/s.

procs -----------memory---------- ---swap-- -----io---- -system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 3  0     33    879     57  24340    0    0 36108 15304 27169 42413 10  3 83  4  0
 2  1     33    852     57  24384    0    0 36576 15762 26833 40486  9  3 85  4  0
 2  0     33    780     57  24439    0    0 47296  9735 21587 33633  8  2 85  4  0
 0  1     33    721     57  24499    0    0 49496 19881 22993 36320  8  3 86  4  0
 4  0     33    683     57  24547    0    0 42176 13993 23573 36176  8  2 87  2  0
 5  2     33    632     57  24595    0    0 38748 10611 26785 41753 11  3 76 10  0
 4  0     33    584     57  24636    0    0 37636 12618 23149 36298 14  2 80  4  0
 6  0     33    551     57  24685    0    0 34060 13504 25268 39642 14  2 79  5  0
 3  0     33    481     57  24739    0    0 50360 10973 24150 37552 13  2 82  3  0

That's a lot of throughput. Is it affecting the CPU by waiting on IO? Well, the 'wa' column in 'cpu' are figures in a percentage of 100%, so the single digit figures compared to the 'id' (idle) column, it's not waiting on IO for very long at all. Therefore, this server is heavily utilised for IO, but it's not affecting CPU utilisation or system load due to having decent IO.

IO is a bit easier to see by using iostat, which gives you % utilised of your IO subsystem.

# iostat -x -d 1

Linux 2.6.27.29-0.1-default (xxxxxxx)      09/19/12        _x86_64_

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util

sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00

sda1              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00

sda2              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00

sdb               0.00     0.00  349.00   71.00 92320.00 21136.00   270.13     0.99    2.35   1.04  43.60

sdc               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00

dm-0            145.00  1795.00  349.00   71.00 92320.00 21136.00   270.13     1.15    2.77   1.07  44.80

dm-1              0.00     0.00  493.00 1857.00 91976.00 14856.00    45.46    30.01   13.64   0.19  44.40

sdd               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00

sde               0.00     0.00    0.00   14.00     0.00  9490.00   677.86     0.20   14.57   2.00   2.80

dm-2              0.00     0.00    0.00   14.00     0.00  9490.00   677.86     0.21   15.14   2.57   3.60

dm-3              0.00     0.00    0.00   14.00     0.00  9490.00   677.86     0.21   15.14   2.57   3.60

Even easier to use is iotop, a layer on top of iostat to make it more like a top style interface.

Finally, on to memory

Memory is really misunderstood in Linux. Unused memory is inefficient. Some people see the below and panic.

# free -m

             total       used       free     shared    buffers     cached

Mem:         96553      93561       2992          0         73      21044

-/+ buffers/cache:      72443      24110
Swap:         9224         31       9192

That's 93GB used of 96GB installed RAM in the server going by the Mem: row.

Wrong. The Linux kernel grabs as much memory as it can, leaving only a small amount unused and then dishes it out to applications which request it. Anything which isn't requested by an application is then utilised for buffers and caches, including the IO buffer. Read the values in the -/+ buffers/cache line. 24GB is free, and 72GB is used by applications. That's obviously still a lot, but this is a database server, and we want to give the database engine as much memory to cache stuff as possible.

Here's another one from a slightly more modest server:


# free -m
             total       used       free     shared    buffers     cached
Mem:           463        397         65          0        134        136
-/+ buffers/cache:        126        336
Swap:          475         10        465

463MB of RAM and only 65MB free?! Nope, 336GB free as the kernel hasn't needed to dish it out and has allocated it to buffers and caches.

Returning to Windows: Part I

This series will run through how I've moved from being a full-time Linux desktop user to using Windows full-time. First up, a little about me.

I've been in the IT industry professionally for about 15 years, but I've always had something to do with computers. I started at around 7 learning BASIC and writing a few programs on my Sony Hit-Bit MSX.

(MSX is a whole other topic for another post, so I won't go into detail here.) 

After that I worked in Technical Support for a UK computer manufacturer before moving into R&D and finally into Linux system administration. During my time in R&D I had a lot to do with Microsoft and Windows in particular, developing PC builds and configurations around Windows from ME to Media Centre. I got to know them very well and didn't like the way either Microsoft, or Windows, worked.

So I changed my home desktop computers to Linux. Mandrake Linux at first as it was extremely user-friendly and attractive. It may look a little dated now, but against Windows 98 it was amazing.

I then moved onto Ubuntu as of Breezy Badger (5.10, released April 2005) and continued to make my protest against the Microsoft wheel corruption racket that I'd experienced when dealing with them.

So fast forward until now. Ubuntu was fantastic at the start, it promised so much, but as of Natty Narwhal (11.04) it's delivered so little. When I first started I always needed a decent video editor for my family videos. KDEnlive, Cinellera and later PiTiVi were video editors which were always halfway there, threatening to become the all-purpose easy editing suite that Windows Movie Maker had become. But 5 years later, it hasn't happened for one reason or another. A few weeks ago I simply couldn't hold out any longer, I installed Windows 7 on my main computer, stopped being a martyr and took the easy life again. My experiences since then have been mixed, but now I'm in the position where I can provide the fairly rare insight of an experienced Linux user discovering the pitfalls of being a newbie Windows user. In all honesty, I know what to expect, it's not that different to where I left it, but I still have a fresh view on most of it.

The next part of this series will be how the installation differed to what I'm used to. How easy is Windows 7 to set up and get ready to use compared to Ubuntu?

Time For Ubuntu to Fork Evolution

No one can deny the current face of Linux to the masses is Ubuntu. It’s massively more popular than any other distro which makes it the flagship for breaking existing market strangleholds.

Take the Enterprise server OS market for instance, a traditionally strong area for Linux anyway, Canonical (the controlling company of Ubuntu) have rightly seen where they need to position themselves to gain the advantage with Server OS’s and have gone down the Cloud route with Ubuntu Enterprise Cloud. Also - beefing up the support options and the packaging to at least align themselves with the normal market leaders Suse and Red Hat helps to gain further server adoption by to using the momentum of all the other Ubuntu areas and user allegiances.

The personal desktop / netbook area is the next to be tackled. Obviously Ubuntu has been trundling along as the best choice for the tiny personal Linux desktop market for a while but it has needed to really stand-out to do battle with Windows and the latest player (rising on the back of the i[Pod|Phone] wave) Mac OS. Again Canonical have pulled the rabbit out of the hat and pointed Ubuntu desktop in exactly the right direction – Social Networking. With Ubuntu Lucid having fully integrated Social Networking and chat they’ve shown they know how people actually use their computers. 9 times out of 10 someone is turning their computer on to participate in Facebook or make Tweets on Twitter, or for the Old-Skoolers chat on MSN. To make the desktop OS actually part of this is exactly the best way to position it and ensures it’s already ahead of the opposition when they realise they need to do the same thing.

Finally, there’s an area that Ubuntu is very weak on and it’s where efforts need to be concentrated next - The Enterprise Desktop.
Novell have previously tried to leverage that market but did it all wrong. They didn’t understand that there is just one killer feature (just as with integrated desktop social networking) that needs to be in there which is Exchange support. Outlook and to a lesser extent Office keeps Windows XP / 7 firmly planted on the Enterprise desktop purely because of its ability to work perfectly with Exchange. Businesses now (rightly or wrongly) revolve around shared mail, contacts, calendaring and scheduling, and Exchange shows no signs of being supplanted yet as the default choice for this functionaility.

With all this in mind I present my recommendation for Ubuntu: Fork Evolution.

Evolution has some good MAPI functionality but for every step forward, it takes 2 steps back. The functionality is very buggy but at the same time is almost there and some real concerted structured development would see it work very nicely and be a drop in replacement for Outlook.
Also, Evolution needs all of this Social Networking goodness that is present in the me menu in Lucid to be integrated into Evolution too. One place for all messaging/contacts/calendaring with Social Networking in there too and we’re getting very close to a framework that supports the multitude of communication mediums we use today.

So in my mind, Ubuntu and Canonical can move one major step forward by leveraging Evolution. Make it fit the new Ubuntu desktop ethos better and make it work properly with Exchange. Once you do that, world domination for FOSS will follow.

The Future of Web Services / The Plight of the Infrastructure Techie

The landscape for Internet Infrastructure is changing, and it may be quite scary for techies..

Currently, we have quite a supply-chain in web service hosting. We do everything ourselves and that means lots of skill-sets in various places such as:

Linux/Windows SysAdmin
Networking SysAdmin
HA/Load-Balancing Specialists
DBAs
Developers
Management

Typically, management instruct developers to do something and this something needs to be consulted on from conception to delivery. The developer needs to make it in such a way that the DBA and SysAdmins are happy that it will scale and perform. If it’s business critical the HA people get involved and ensure development is geared around being resilient too. The whole thing creates an infrastructure ecosystem of staff because it’s all very DIY so lots of diverse skills are required.

What happens when all of that is someone elses problem, without the disadvantage of outsourcing costs?

Using Cloud Computing you can build web applications to be self-aware. All of the nasty stuff that you used to have to worry about is gone:

Scaling – Amazon Elastic Compute lets you boot loads of instances of your pre-configured application OS images (read-only) manually or automatically. Because the booting of extra instances is web service controlled your app can decide if it needs more power or not.

Load-balancing – Single IP endpoints now control this. No more multiple servers with non-ARPing interfaces and virtual IPs. Amazon Elastic Load Balancing distributes load depending on instance issues, load metrics etc. OR you can plug it into Amazon Cloudwatch which monitors certain metrics and load-balance depending on that.

Hardware – It’s all virtual innit? It’s no longer our problem.

Database Scaling and Capacity – Apps just abstract DB API’s these days anyway. All apps ever do is ask the connector to pull data out into an object or insert data in, or do stuff with the data that’s there. If all you ever see is an interface into a limitless space of data then all you have to worry about is what you do with the data. Amazon SimpleDB does this for you.

Shared data storage – Amazon Elastic Block Storage behave like a SAN where you can have data sat on a filesystem to be accessed by apps on multiple instances. Again, any problems with this are someone else’s. All I care about is accessing my data.

Data mining – If you have lots of data to interrogate you can use Amazon MapReduce to process it.

There are lots of other ways you can use elastic cloud computing to remove a LOT of current infrastructure costs and concerns, eventually leading you to just have to worry about the application. Once you get to that stage you can just concentrate on making it work and doing it right. For a business, it’s a non-decision – or at least it should be. Without all the associated costs of infrastructure and infrastructure people a business can save obscene amounts of money and be a lot more efficient to boot. So why aren’t they?

Well, businesses are just scared of anything new. Elastic computing has to continue to innovate and provide something all-in-one and pre-packaged that it makes a lot more sense (not just a bit more sense) to migrate. Also business don’t like the idea of their data being somewhere they can’t control. That mindset just has to change, and it will as more and more large enterprises get on the cloud application bandwagon and start using tools such as Google Calendar / Mail / Contacts for company business. SLA’s give some peace of mind to directors but not quite yet enough.

But once the scales start tipping, the only people safe are developers.

Printing, And How It's Like Cars

Printers are like the car industry. Stay with me. I'll explain.

Back in the day, printers were tough as old boots. They supported either Postscript or PCL. That was fine, there were 2 standards and most printers supported either.

Then, along came Microsoft with WDM and the ability to use drivers to offload the printers processing into software. Suddenly people like Epson could build cheap mass-produced shit like the Stylus C20 USB printers etc. that didn't have a hope of supporting a standard like PCL or PS. 1-0 to Microsoft, as with their WDM drivers for modems, they played their market advantage to ensure that only people running Windows could use software printers (winprinters).

HP were caught a little on the hop. They suddenly had to play catchup and started doing this winprinter rubbish too. They did, but at the expense of their quality and their reputation in the marketplace. Eventually HP came back around to supporting PCL and Postscript in almost all of their printers and now they're back on top.

Compare to the car industry in the UK - Ford and Vauxhall have always been the mainstream cheap brands, they got good at shaving costs without compromising too much in reliability. Suddenly people like VAG, BMW etc. had to lower prices to compete but they weren't used to doing things on the cheap so the stuff they produced was often poor quality in the late 90's / early 2000's and the Ford and Vauxhall stuff was actually more reliable. It's only today that the higher marques have been able to compete on price and also quality with the cheaper stuff.

'til next time.

Apple are scared of Linux

It's the only explanation I can think of when trying to work out why, after many years of Linux gaining ordinary user popularity, Apple continue to refuse to allow Linux users to use iPods.

I'm a Linux user. I can't afford a Mac (and TBH don't really see the need, it's Linux with all the disadvantages but none of the advantages) and I don't want Windows. I use Kubuntu on my work laptop, Kubuntu on my home desktop and Ubuntu on my old home laptop. If I got given an iPod, or I had an iPhone I couldn't register it. Sure I could take it somewhere with a Windows PC and plug it in to register it but that's not the easiest task for me these days. I don't really have access to any Windows PCs where I can use the Internet and plug random things into them.

After that how do I sync my music? You have to use iTunes and there's no Linux version, even though Mac OS X is based on Unix and Apple obviously had to spend a lot of time and money on making a Windows version. Surely it wouldn't be that difficult to make a Linux version of iTunes?

The answer to all this is simple - Apple see Linux and probably more specifically Ubuntu as a direct competitor to the traction Mac OS X is getting against Windows. Windows is dropping market share and currently Apple is using all the marketing it can to ensure it picks up the slack. With Ubuntu available for free on standard PC hardware and with arguably more power Apple should be worried. But while Apple continue to take this stance against Linux I'm afraid I can't use any of their products. They exclude me. Android doesn't...