Linux Performance Monitoring and Tuning Introduction

by Ramesh Natarajan on March 8, 2011

This is the 1st article in our new series on Linux performance monitoring and tuning.

Linux system administrators should be proficient in Linux performance monitoring and tuning. This article gives a high level overview on how we should approach performance monitoring and tuning in Linux, and the various subsystems (and performance metrics) that needs to be monitored.

To identify system bottlenecks and come up with solutions to fix it, you should understand how various components of Linux works. For example, how the kernel gives preference to one Linux process over others using nice values, how I/O interrupts are handled, how the memory management works, how the Linux file system works, how the network layer is implemented in Linux, etc.,

Please note that understanding how various components (or subsystems) works is not the same as knowing what command to execute to get certain output. For example, you might know that “uptime” or “top” command gives the “load average”. But, if you don’t know what it means, and how the CPU (or process) subsystem works, you might not be able to understand it properly. Understanding the subsystems is an on-going task, which you’ll be constantly learning all the time.

On a very high level, following are the four subsystems that needs to be monitored.

  • CPU
  • Memory
  • I/O
  • Network

1. CPU

You should understand the four critical performance metrics for CPU — context switch, run queue, cpu utilization, and load average.

Context Switch

  • When CPU switches from one process (or thread) to another, it is called as context switch.
  • When a process switch happens, kernel stores the current state of the CPU (of a process or thread) in the memory.
  • Kernel also retrieves the previously stored state (of a process or thread) from the memory and puts it in the CPU.
  • Context switching is very essential for multitasking of the CPU.
  • However, a higher level of context switching can cause performance issues.

Run Queue

  • Run queue indicates the total number of active processes in the current queue for CPU.
  • When CPU is ready to execute a process, it picks it up from the run queue based on the priority of the process.
  • Please note that processes that are in sleep state, or i/o wait state are not in the run queue.
  • So, a higher number of processes in the run queue can cause performance issues.

Cpu Utilization

  • This indicates how much of the CPU is currently getting used.
  • This is fairly straight forward, and you can view the CPU utilization from the top command.
  • 100% CPU utilization means the system is fully loaded.
  • So, a higher %age of CPU utilization will cause performance issues.

Load Average

  • This indicates the average CPU load over a specific time period.
  • On Linux, load average is displayed for the last 1 minute, 5 minutes, and 15 minutes. This is helpful to see whether the overall load on the system is going up or down.
  • For example, a load average of “0.75 1.70 2.10″ indicates that the load on the system is coming down. 0.75 is the load average in the last 1 minute. 1.70 is the load average in the last 5 minutes. 2.10 is the load average in the last 15 minutes.
  • Please note that this load average is calculated by combining both the total number of process in the queue, and the total number of processes in the uninterruptable task status.

2. Network

  • A good understanding of TCP/IP concepts is helpful while analyzing any network issues. We’ll discuss more about this in future articles.
  • For network interfaces, you should monitor total number of packets (and bytes) received/sent through the interface, number of packets dropped, etc.,

3. I/O

  • I/O wait is the amount of time CPU is waiting for I/O. If you see consistent high i/o wait on you system, it indicates a problem in the disk subsystem.
  • You should also monitor reads/second, and writes/second. This is measured in blocks. i.e number of blocks read/write per second. These are also referred as bi and bo (block in and block out).
  • tps indicates total transactions per seconds, which is sum of rtps (read transactions per second) and wtps (write transactions per seconds).

4. Memory

  • As you know, RAM is your physical memory. If you have 4GB RAM installed on your system, you have 4GB of physical memory.
  • Virtual memory = Swap space available on the disk + Physical memory. The virtual memory contains both user space and kernel space.
  • Using either 32-bit or 64-bit system makes a big difference in determining how much memory a process can utilize.
  • On a 32-bit system a process can only access a maximum of 4GB virtual memory. On a 64-bit system there is no such limitation.
  • The unused RAM will be used as file system cache by the kernel.
  • The Linux system will swap when it needs more memory. i.e when it needs more memory than the physical memory. When it swaps, it writes the least used memory pages from the physical memory to the swap space on the disk.
  • Lot of swapping can cause performance issues, as the disk is much slower than the physical memory, and it takes time to swap the memory pages from RAM to disk.

All of the above 4 subsystems are interrelated. Just because you see a high reads/second, or writes/second, or I/O wait doesn’t mean the issue is there with the I/O sub-system. It also depends on what the application is doing. In most cases, the performance issue might be caused by the application that is running on the Linux system.

Remember the 80/20 rule — 80% of the performance improvement comes from tuning the application, and the rest 20% comes from tuning the infrastructure components.

There are various tools available to monitor Linux system performance. For example: top, free, ps, iostat, vmstat, mpstat, sar, tcpump, netstat, iozone, etc., We’ll be discussing more about these tools and how to use them in the upcoming articles in this series.

Following is the 4 step approach to identify and solve a performance issue.

  • Step 1 – Understand (and reproduce) the problem: Half of the problem is solved when you clearly understand what the problem is. Before trying to solve the performance issue, first work on clearly defining the problem. The more time you spend on understanding and defining the problem will give you enough details to look for the answers in the right place. If possible, try to reproduce the problem, or at least simulate a situation that you think closely resembles the problem. This will later help you to validate the solution you come up to fix the performance issue.
  • Step 2 – Monitor and collect data: After defining the problem clearly, monitor the system and try to collect as much data as possible on various subsystems. Based on this data, come up list of potential issues.
  • Step 3 – Eliminate and narrow down issues: After having a list of potential issues, dive into each one of them and eliminate any non issues. Narrow it down further to see whether it is an application issue, or an infrastructure issue. Drill down further and narrow it down to a specific component. For example, if it is an infrastructure issue, narrow it down and identify the subsystem that is causing the issue. If it is an I/O subsystem issue, narrow it down to a specific partition, or raid group, or LUN, or disk. Basically, keep drilling down until you put your finger on the root cause of the issue.
  • Step 4 – One change at a time: Once you’ve narrowed down to a small list of potential issues, don’t try to make multiple changes at one time. If you make multiple changes, you wouldn’t know which one fixed the original issue. Multiple changes at one time might also cause new issues, which you’ll be chasing after instead of fixing the original issue. So, make one change at a time, and see if it fixes the original problem.

In the upcoming articles of the performance series, we’ll discuss more about how to monitor and address performance issues on CPU, Memory, I/O and Network subsystem using various Linux performance monitoring tools.


Linux Sysadmin Course Linux provides several powerful administrative tools and utilities which will help you to manage your systems effectively. If you don’t know what these tools are and how to use them, you could be spending lot of time trying to perform even the basic administrative tasks. The focus of this course is to help you understand system administration tools, which will help you to become an effective Linux system administrator.
Get the Linux Sysadmin Course Now!

If you enjoyed this article, you might also like..

  1. 50 Linux Sysadmin Tutorials
  2. 50 Most Frequently Used Linux Commands (With Examples)
  3. Top 25 Best Linux Performance Monitoring and Debugging Tools
  4. Mommy, I found it! – 15 Practical Linux Find Command Examples
  5. Linux 101 Hacks 2nd Edition eBook Linux 101 Hacks Book

Bash 101 Hacks Book Sed and Awk 101 Hacks Book Nagios Core 3 Book Vim 101 Hacks Book

{ 35 comments… read them below or add one }

1 kiranjith March 8, 2011 at 4:58 am

Great piece of article..
Thanks a lot.

2 Navin March 8, 2011 at 5:09 am

Thanks good information on monitoring

3 Vinit Khandagle March 8, 2011 at 7:36 am

Nice and concise article….. I am looking forward for the information on the commands and tools. Way to Go Ramesh…… By the way congrats on the birth of your child…… She going be the Kernel of your life now!!!!!!! :)

4 Ganesh March 8, 2011 at 7:56 am

Such a nice and Interesting topic … Pls keep posting …tks…

5 Shelly March 8, 2011 at 7:58 am

Thanks for the great article, Ramesh. New subscriber here. :)

6 Rob M March 8, 2011 at 8:54 am

Hi Ramesh.

I’ve been a subscriber for a fair while now & would like to compliment you
on the quality of your tutorials. They’re refreshingly ‘to the point’ !!

7 Alok March 8, 2011 at 8:57 am

Very good article. Waiting for rest of the articles on this topic.

8 Dave Gavin March 8, 2011 at 9:30 am

What about PAE kernels ? Aren’t they able to address beyond 4G ?

Thanks for the series, I’ve learned quite a bit following your blog.

9 Morphik March 8, 2011 at 1:41 pm

Very good article!
Waiting for the next parts!

10 shanil March 8, 2011 at 3:32 pm

great piece of information…excellent

11 Phani March 9, 2011 at 8:00 am

Good artical

12 Nigel Griffiths March 9, 2011 at 8:10 am

Excellent “get you started” article – its easy to forget those starting out on the performance tuning journey after you have been doing this for a few years. I hope you are going to include “nmon” in the list of useful commands in the next article in the series as a one stop shop.

13 Steve March 9, 2011 at 9:09 am

The load average also includes network activity; not cpu processes alone. It is possible to have a high load average without dramatic cpu usage.

14 Steve March 9, 2011 at 9:11 am

… The load average also includes disk usage…

15 jameslee March 9, 2011 at 9:54 am

Good article.very useful.Thank u Ramesh..

16 Nagendra March 10, 2011 at 10:00 am

Really very good information and it helps us to increase our knowledge in more details.
Looking forward for next articles.

17 Júlio Hoffimann Mendes March 12, 2011 at 10:55 am

Great serie! I’m anxious to the next articles.

Regards,
Júlio.

18 Jeevan Kumar March 15, 2011 at 3:08 am

Detailed information. No need to refer any other materials on the internet.
Eagerly waiting for more information on this topic.

19 Mark Seger March 16, 2011 at 6:48 am

The first thing that jumped out at me is your comment about a high i/o wait being a bad thing. This is totally false and a huge source for confusion. All this means is that the CPU was busy while I/O was in progress. If you look at any busy nfs server you’ll see a high I/O wait percentage. If you see a process doing a large burst of I/O you’ll see a high burst of I/O wait. If a compute bound job happens to be running at the same time you’ll see a low I/O wait because the CPU is busy doing other things at the same time.

btw – when you talk about performance monitoring, be sure to pay attention to the open source tool collectl, which essentially replaces most other tools, collecting all the data in one place and providing an easy-to-read simple display. Text based of course, but if you really want plots, its companion utility colplot can do that too.

-mark

20 David March 19, 2011 at 9:50 pm

Very good article.
However in first 2 paragraphs (and some other places) CPU and kernel have been used interchangeably , which could be confusing to people new to operating system. Please update it, if you like.

21 David March 19, 2011 at 10:06 pm

Mark, l think Ramesh’s comment about I/O is not confusing at all, but your’s is.
In general, CPU might be waiting for I/O to finish, which is a typical case for busy waiting, how ever if CPU is doing DMA or any other interrupt based data transfer then CPU is free to perform any other task, and should not be considered as busy. However, some I/O driver (e.g. some disk driver) does do busy wait while performing I/O, because there wait period is much smaller (if we compare this with interrupt service and rescheduling the ongoing task).

22 rohan March 22, 2011 at 1:14 am

Eagerly waiting for next series of yours.

23 Cy March 23, 2011 at 3:52 am

Lookin forward for the next article! Great job..!

/Cy

24 JCIP April 27, 2011 at 12:24 pm

Thanks for the explanation, I have followed your blog for some time and I have become a recurring reader, thanks and hope you continue so for long.

25 jameslee May 2, 2011 at 8:09 pm

Thanks..it is nice to know these concepts.thank you for your valuable informations..

26 arka misra July 29, 2011 at 10:47 am

when will the nest post come??

27 intic_intic July 5, 2012 at 5:53 am

Great article… Thanks

28 Joe W August 15, 2012 at 1:44 pm

As part of my job, I am required to analyze performance of Unix, linux, and Windows server infrastructure. I could not agree more with this entire article, especially the 80/20 rule. It is one thing to see that the system has memory pressure, or CPU pressure. It is another thing entirely to figure out how to fix it without having to add physical memory or processors.

29 Bashir Ahamad September 17, 2012 at 5:00 am

Great Article..at really works..! Thanks..

30 Amy December 24, 2012 at 4:28 pm

Thank you so much for writing and posting this article, Ramesh. A great overview for newbies in performance tuning.

31 irshad February 16, 2013 at 4:08 am

Hi Ramesh,
Good one for Beginners:) keep up the good work

32 Raj April 3, 2013 at 6:02 pm

Awesome article! Thanks a lot:)

33 Selvam August 15, 2013 at 8:54 pm

Hi Ramesh,

Really Great article!

Thanks a lot for your good work.

34 vinod kumar September 3, 2013 at 10:16 am

Thanks a lot

35 kiran January 13, 2014 at 8:39 am

linux [cpu+memory+i/o+n/w] performance tuning & monitoring nice article

Leave a Comment

Previous post:

Next post: