<<
Need to monitor Linux server system performance? Try these built-in or bundled tools. Most Linux distributions come equipped with a wealth of monitoring tools. These tools provide metrics that can be used to obtain relevant information and system activity. You can use these tools to discover possible causes of performance issues. The tools discussed here are just some of the most fundamental ones for analyzing and debugging server problems.
1. Identify Bottlenecks
2. Hard Disk (Storage) Bottlenecks
3. CPU and Memory Bottlenecks
4. Network Bottlenecks
#1: top – Process Activity
top provides a real-time, dynamic view of a currently running system, i.e., running processes. By default, it displays the tasks with the highest CPU usage on the system and refreshes every 5 seconds.
Figure 01. Linux top command
Common Hot Keys
| Hot Key | Usage |
|---|---|
| t | Toggles display of summary information. |
| m | Toggles display of memory information. |
| A | Sorts display by different system resource heavyweights. Helps quickly identify resource-hungry tasks on the system. |
| f | Adds or removes fields to be displayed. |
| o | Adjusts the order of displayed fields. |
| r | Adjusts the Nice value of a running process. |
| k | Kills a running process. |
| z | Toggles color/monochrome display. |
#2: vmstat – System Activity, Hardware & System Information
Use the vmstat command to get information about processes, memory, paging, block IO, traps, and CPU activity.
# vmstat 3
Sample output:
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 0 0 2540988 522188 5130400 0 0 2 32 4 2 4 1 96 0 0
1 0 0 2540988 522188 5130400 0 0 0 720 1199 665 1 0 99 0 0
0 0 0 2540956 522188 5130400 0 0 0 0 1151 1569 4 1 95 0 0
0 0 0 2540956 522188 5130500 0 0 0 6 1117 439 1 0 99 0 0
0 0 0 2540940 522188 5130512 0 0 0 536 1189 932 1 0 98 0 0
0 0 0 2538444 522188 5130588 0 0 0 0 1187 1417 4 1 96 0 0
0 0 0 2490060 522188 5130640 0 0 0 18 1253 1123 5 1 94 0 0
Display detailed memory usage
# vmstat -m
Display active/inactive memory information
# vmstat -a
#3: w – Shows Who Is Logged On and What They Are Doing
The w command displays information about the users currently on the system and their running processes.
# w username
# w vivek
Sample output:
17:58:47 up 5 days, 20:28, 2 users, load average: 0.36, 0.26, 0.24
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
root pts/0 10.1.3.145 14:55 5.00s 0.04s 0.02s vim /etc/resolv.conf
root pts/1 10.1.3.145 17:43 0.00s 0.03s 0.00s w
#4: uptime – Tells How Long the System Has Been Running
The uptime command used to only show how long the system has been running. Now, it can display how long the system has been running, how many users are currently logged on, and the system load averages for the past 1, 5, and 15 minutes.
# uptime
Sample input:
18:02:41 up 41 days, 23:42, 1 user, load average: 0.00, 0.00, 0.00
1 can be considered the optimal load value. The load will vary from system to system. A load of 1-3 for a single-CPU system and 6-10 for an SMP system might be acceptable.
#5: ps – Displays Processes
The ps command displays a snapshot of currently running processes. Use -A or -e to display all processes.
# ps -A
Sample output:
PID TTY TIME CMD
1 ? 00:00:02 init
2 ? 00:00:02 migration/0
3 ? 00:00:01 ksoftirqd/0
4 ? 00:00:00 watchdog/0
5 ? 00:00:00 migration/1
6 ? 00:00:15 ksoftirqd/1
….
…..
4881 ? 00:53:28 java
4885 tty1 00:00:00 mingetty
4886 tty2 00:00:00 mingetty
4887 tty3 00:00:00 mingetty
4888 tty4 00:00:00 mingetty
4891 tty5 00:00:00 mingetty
4892 tty6 00:00:00 mingetty
4893 ttyS1 00:00:00 agetty
12853 ? 00:00:00 cifsoplockd
12854 ? 00:00:00 cifsdnotifyd
14231 ? 00:10:34 lighttpd
14232 ? 00:00:00 php-cgi
54981 pts/0 00:00:00 vim
55465 ? 00:00:00 php-cgi
55546 ? 00:00:00 bind9-snmp-stat
55704 pts/1 00:00:00 ps
ps is very similar to top, but ps provides more information.
Output in long format
# ps -Al
Output with extra full format (shows arguments passed to the process during execution)
# ps -AlF
Display process hierarchy
# ps -AlFH
Show threads after processes
# ps -AlLm
Print all processes on the server
# ps ax
# ps axu
Print process tree
# ps -ejH
# ps axjf
# pstree
Print security information
# ps -eo euser,ruser,suser,fuser,f,comm,label
# ps axZ
# ps -eM
View processes run under the username Vivek
# ps -U vivek -u vivek u
Set custom output format
# ps -eo pid,tid,class,rtprio,ni,pri,psr,pcpu,stat,wchan:14,comm
# ps axo stat,euid,ruid,tty,tpgid,sess,pgrp,ppid,pid,pcpu,comm
# ps -eopid,tt,user,fname,tmout,f,wchan
Display only the process ID of Lighttpd
# ps -C lighttpd -o pid=
Or
# pgrep lighttpd
Or
# pgrep -u vivek php-cgi
Display the process name for PID 55977
# ps -p 55977 -o comm=
Find the top 10 memory-consuming processes
# ps -auxf | sort -nr -k 4 | head -10
Find the top 10 CPU-consuming processes
# ps -auxf | sort -nr -k 3 | head -10
#6: free – Memory Usage
The free command displays the amount of free and used physical and swap memory in the system, as well as the buffers used by the kernel.
# free
Sample output:
total used free shared buffers cached
Mem: 12302896 9739664 2563232 0 523124 5154740
-/+ buffers/cache: 4061800 8241096
Swap: 1052248 0 1052248
#7: iostat – Average CPU Load, Hard Disk Activity
The iostat command reports Central Processing Unit (CPU) statistics and input/output statistics for devices, partitions, and network filesystems.
# iostat
Sample output:
Linux 2.6.18-128.1.14.el5 (www03.nixcraft.in) 06/26/2009
avg-cpu: %user %nice %system %iowait %steal %idle
3.50 0.09 0.51 0.03 0.00 95.86
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 22.04 31.88<<
sp 0.00 0.38 0.03 0.00 97.52
06:45:24 PM all 0.94 0.00 0.28 0.00 0.00 98.78
06:45:28 PM all 1.56 0.00 0.22 0.00 0.00 98.22
06:45:32 PM all 3.53 0.00 0.25 0.03 0.00 96.19
Average: all 2.02 0.00 0.27 0.01 0.00 97.70
#9: mpstat – Multi-Processor Usage
The mpstat command displays the usage of all available processors, with processor numbers starting from 0. mpstat -P ALL shows the average usage for each processor.
# mpstat -P ALL
Sample output:
Linux 2.6.18-128.1.14.el5 (www03.nixcraft.in) 06/26/2009
06:48:11 PM CPU %user %nice %sys %iowait %irq %soft %steal %idle intr/s
06:48:11 PM all 3.50 0.09 0.34 0.03 0.01 0.17 0.00 95.86 1218.04
06:48:11 PM 0 3.44 0.08 0.31 0.02 0.00 0.12 0.00 96.04 1000.31
06:48:11 PM 1 3.10 0.08 0.32 0.09 0.02 0.11 0.00 96.28 34.93
06:48:11 PM 2 4.16 0.11 0.36 0.02 0.00 0.11 0.00 95.25 0.00
06:48:11 PM 3 3.77 0.11 0.38 0.03 0.01 0.24 0.00 95.46 44.80
06:48:11 PM 4 2.96 0.07 0.29 0.04 0.02 0.10 0.00 96.52 25.91
06:48:11 PM 5 3.26 0.08 0.28 0.03 0.01 0.10 0.00 96.23 14.98
06:48:11 PM 6 4.00 0.10 0.34 0.01 0.00 0.13 0.00 95.42 3.75
06:48:11 PM 7 3.30 0.11 0.39 0.03 0.01 0.46 0.00 95.69 76.89
#10: pmap – Process Memory Usage
The pmap command displays the memory map of a process. Use this command to identify the causes of memory bottlenecks.
# pmap -d PID
Displays memory information for the process with PID 47394.
# pmap -d 47394
Sample output:
47394: /usr/bin/php-cgi
Address Kbytes Mode Offset Device Mapping
0000000000400000 2584 r-x– 0000000000000000 008:00002 php-cgi
0000000000886000 140 rw— 0000000000286000 008:00002 php-cgi
00000000008a9000 52 rw— 00000000008a9000 000:00000 [ anon ]
0000000000aa8000 76 rw— 00000000002a8000 008:00002 php-cgi
000000000f678000 1980 rw— 000000000f678000 000:00000 [ anon ]
000000314a600000 112 r-x– 0000000000000000 008:00002 ld-2.5.so
000000314a81b000 4 r—- 000000000001b000 008:00002 ld-2.5.so
000000314a81c000 4 rw— 000000000001c000 008:00002 ld-2.5.so
000000314aa00000 1328 r-x– 0000000000000000 008:00002 libc-2.5.so
000000314ab4c000 2048 —– 000000000014c000 008:00002 libc-2.5.so
…..
……
..
00002af8d48fd000 4 rw— 0000000000006000 008:00002 xsl.so
00002af8d490c000 40 r-x– 0000000000000000 008:00002 libnss_files-2.5.so
00002af8d4916000 2044 —– 000000000000a000 008:00002 libnss_files-2.5.so
00002af8d4b15000 4 r—- 0000000000009000 008:00002 libnss_files-2.5.so
00002af8d4b16000 4 rw— 000000000000a000 008:00002 libnss_files-2.5.so
00002af8d4b17000 768000 rw-s- 0000000000000000 000:00009 zero (deleted)
00007fffc95fe000 84 rw— 00007ffffffea000 000:00000 nbsp; 8192 —– 0000000000000000 000:00000 [ anon ]
mapped: 933712K writeable/private: 4304K shared: 768000K
The last line is very important:
* mapped: 933712K — Total size of memory mappings
* writeable/private: 4304K — Size of private address space
* shared: 768000K — Size of shared address space
#11 and #12: netstat and ss — Network-Related Information
netstat displays network connections, routing tables, interface statistics, masquerade connections, and multicast memberships. The ss command shows socket statistics, allowing you to display information similar to netstat. For usage details on ss and netstat, refer to the resources below.
#13: iptraf — Real-Time Network Information
iptraf is an interactive IP network monitoring tool. It generates various network statistics including: TCP information, UDP counts, ICMP and OSPF information, Ethernet load information, node status, IP checksum errors, and more. It provides the following information formats:
- TCP connection traffic by network
- IP traffic by network interface
- Network traffic by protocol
- Network traffic by TCP/UDP port and packet size
- Network traffic by Layer 2 address
Figure 02: General Interface Statistics: IP Traffic by Network Interface
Figure 03: TCP Connection Traffic by Network
#14: tcpdump — Detailed Network Traffic Analysis
tcpdump is a simple network traffic dump tool, but using it effectively requires deep familiarity with the TCP/IP protocol suite. For example, to display DNS-related network traffic, enter:
# tcpdump -i eth1 'udp port 53'
To display all IPv4 HTTP packets going to and from port 80, printing only packets that contain data (e.g., excluding SYN, FIN, and ACK-only packets), enter:
# tcpdump 'tcp port 80 and (((ip[2:2] – ((ip[0]&0xf)<<2)) – ((tcp[12]&0xf0)>>2)) != 0)'
To display all FTP sessions to a specific host, enter:
# tcpdump -i eth1 'dst 202.54.1.5 and (port 21 or 20'
To display all HTTP sessions to 192.168.1.5:
# tcpdump -ni eth0 'dst 192.168.1.5 and tcp and port http'
To browse detailed information in a dump file using Wireshark, enter:
# tcpdump -n -i eth1 -s 0 -w output.txt src or dst port 80
#15: strace — System Calls
Trace system calls and signals, which is extremely useful for debugging web servers and other server applications. Learn how to trace a process and its functions.
#16: /proc File System — Various Kernel Information
Files under the /proc directory provide detailed information about various hardware devices and the kernel. For more details, see Linux kernel /proc documentation. Common /proc examples include:
# cat /proc/cpuinfo
# cat /proc/meminfo
# cat /proc/zoneinfo
# cat /proc/mounts
#17: Nagios — Server and Network Monitoring
Nagios is a very popular system and network monitoring application. You can easily monitor all your hosts, network devices, and services. It sends alerts when failures occur and again when recovery happens. FAN stands for “Fully Automated Nagios.” The goal of FAN is to provide Nagios installation resources to the Nagios community. To make Nagios server installation easier, FAN offers a standard ISO format CD image. This distribution also includes a set of tools to enhance the user experience.
#18: Cacti — Web-Based Monitoring Tool
Cacti is a complete network graphing solution built upon RRDTool’s data storage and graphing capabilities. Cacti provides a fast poller, advanced graph templates, multiple data acquisition methods, and user management features. All of this is wrapped in a user-friendly interface, ensuring it can be deployed in complex networks with hundreds of devices. It provides information on network, CPU, memory, logged-in users, Apache, DNS, and more.
#19: KDE System Guard
KSysguard is a networked system monitoring tool for the KDE desktop. This tool can be run over an SSH session. It offers many features, such as a client/server architecture for monitoring local and remote hosts, with a frontend GUI that uses so-called sensors to retrieve and display information. A sensor can return a simple value or a set of tabular data. For different information types, one or more displays are provided. These displays are organized into multiple worksheets that can be individually saved and loaded. Thus, KSysguard is not just a simple task manager but a powerful tool capable of controlling multiple servers.
Figure 05: KDE System Guard
#20: Gnome System Monitor
System Monitor displays basic system information, monitors system processes, and tracks system resource and file system usage. You can also use System Monitor to monitor and modify system behavior. Although not as powerful as KDE System Guard, the basic information it provides is very useful for beginners.
* Displays various basic information about computer hardware and software.
* Linux kernel version
* GNOME version
* Hardware
* Installed memory
* Processor and its speed
* System status
* Currently available hard disk space
* Processes
* Memory and swap space
* Network usage
* File systems
* All mounted file systems and their basic information
Figure 06: The Gnome System Monitor application
Source: http://hi.baidu.com/imlidapeng/blog/item/76cc8b15bf38265af2de32cc.html