200.1 Measure and Troubleshoot Resource Usage (weight: 6)
Candidates should be able to measure hardware resource and network bandwidth, identify and troubleshoot resource problems.
Key Knowledge Areas:
Measure CPU usage.
Measure memory usage.
Measure disk I/O.
Measure network I/O.
Measure firewalling and routing throughput.
Map client bandwidth usage.
Match / correlate system symptoms with likely problems.
Estimate throughput and identify bottlenecks in a system including networking.
The following is a partial list of the used files, terms and utilities:
iostat
iotop
vmstat
netstat
ss
iptraf
pstree, ps
w
lsof
top
htop
uptime
sar
swap
processes blocked on I/O
blocks in
blocks out
Measuring System Resource Usage
iostat: Report on CPU statistics and input/output statistics for devices and partitions. Provided by the sysstat package. By default, the command generates two reports: CPU and device utilization.
sar: Collect, report, or save system activity information. Provided by the sysstat command. sar creates a binary file for each day of the month with all the information collected (e.g., sa01), as well as a daily summary (e.g.,sar01). These files are overwritten the following month. The sysstat service must be started for sar to start collecting data and the duration of collection can be changed by modifying /etc/cron.d/sysstat.
[root@linuxmaster sa]# cat /etc/cron.d/sysstat # Run system activity accounting tool every 10 minutes*/10 **** root /usr/lib64/sa/sa1 1 1# 0 * * * * root /usr/lib64/sa/sa1 600 6 &# Generate a daily summary of process accounting at 23:535323***root/usr/lib64/sa/sa2-A
free: Display the amount of free and used memory in the system (default is Kb). The information displayed by free is pulled from the /proc/meminfo file. Swap is a substitute for physical memory that uses the hard disk. The /proc/sys/vm/swappiness file determines how aggressively a system will swap.
vmstat: Reports virtual memory statistics as well as statistics on paging and block I/O. By default, vmstat prints a summary of usage statistics since the last reboot. vmstat can run at an interval with a count or continuously. The first report in the interval is the summary report.
lsof: Lists open files on the system. By default, the command will display all open files belonging to all active processes. When multiple options are used, it will display open files that meet any of the criteria (ORing), but the -a option can be used to AND the selections (ANDing).
lsof [OPTIONS] [NAMES]
Option
Purpos
-a
Causes selections to be ANDed
-c
List files based on command entered
-i
List files based on a network connection
+d
List files associated with a directory
+D
List files associated with a directory (includes sub directories)
ps: Report a snapshot of the current processes. By default, ps selects all process with the same effective user ID as the current user and the associated terminal.
ps [OPTIONS]
ps accept several different kinds of options:
UNIX - May be grouped and must be preceded by a dash.
top: Display a dynamic, real-time view of processes running on the system. The program provides a limited interactive interface for process mainpulation, as well as a much more extensive interface for personal configuration.
netstat: Print information on network connections, routing tables, interface statistics, masquerade connections, and multicast memberships. By default, netstat displays a list of open sockets. The netstat command is obsolete and has been replaced by ss. other replacements include: ip route for netstat -r, ip -s link for netstat -i, and ip maddr for netstat -g.
w: Display information about the users currently on the machine, and their processes. The header shows the current time, how long the system has been running, how many users are currently logged on, and the system load averages for the past 1, 5, and 15 minutes. The following entries are displayed for each user: login name, the tty name, the remote host, login time, idle time, JCPU, PCPU, and the command line of their current process.
w [OPTIONS] USER
Option
Purpose
-h
Don't print the header
-s
Use short format
-i
Display IP addresses rather than hostnames
user
Display information for USER specified
[root@linuxmaster ~]# w12:39:35up1:07,1user,loadaverage:0.00,0.01,0.05USERTTYFROMLOGIN@IDLEJCPUPCPUWHATrootpts/088.232.191.13411:317.00s0.32s0.00sw[root@linuxmaster ~]# w -hrootpts/088.232.191.13411:314.00s0.37s0.01sw-h[root@linuxmaster ~]# w -s12:51:12up1:19,1user,loadaverage:0.00,0.01,0.05USERTTYFROMIDLEWHATrootpts/088.232.191.1340.00sw-s
uptime: Display how long the system has been running. By default, displays the same information found in the header of the w command. Shows the current time, how long the system has been running, how many users are currently logged on, and the system load averages for the past 1, 5, and 15 minutes. Both w and uptime get their information from /var/run/utmp.
Candidates should be able to monitor resource usage to predict future resource needs.
\
Key Knowledge Areas:
Use monitoring and measurement tools to monitor IT infrastructure usage.
Predict capacity break point of a configuration.
Observe growth rate of capacity usage.
Graph the trend of capacity usage.
Awareness of monitoring solutions such as Icinga2, Nagios, collectd, MRTG and Cacti
The following is a partial list of the used files, terms and utilities:
diagnose
predict growth
resource exhaustion
Monitoring Solutions
Nagios
Nagios is an industry-standard IT monitoring solution.
Provides complete monitoring for Linux operating systems and distributions
Provides two options for monitoring hosts
Using dedicated Nagios agents
Using SNMP for "agentless" monitoring
Provides advanced graphs and visual aids
Provides automated and integrated capacity planning
Icinga2
Icinga2 is a monitoring solution that reports on various usage statistics by using a primary node, satellites, and agents.
Monitors network resources
Notifies users of outages
Generates performance data
Runs on Linux/Unix and Windows
MRTG
The Multi Router Traffic Grapher is a tool for monitoring routers and other network devices.
Monitors SNMP network devices
Monitors traffic load on network links
Generates HTML pages containing PNG images of live data
Implements a data consolidation algorithm
Provides built-in hooks for using RRDTool
Cacti
A network graphing solution designed to work with RRDTool.
Provides a complete frontend to RRDTool
Uses a MySQL database to store data
Provides a built-in user-based management system
Provides graphs for displaying data
Allows data sources to be defined for data collection
Collectd
Collectd is a service daemon that collects system and applicatkin performance metrics periodically and provides mechanisms to store the values in a variety of ways (e.g. RRD files).
Gathers metrics from various sources
Stores metrics gathered or makes them available over the network
Provides information for performance analysis and capacity planning
Provides over 100 plugins for monitoring and integration with other solutions
iostat: Report on CPU statistics and input/output statistics for devices and partitions.
sar: Collect, report, or save system activity information.
sar creates a binary file for each day of the month with all the information collected (e.g., sa01), as well as a daily summary (e.g.,sar01). These files are overwritten the following month. The sysstat service must be started for sar to start collecting data and the duration of collection can be changed by modifying /etc/cron.d/sysstat.
free: Display the amount of free and used memory in the system.
vmstat: Reports virtual memory statistics as well as statistics on paging and block I/O.
lsof: Lists open files on the system.
ps: Report a snapshot of the current processes.
pstree: Display running processes as a visual tree.
top: Display a dynamic, real-time view of processes running on the system.
netstat: Print information on network connections, routing tables, interface statistics, masquerade connections, and multicast memberships.
w: Display information about the users currently on the machine, and their processes.
uptime: Display how long the system has been running.
Nagios using SNMP for "agentless" monitoring.
Icinga2 originally created as a fork of Nagios.
MRTG provides built-in hooks for using RRDTool.
Cacti a network graphing solution designed to work with RRDTool.
Collectd is a service daemon that collects system and applicatkin performance metrics periodically and provides mechanisms to store the values in a variety of ways (e.g. RRD files).
Sample Questions
1. Which of the following provides a complete frontend for RRDTool?
2. What command displays a dynamic view of processes running on the system?
3. Which of the following was originally created as a fork of Nagios?
4. What file is used to change the duration of collection for sar?
5. What option for lsof lists open files based on network connections?
6. What commands show how long the system has been running?
7. What option for netstat displays the kernel routing table?
8. Which of the following are aspects of capacity planning?
9. What option only generates the device report with the iostat command?
10. What kinds of options are accepted by the ps command? (choose all that apply)
11. Which command reports information on memory usage, paging and block input/output?
12. Which program lists information about files and network connections opened by processes? (Specify ONLY the command without any path or parameters.)
Answer: lsof
13. Which of the following commands by default provides the PIDs of the processes sorted by which are using the most CPU cycles on the Linux system?
14. In the following output of the command uptime, the load averages represent the system load averages for what time frames?
15. In order to gather performance data with sar over a longer period of time, which command should be run periodically from cron?
16. In the output of sar –b, what does tps stand for?
17. Which of the following monitoring commands allow you to view system memory information? (Choose all that apply.)
18. Which of the following will show CPU usage information 20 times in 2-second intervals?
19. Which of the following are considered to be system resource-monitoring solutions? (Choose all that apply)
20. Which of the following are true about Linux system memory? (Choose all that apply)
Answer: B, C. Memory for idle processes can be swapped out from RAM to a special partition called swap space, swap, or virtual memory. Therefore, options B and C are correct. Memory is divided up into chunks called pages (option A), not sheets. Swapping is the memory management technique where idle processes’ memory is swapped out to swap. The processes do not trade memory (option D). Finally, the psmap tool is primarily for mapping processes (option E). The best tools for viewing memory statistics are tools like free, sar, and vmstat.
21. In the following output from top, which processes contribute to the percentage of time that the CPU spends in the state of wa?
23. When is historical data of resource usage important? (Select THREE correct answers.)
24. In capacity planning exercises, which tools assist in listing and identifying processes of interest? (Choose TWO correct answers.)
25. In the below example output, which columns detail the percent of time the CPU spent running non-kernel code and the percent of time the CPU spent running kernel code? (Choose TWO correct answers.)
27. Which of the following tools are used to measure memory usage? (Choose THREE correct answers.)
28. Which of the following is a side effect of extensive usage of swap space?
29. Which of the following commands will provide the PIDs of the processes sorted by which are using the most CPU cycles on the Linux system?
30. Which command will report information on memory usage, paging and block input/output?
31. Using the iostat command you want to display more detailed statistics (Extended), but you want the output of this command to be printed on the screen 5 times between 2-second update intervals. Which of the following is correct to accomplish this task?
32. Which of the commands below will return virtual memory statistics information about processes, memory, pagination, block IO, trap and CPU usage?
33. Which of the commands below is used to monitor the load of system input / output (I/O) devices. Where is this done by observing the devices are active in relation to their average transfer rates?
34. Which of the commands below collects, reports, or saves system activity information?
35. Which of the iostat command options can be used to display device statistics information (disks and partitions)?
36. Which of the iostat options can be used to display a CPU usage report?
37. Which of the iostat command options can be used to display a device usage report (Disks)?
38. Which command below displays information about the users currently connected to the machine, their processes and the same statistics as provided by the uptime command?
39. When outputting the vmstat command, in the procs field, what do the columns r and b mean? (Select 2 responses).
40. Which of the columns below is related to the system field in the output of the vmstat command that shows the number of contexts per second, including the clock?
41. Which of the columns below is related to the system field in the output of the vmstat command that shows the number of interrupts per second, including the clock?
42. Which of the alternatives below are columns related to the swap field presented in the output of the vmstat command? (Select 2 responses).
43. Which of the alternatives below are columns related to the io field presented in the output of the vmstat command? (Select 2 responses).
44. Which of the following, using the vmstat command, will display event counter statistics?
45. Which of the following, using the vmstat command, will display disk statistics?
46. Which of the following, using the vmstat command, will display a summary of disk statistics?
47. Which of the following, using the vmstat command, will display statistics for a specific partition?
48. Which of the free command options will update the command output every 2 seconds for 5 times on the screen? (Select 2 responses).
49. Which of the free command options will update the command output on the screen every 2 seconds?
50. Which of the alternatives below will list detailed memory statistics, showing the total RAM + SWAP memory, printing the result in the output in a more readable (- human) form for the user?
51. Which option in the sar command displays paging statistics?
52. Which option in the sar command displays a memory usage statistics report?
53. Which option of the sar command will report the queue size and load averages?
54. Which option of the sar command will report an I/O transfer rate report?
55. Which option of the sar command will report swap statistics? (Select 2 responses).
56. Which option of the sar command will report the activity of each block device?
57. Which option in the sar command will report the activity of TTY (Remote Terminals) devices?
58. Which of the commands below can be used to view the PID of a program that we have just started? (Select 2 responses).
59. What is the result of the ps ef command?
60. Which of the commands below will return a list of processes that are running on the current terminal?
61. Which command will list all processes, displaying the names of the users who started them?
62. Which command displays details about the processes that are running in real time on the server?
63. Which of the options ("Internal option of the top command") below can be used to change the priority of a process already running?
64. Which of the following will set the update time to 2 seconds on standard output using the top command?
65. Which "top" command will ignore idle processes?
66. Which command will update the top command's standard output 4 times and then exit the program?
67. Which top command will only list the processes of the user root?
