Several performance tools provide memory usage reports. The reports of most interest are from the vmstat, ps, and svmon commands.
The vmstat command summarizes the total active virtual memory used by all of the processes in the system, as well as the number of real-memory page frames on the free list. Active virtual memory is defined as the number of virtual-memory working segment pages that have actually been touched. For a more detailed definition, see Late page space allocation. This number can be larger than the number of real page frames in the machine, because some of the active virtual-memory pages may have been written out to paging space.
When determining if a system might be short on memory or if some memory tuning needs to be done, run the vmstat command over a set interval and examine the pi and po columns on the resulting report. These columns indicate the number of paging space page-ins per second and the number of paging space page-outs per second. If the values are constantly non-zero, there might be a memory bottleneck. Having occasional non-zero values is not be a concern because paging is the main principle of virtual memory.
# vmstat 2 10 kthr memory page faults cpu ----- ----------- ------------------------ ------------ ----------- r b avm fre re pi po fr sr cy in sy cs us sy id wa 1 3 113726 124 0 14 6 151 600 0 521 5533 816 23 13 7 57 0 3 113643 346 0 2 14 208 690 0 585 2201 866 16 9 2 73 0 3 113659 135 0 2 2 108 323 0 516 1563 797 25 7 2 66 0 2 113661 122 0 3 2 120 375 0 527 1622 871 13 7 2 79 0 3 113662 128 0 10 3 134 432 0 644 1434 948 22 7 4 67 1 5 113858 238 0 35 1 146 422 0 599 5103 903 40 16 0 44 0 3 113969 127 0 5 10 153 529 0 565 2006 823 19 8 3 70 0 3 113983 125 0 33 5 153 424 0 559 2165 921 25 8 4 63 0 3 113682 121 0 20 9 154 470 0 608 1569 1007 15 8 0 77 0 4 113701 124 0 3 29 228 635 0 674 1730 1086 18 9 0 73
In the example output above, notice the high I/O wait in the output and also the number of threads on the blocked queue. Other I/O activities might cause I/O wait, but in this particular case, the I/O wait is most likely due to the paging in and out from paging space.
To see if the system has performance problems with its VMM, examine the columns under memory and page:
Provides information about the real and virtual memory.
The Active Virtual Memory, avm, column represents the number of active virtual memory pages present at the time the vmstat sample was collected. The deferred page space policy is the default policy. Under this policy, the value for avm might be higher than the number of paging space pages used. The avm statistics do not include file pages.
The fre column shows the average number of free memory pages. A page is a 4 KB area of real memory. The system maintains a buffer of memory pages, called the free list, that will be readily accessible when the VMM needs space. The minimum number of pages that the VMM keeps on the free list is determined by the minfree parameter of the vmtune command. For more details, see VMM page replacement tuning with the vmtune command.
When an application terminates, all of its working pages are immediately returned to the free list. Its persistent pages, or files, however, remain in RAM and are not added back to the free list until they are stolen by the VMM for other programs. Persistent pages are also freed if the corresponding file is deleted.
For this reason, the fre value may not indicate all the real memory that can be readily available for use by processes. If a page frame is needed, then persistent pages related to terminated applications are among the first to be handed over to another program.
If the fre value is substantially above the maxfree value, it is unlikely that the system is thrashing. Thrashing means that the system is continuously paging in and out. However, if the system is experiencing thrashing, you can be assured that the fre value will be small.
Information about page faults and paging activity. These are averaged over the interval and given in units per second.
The pi column details the number of pages paged in from paging space. Paging space is the part of virtual memory that resides on disk. It is used as an overflow when memory is over committed. Paging space consists of logical volumes dedicated to the storage of working set pages that have been stolen from real memory. When a stolen page is referenced by the process, a page fault occurs, and the page must be read into memory from paging space.
Due to the variety of configurations of hardware, software and applications, there is no absolute number to look out for. This field is important as a key indicator of paging-space activity. If a page-in occurs, there must have been a previous page-out for that page. It is also likely in a memory-constrained environment that each page-in will force a different page to be stolen and, therefore, paged out.
The po column shows the number (rate) of pages paged out to paging space. Whenever a page of working storage is stolen, it is written to paging space, if it does not yet reside in paging space or if it was modified. If not referenced again, it will remain on the paging device until the process terminates or disclaims the space. Subsequent references to addresses contained within the faulted-out pages results in page faults, and the pages are paged in individually by the system. When a process terminates normally, any paging space allocated to that process is freed. If the system is reading in a significant number of persistent pages, you might see an increase in po without corresponding increases in pi. This does not necessarily indicate thrashing, but may warrant investigation into data-access patterns of the applications.
Number of pages that were freed per second by the page-replacement algorithm during the interval. As the VMM page-replacement routine scans the Page Frame Table, or PFT, it uses criteria to select which pages are to be stolen to replenish the free list of available memory frames. The criteria include both kinds of pages, working (computational) and file (persistent) pages. Just because a page has been freed, it does not mean that any I/O has taken place. For example, if a persistent storage (file) page has not been modified, it will not be written back to the disk. If I/O is not necessary, minimal system resources are required to free a page.
Number of pages that were examined per second by the page-replacement algorithm during the interval. The page-replacement algorithm might have to scan many page frames before it can steal enough to satisfy the page-replacement thresholds. The higher the sr value compared to the fr value, the harder it is for the page-replacement algorithm to find eligible pages to steal.
Number of cycles per second of the clock algorithm. The VMM uses a technique known as the clock algorithm to select pages to be replaced. This technique takes advantage of a referenced bit for each page as an indication of what pages have been recently used (referenced). When the page-stealer routine is called, it cycles through the PFT, examining each page's referenced bit.
The cy column shows how many times per second the page-replacement code has scanned the PFT. Because the free list can be replenished without a complete scan of the PFT and because all of the vmstat fields are reported as integers, this field is usually zero.
One way to determine the appropriate amount of RAM for a system is to look at the largest value for avm as reported by the vmstat command. Multiply that by 4 K to get the number of bytes and then compare that to the number of bytes of RAM on the system. Ideally, avm should be smaller than total RAM. If not, some amount of virtual memory paging will occur. How much paging occurs will depend on the difference between the two values. Remember, the idea of virtual memory is that it gives us the capability of addressing more memory than we have (some of the memory is in RAM and the rest is in paging space). But if there is far more virtual memory than real memory, this could cause excessive paging which then results in delays. If avm is lower than RAM, then paging-space paging could be caused by RAM being filled up with file pages. In that case, tuning the minperm,maxperm, and maxclient values could reduce the amount of paging-space paging. Refer to VMM page replacement tuning with the vmo command for more information.
The vmstat -I command displays additional information, such as file pages in per-second, file pages out per-second which means any VMM page-ins and page-outs that are not paging space page-ins or paging space page-outs. The re and cy columns are not reported with this flag.
The summary option, -s, sends a summary report to standard output starting from system initialization expressed in absolute counts rather than on an interval basis. The recommended way of using these statistics is to run this command before a workload, save the output, and then run it again after the workload and save its output. The next step is to determine the difference between the two sets of output. An awk script called vmstatit that does this automatically is provided in Determining whether the problem is related to disk or memory.
# vmstat -s
3231543 total address trans. faults
63623 page ins
383540 page outs
149 paging space page ins
832 paging space page outs
0 total reclaims
807729 zero filled pages faults
4450 executable filled pages faults
429258 pages examined by clock
8 revolutions of the clock hand
175846 pages freed by the clock
18975 backtracks
0 lock misses
40 free frame waits
0 extend XPT waits
16984 pending I/O waits
186443 start I/Os
186443 iodones
141695229 cpu context switches
317690215 device interrupts
0 software interrupts
0 traps
55102397 syscalls
The page-in and page-out numbers in the summary represent virtual memory activity to page in or out pages from page space and file space. The paging space ins and outs are representative of only page space.
The ps command can also be used to monitor memory usage of individual processes. The ps v PID command provides the most comprehensive report on memory-related statistics for an individual process, such as:
The following is an example:
# ps v PID TTY STAT TIME PGIN SIZE RSS LIM TSIZ TRS %CPU %MEM COMMAND 36626 pts/3 A 0:00 0 316 408 32768 51 60 0.0 0.0 ps v
The most important columns on the resulting ps report are described as follows:
The svmon command provides a more in-depth analysis of memory usage. It is more informative, but also more intrusive, than the vmstat and ps commands. The svmon command captures a snapshot of the current state of memory. However, it is not a true snapshot because it runs at the user level with interrupts enabled.
To determine whether svmon is installed and available, run the following command:
# lslpp -lI bos.perf.tools
The svmon command can only be executed by the root user.
If an interval is used, which is the -i option, statistics will be displayed until the command is killed or until the number of intervals, which can be specified right after the interval, is reached.
You can use the following different reports to analyze the displayed information:
To print out global statistics, use the -G flag. In this example, we will repeat it five times at two-second intervals.
# svmon -G -i 2 5
m e m o r y i n u s e p i n p g s p a c e
size inuse free pin work pers clnt work pers clnt size inuse
16384 16250 134 2006 10675 2939 2636 2006 0 0 40960 12674
16384 16254 130 2006 10679 2939 2636 2006 0 0 40960 12676
16384 16254 130 2006 10679 2939 2636 2006 0 0 40960 12676
16384 16254 130 2006 10679 2939 2636 2006 0 0 40960 12676
16384 16254 130 2006 10679 2939 2636 2006 0 0 40960 12676
The columns on the resulting svmon report are described as follows:
In the example, there are 16384 pages of total size of memory. Multiply this number by 4096 to see the total real memory size (64 MB). While 16250 pages are in use, there are 134 pages on the free list and 2006 pages are pinned in RAM. Of the total pages in use, there are 10675 working pages in RAM, 2939 persistent pages in RAM, and 2636 client pages in RAM. The sum of these three parts is equal to the inuse column of the memory part. The pin part divides the pinned memory size into working, persistent and client categories. The sum of them is equal to the pin column of the memory part. There are 40960 pages (160 MB) of total paging space, and 12676 pages are in use. The inuse column of memory is usually greater than the inuse column of pg space because memory for file pages is not freed when a program completes, while paging-space allocation is.
The following output is an example of the svmon command:
# svmon -G -i 2 5
size inuse free pin virtual
memory 65527 64087 1440 5909 81136
pg space 131072 55824
work pers clnt
pin 5918 0 0
in use 47554 13838 2695
size inuse free pin virtual
memory 65527 64091 1436 5909 81137
pg space 131072 55824
work pers clnt
pin 5918 0 0
in use 47558 13838 2695
size inuse free pin virtual
memory 65527 64091 1436 5909 81137
pg space 131072 55824
work pers clnt
pin 5918 0 0
in use 47558 13838 2695
size inuse free pin virtual
memory 65527 64090 1437 5909 81137
pg space 131072 55824
work pers clnt
pin 5918 0 0
in use 47558 13837 2695
size inuse free pin virtual
memory 65527 64168 1359 5912 81206
pg space 131072 55824
work pers clnt
pin 5921 0 0
in use 47636 13837 2695
The additional output field is the virtual field, which shows the number of pages allocated in the system virtual space.
The following command displays the memory usage statistics for the top ten processes. If you do not specify a number, it will display all the processes currently running in the system.
# svmon -Pau 10
Pid Command Inuse Pin Pgspace
15012 maker4X.exe 4783 1174 4781
2750 X 4353 1178 5544
15706 dtwm 3257 1174 4003
17172 dtsession 2986 1174 3827
21150 dtterm 2941 1174 3697
17764 aixterm 2862 1174 3644
2910 dtterm 2813 1174 3705
19334 dtterm 2813 1174 3704
13664 dtterm 2804 1174 3706
17520 aixterm 2801 1174 3619
Pid: 15012
Command: maker4X.exe
Segid Type Description Inuse Pin Pgspace Address Range
1572 pers /dev/hd3:62 0 0 0 0..-1
142 pers /dev/hd3:51 0 0 0 0..-1
1bde pers /dev/hd3:50 0 0 0 0..-1
2c1 pers /dev/hd3:49 1 0 0 0..7
9ab pers /dev/hd2:53289 1 0 0 0..0
404 work kernel extension 27 27 0 0..24580
1d9b work lib data 39 0 23 0..607
909 work shared library text 864 0 7 0..65535
5a3 work sreg[4] 9 0 12 0..32768
1096 work sreg[3] 32 0 32 0..32783
1b9d work private 1057 1 1219 0..1306 : 65307..65535
1af8 clnt 961 0 0 0..1716
0 work kernel 1792 1146 3488 0..32767 : 32768..65535
...
The output is divided into summary and detail sections. The summary section lists the top ten highest memory-usage processes in descending order.
Pid 15012 is the process ID that has the highest memory usage. The Command indicates the command name, in this case maker4X.exe. The Inuse column, which is the total number of pages in real memory from segments that are used by the process, shows 4783 pages. Each page is 4 KB. The Pin column, which is the total number of pages pinned from segments that are used by the process, shows 1174 pages. The Pgspace column, which is the total number of paging-space pages that are used by the process, shows 4781 pages.
The detailed section displays information about each segment for each process that is shown in the summary section. This includes the segment ID, the type of the segment, description (a textual description of the segment, including the volume name and i-node of the file for persistent segments), number of pages in RAM, number of pinned pages in RAM, number of pages in paging space, and address range.
The Address Range specifies one range for a persistent or client segment and two ranges for a working segment. The range for a persistent or a client segment takes the form '0..x,' where x is the maximum number of virtual pages that have been used. The range field for a working segment can be '0..x : y..65535', where 0..x contains global data and grows upward, and y..65535 contains stack area and grows downward. For the address range, in a working segment, space is allocated starting from both ends and working towards the middle. If the working segment is non-private (kernel or shared library), space is allocated differently. In this example, the segment ID 1b9d is a private working segment; its address range is 0..1306 : 65307..65535. The segment ID 909 is a shared library text working segment; its address range is 0..65535.
A segment can be used by multiple processes. Each page in real memory from such a segment is accounted for in the Inuse field for each process using that segment. Thus, the total for Inuse may exceed the total number of pages in real memory. The same is true for the Pgspace and Pin fields. The sum of Inuse, Pin, and Pgspace of all segments of a process is equal to the numbers in the summary section.
The Vsid column is the virtual segment ID, and the Esid column is the effective segment ID. The effective segment ID reflects the segment register that is used to access the corresponding pages.
You can use one of the following commands to display the file name associated with the i-node:
The -D option displays detailed memory-usage statistics for segments.
The following is an example:
# svmon -D 629 -b
Segid: 629
Type: working
Address Range: 0..77
Size of page space allocation: 7 pages ( 0.0 Mb)
Virtual: 11 frames ( 0.0 Mb)
Inuse: 7 frames ( 0.0 Mb)
Page Frame Pin Ref Mod
0 32304 N Y Y
3 32167 N Y Y
7 32321 N Y Y
8 32320 N Y Y
5 32941 N Y Y
1 48357 N N Y
77 47897 N N Y
The explanation of the columns are as follows:
The -b flag shows the status of the reference and modified bits of all the displayed frames. After it is shown, the reference bit of the frame is reset. When used with the -i flag, it detects which frames are accessed between each interval.
The -S option is used to sort segments by memory usage and to display the memory-usage statistics for the top memory-usage segments. The following command sorts system and non-system segments by the number of pages in real memory. The -t option can be used to limit the number of segments displayed to the count specified. The -u flag sorts the output in descending order by the total number of pages in real memory.
The following is example output of the svmon command with the -S, -t, and -u options:
# svmon -Sut 10
Vsid Esid Type Description Inuse Pin Pgsp Virtual Addr Range
1966 - work 9985 4 31892 32234 0..32272 :
65309..65535
14c3 - work 5644 1 161 5993 0..6550 :
65293..65535
5453 - work 3437 1 2971 4187 0..4141 :
65303..65535
4411 - work 3165 0 1264 1315 0..65535
5a1e - work 2986 1 13 2994 0..3036 :
65295..65535
340d - work misc kernel tables 2643 0 993 2645 0..15038 :
63488..65535
380e - work kernel pinned heap 2183 1055 1416 2936 0..65535
0 - work kernel seg 2044 1455 1370 4170 0..32767 :
65475..65535
6afb - pers /dev/notes:92 1522 0 - - 0..10295
2faa - clnt 1189 0 - - 0..2324
There is a correlation between the svmon and vmstat outputs.
The following is example output from the svmon command:
# svmon -G
size inuse free pin virtual
memory 2097136 1347642 749494 156369 205041
pg space 249856 871
work pers clnt lpage
pin 156369 0 0 0
in use 205060 1130522 12060 0
The vmstat command was run in a separate window while the svmon command was running. The vmstat report follows:
# vmstat 5 kthr memory page faults cpu ----- ----------- ------------------------ ------------ ----------- r b avm fre re pi po fr sr cy in sy cs us sy id wa 1 5 205031 749504 0 0 0 0 0 0 1240 248 318 0 0 99 0 0 0 205042 749493 0 0 0 0 0 0 1242 663 319 0 0 99 0 0 0 205042 749493 0 0 0 0 0 0 1244 658 320 0 0 99 0 0 0 205042 749493 0 0 0 0 0 0 1241 665 317 0 0 99 0 0 0 205042 749493 0 0 0 0 0 0 1242 655 318 0 0 99 0 0 0 205042 749493 0 0 0 0 0 0 1242 656 320 0 0 99 0 0 0 205042 749493 0 0 0 0 0 0 1241 654 316 0 0 99 0
The global svmon report shows related numbers. The fre column of the vmstat command relates to the memory fre column of the svmon command. The Active Virtual Memory, avm, value of 205042 that the vmstat command reports is similar to the virtual memory value of 205041 that the svmon command reports.
There are some relationships between the svmon and ps command outputs. The svmon command output is as follows:
# svmon -P 770204
-------------------------------------------------------------------------------
Pid Command Inuse Pin Pgsp Virtual 64-bit Mthrd LPage
770204 -ksh 7930 3273 869 7870 N N N
Vsid Esid Type Description LPage Inuse Pin Pgsp Virtual
0 0 work kernel segment - 5994 3269 869 5994
509aa d work loader segment - 1794 0 0 1794
98fb3 2 work process private - 59 4 0 59
8c21 1 clnt code,/dev/hd2:88 - 58 0 - -
88fd1 f work shared library data - 23 0 0 23
58fcb - clnt /dev/hd4:1158 - 2 0 - -
Compare the above example with the ps report which follows:
# ps v 770204
PID TTY STAT TIME PGIN SIZE RSS LIM TSIZ TRS %CPU %MEM COMMAND
770204 pts/0 A 0:00 0 328 560 xx 202 232 0.0 0.0 -ksh
The SIZE value of 328 correlates to the Virtual value of the svmon command for process private value of 59 plus the shared library data value of 23, which is in 1 KB units. This number is equal to the number of working segment pages of the process that have been touched (that is, the number of paging-space pages that have been allocated) times 4. It must be multiplied by 4 because pages are in 4 K units and SIZE is in 1 K units. If some working segment pages are currently paged out, this number is larger than the amount of real memory being used. The SIZE value (240) correlates with the Pgspace number from the svmon command for private (32) plus lib data (28) in 1 K units.
RSS refers to the real memory (resident set) size in KB of the process. This number is equal to the sum of the number of working segment and code segment pages in memory times 4. Remember that code segment pages are shared among all of the currently running instances of the program. If 26 ksh processes are running, only one copy of any given page of the ksh executable program would be in memory, but the ps command would report that code segment size as part of the RSS of each instance of the ksh program. The RSS value of 560 correlates with the Inuse numbers from the svmon command for the private value of 59 working-storage segments, for 58 code segments, and for the shared library data value of 23 of the process in 1 KB units.
The TRS value refers to the size of the resident set (real memory) of text. This is the number of code segment pages times four. As was noted earlier, this number exaggerates memory use for programs of which multiple instances are running. This does not include the shared text of the process. The TRS value of 232 correlates with the number of the svmon pages in the code segment (58) of the Inuse column in 1 K units. The TRS value can be higher than the TSIZ value because other pages, such as the XCOFF header and the loader section, may be included in the code segment.
The following calculations can be made for the values mentioned:
SIZE = 4 * Pgspace of (work lib data + work private) RSS = 4 * Inuse of (work lib data + work private + pers code) TRS = 4 * Inuse of (pers code)
The formula to calculate the minimum memory requirement of a program is the following:
Total memory pages (4 KB units) = T + ( N * ( PD + LD ) ) + F
where:
Multiply the result by 4 to obtain the number of kilobytes required. You may want to add in the kernel, kernel extension, and shared library text segment values to this as well even though they are shared by all processes on the system. For example, some applications like CATIA and databases use very large shared library modules. Note that because we have only used statistics from a single snapshot of the process, there is no guarantee that the value we get from the formula will be the correct value for the minimum working set size of a process. To get working set size, one would need to run a tool such as the rmss command or take many snapshots during the life of the process and determine the average values from these snapshots. See Memory requirements assessment with the rmss Command for more information.