1.
Introduction
The MicrosoftÒ Windows NTÒ 4.0 operating
system includes a performance monitoring tool called Windows NT Performance
Monitor. The executable name for the
Performance Monitor is Perfmon.exe. This tool, like any other tool, has various capabilities as well
as limitations. Understanding these
capabilities and limitations provides greater user productivity in using the
tool, as well as hope that the tool will not be misused. A primary source for information about
Performance Monitor is found in Chapter 10, About
Performance Monitor, in the Microsoft Windows NT Workstation 4.0 Resource
Kit. This paper reviews some of the information found in the Resource Kit and
then provides additional information.
This paper also mentions various tools that are found on the Resource
Kit CD.
Objects, counters and instances are
first discussed to clarify how many types of counters are defined in the
Windows NT 4.0 operating system. An
understanding of what type of counter is being used is important to the proper
use of the Performance Monitor tool and the data that it presents. Next, the features and limitations of the
chart view and report view, when looking at both current activity and logged
data, are discussed. Section four
discusses settings files, which are
very useful, but may also have limitations in working with Performance Monitor
log files. The fifth section presents
some common mistakes made in analyzing exported Performance Monitor chart data
and section six covers relogging log files.
2.
Objects, counters and instances
So what is it that the Performance
Monitor does anyway? "Performance Monitor measures the behavior of objects in
your computer. The objects represent
threads and processes, sections of shared memory, and physical devices. Performance Monitor collects data on
activity, demand, and space used by the objects." & "Performance
Monitor does not really count anything.
Its counters collect, average, and display data from internal counters
by using the Windows NT Registry and the Performance Library
DLLs."[1]
A counter in the Windows NT operating system is completely specified
by an object name, a counter name and possibly an instance name. Objects contain one or more counters and
counters may contain zero or more instances.
Some examples of these are:
·
Objects
Processor,
memory, cache, physical disk, logical disk, system, process, thread, objects,
paging file, &
·
Counters
For the
Processor object: %DPC Time, %Interrupt
Time, %Privileged Time, %Processor Time, %User time, &
·
Instances
For the
counter %Processor Time in the Processor object: 0, 1, 2, 3 on a 4-way SMP
system
The counters within the objects may be
of different types and the counter type
determines how the raw counter performance data is handled and displayed. The Resource Kit book talks about three
types of counters: instantaneous, averaging, and difference. However, the counters.hlp file (a Windows NT
help file) on the CD of the Resource Kit says that there are over 30 types of
counters that fall into four general (or generic) categories. The four general categories being:
instantaneous, average, percentage and text.
The counters.hlp file then goes on to define some of the counters of
being of a generic type called "difference." Thus, there really appears to be over 30
types of counters that fall into five general categories. But, of those five general categories,
Windows NT 4.0 uses four of them. There
are no defined counters of the general
category called "text."
An example of counter types, and the
number of defined counters of each type, seen on a particular Windows NT 4.0
system is:
Figure
1: An example of counter types
The Resource Kit tool named ctrlist.exe
produces a text listing of the counters on a system, along with the counter
type, a short description of the counters and other information. The "About counter types" section
of the counters.hlp help file gives the counter type names, the formula used to
calculate counter values from raw performance data collected, the generic type,
the formula used to calculate averages of the data, an example and a
description.
New objects and counters may be added
to Performance Monitor as "extensible counters." Many applications do define extensible
counters, which are added to the system when the application is installed. A Resource Kit tool called exctrlst.exe may
be used to list the extensible counters that have been installed on a
system. Since extensible counters are
not shipped as part of the Windows NT 4.0 operating system and may not be
reliable, there is testing done on the buffer pointers and contents of these
counters every time they are accessed.
The level of testing done on the extensible counters is settable in the
NT registry (see the Resource Kit book for more details). Additionally, the open procedure call of all
extensible counters is also timed on each access. If the open procedure call takes too long, an error is written to
the Event Viewer application event log.
The amount of time the system waits for the open procedure to complete
is another item that may be set in the NT registry (again, see the Resource Kit
book for more details).
3. Charts and reports for current activity versus charts and reports for logged data
Performance Monitor has three reporting
views and two input sources. The
reporting views are:
·
charts (line charts and histograms),
·
reports and
·
alerts.
The input sources are:
·
current activity and
·
log files.
The current activity input source
provides real-time performance data from the local system (the system perfmon
is running on), remote systems, or both the local and remote systems at the
same time. Log files, generated by
Performance Monitor or by a Resource Kit tool called monitor.exe, may also
contain data from local and remote systems.
Log files provide the ability to store performance data in a file so
that charts, reports, and alert information can be generated at a later time.
3.1 Chart view showing current activity
The most widely used reporting view of
Performance Monitor is the chart view displaying line charts. In this view, up to 100 samples of counter
data are displayed for all of the performance counters that have been added to
the chart. Some of the features and limitations
to remember when using the chart view showing
current activity are:
·
One hundred (100) x-axis points are displayed on the line
chart for each performance counter being displayed.
·
The sample interval, which has a default of 1 second, times
100 determines the amount of wall time the chart view will display.
·
The last, average, min, and max values
displayed at the top of the legend bar are for the counter that is highlighted
in the legend bar and are calculated using the most recent 100 samples (the ones
that are displayed on a line chart).
·
A line on a line chart, or a bar on a histogram, may be
highlighted in white by pressing ctrl-h (backspace). The line or bar highlighted in white is the one that corresponds
to the counter that is highlighted in the legend bar.
·
The bars in the histogram chart show only the data for the
most recent sample. The average, min, and max values are
still for the most recent 100 samples.
·
The Performance Monitor chart display may be . frozen. by
going into chart options and selecting manual updates instead of periodic
updates.
·
The default y-axis for line charts is 0 to 100. The maximum y-axis value may be changed in
the chart options dialog box.
·
The counter data displayed in a line chart may be scaled by
powers of 10 (only powers of 10) to more clearly see the data. There is no autoscale feature in Performance
Monitor.
·
A counter value at the y-axis maximum means that the value
of the counter for that sample is equal to, or greater than, the value of the
y-axis maximum.
·
Ctrl-g toggles the legend bar on and off.
·
Ctrl-m (enter key), ctrl-s, and ctrl-t toggle on and off the
menu bar, status bar, and tools bar respectively.
·
Exporting chart data when charting current activity writes the values of the 100 most recent
samples to a tab or comma separated file.
3.2 Report view showing current activity
The features and limitations to
remember when using report view showing
current activity are:
·
Only the most current sample of the current activity is
shown in the report. No average, min or
max is given.
·
The default sample interval is 5 seconds and does not need
to be the same as the sample interval for the chart view, the alert view, or
the interval at which performance data is being logged to a log file.
3.3 Chart view showing logged data
The behavior
of Performance Monitor changes in some significant ways when logged data is
being reported instead of reporting the current activity (real-time). Important things to note are:
·
Only one hundred (100) x-axis points are displayed on the
line chart for each selected performance counter, regardless of the number of
samples in the log file.
·
The amount of time displayed on a line chart is determined
by the time window which has been set.
By default, the line chart shows the entire time span that is covered in
the log file.
·
If the log file contains more than 100 samples, how the
extra samples are handled depends on the generic type of the counter. (The Performance Monitor documentation does
not indicate this.)
·
For instantaneous type counters, every n-th sample is
displayed on a line chart. The value of
n is approximately the number of samples in the log file that are in the
current time window divided by 100.
·
For percentage, average and difference counters, n samples
are combined to generate one of the 100 displayed data (sample) points. The samples are combined in a way that is
appropriate for the type of counter.
·
The last, average,
min, and max values displayed at the top of the legend bar are for the
counter that is highlighted in the legend bar and are calculated using the
displayed data (remember that multiple samples may have been combined to
generate one of the displayed data points).
For example,
the log file may contain Processor\%Processor Times of 0% and 100%, but if the
displayed data is a combination of multiple samples in the log file a min of
30% and a max of 40% may be seen.
·
As with the current activity input mode, a line on a line
chart or a bar on a histogram may be highlighted in white by pressing ctrl-h
(backspace). The line or bar
highlighted in white is the one that corresponds to the counter that is
highlighted in the legend bar.
·
The bars in the histogram chart show the average value of
the samples in the time window for percentage, average and difference type
counters. The value displayed is equal
to the value that is given in a report for the same time window. Beware, however, that the value displayed
graphically will typically be off a little from the average given in the legend
bar.
·
The bars in the histogram chart show the last sample value
in the time window for instantaneous type counters (which corresponds to the
last displayed data point shown in line chart view).
·
The y-axis maximum value and scaling of lines is handled
just like it is for the current activity input mode.
·
Exporting chart data when charting logged data writes the
values of the all the samples in the log file for the current time window to a
file, independent of any combining of samples that may be occurring for the
displayed chart data.
·
The "Explain>>" button in the "Add to &
" dialog boxes is not available.
The following
three copies of Performance Monitor line chart views show graphically the
effects of having only 100 x-axis display points on the graph and combining n
samples to form one of the displayed data points. The full log file contained 1,000 samples and the periodic
behavior of processor utilization continued throughout the time the log file
was logging performance data.
The following
graph shows the first 100 samples of the log file being displayed by
Performance Monitor.
Figure
2: First 100 data
points of a log file
Figure 3 has a
time window of 300 samples and has three samples being combined to form one of
the 100 displayed data points.
Figure
3: Three hundred
data points of the log file
The final
graph (Figure 4) has a time window of 1,000 samples so that ten samples are
being combined to form one of the 100 displayed data points.
Figure
4: One thousand
data points of the log file
One way of
understanding what Figures 2, 3 and 4 show, is to explain it in terms of sample
rates. Figure 2 shows 100 samples taken
at one second intervals. Figure 3 shows
100 samples taken at three second intervals and Figure 4 shows 100 samples
taken at ten second intervals. Using
longer sample intervals (combining multiple samples to be able to display only
100 x-axis data points) masks seeing the true periodic nature of the data in
this particular log file.
3.4 Report view showing logged data
The report
view of Performance Monitor also varies when using logged data instead of
current activity. In the logged data
input mode:
·
The value given in the report is the average of the samples
for the currently specified time window for percentage, average and difference
type counters.
·
The value given in the report is the value of the last
sample for the currently specified time window for instantaneous type counters.
·
The average value as given in the report typically varies a
small amount from the average value given in the chart view legend bar. Use the value in the report.
4. Settings files
Settings files
store the selected counters and options for a view, or store the selected
counters and options for all the views.
They are very useful and can increase productivity in using Performance
Monitor. The file extensions associated
with settings files are as follows:
·
Alert view - .pma
·
Chart view - .pmc
·
Report view - .pmr
·
Log view - .pml
·
Workspace (all views) - .pmw
A chart view settings file will save
all the counter selections added to a chart along with the counter scaling
factors, line colors, line sizes, y-axis maximum, and other options for the
chart view. Once created, a settings
file may be double clicked in the NT Explorer to start a copy of Performance
Monitor with the settings file applied to it.
Additionally, various settings files may be dragged and dropped onto a
running copy of the Performance Monitor.
Doing this with data from a log file allows for a quick look at the
logged performance data.
Settings files are not without their
annoyances, however. Here are some
things to know about them:
·
If the counters stored in a settings file (such as the ones
being displayed on a chart) were only displaying values from the local system
when the settings file was saved, then the settings file may be copied from
system to system and will work with the current activity input mode on the
other systems. They will also work for
log files on other systems if the log files contain only local system data.
As
an example, a chart settings file created on System_A which was displaying only
counters local to System_A, may be copied to System_B and used there to produce a chart (from current data or a log
file) that displays counters from System_B being monitored on System_B.
This
is due to the fact that the local system name is used as the default in
Performance Monitor and the system name (computer name) is not stored in the
settings file with the counter names when it is a local counter.
·
If the counters in a settings file were displaying data from
a remote system when the settings file was created, then the remote system name
is saved in the settings file with the counter name. Thus, when the settings file is applied to a running copy of
Performance Monitor (or opened), Performance Monitor tries to connect to the
remote system if the current activity input mode is active, or tries to find
data in the log file for the settings file specified system. (Log files keep the system name along with
the object\counter\instance names in the log files.)
What this means for the common case of:
1. create a
Performance Monitor log file on a system under test;
2. move the
Performance Monitor log file to another system for analysis while the system
under test runs other tests;
3. run
Performance Monitor on the analysis system to analyze the logged performance
data;
is that settings files must have the
specific name of the test system in them to work with the logged data that came
from the test system. Setting files
that include the name of a specific system with the object\counter\instance
names, may be created by, 1) creating the chart desired on the analysis system
while using the perfmon log file from the test system and then saving the chart
settings to a *.pmc file, or 2) take an existing *.pmc file and run the
setedit.exe program (from the Resource Kit CD) on it to add in the specific
system name needed.
Settings files generally work well,
however, there is a case when they do not work well. When applications loaded on the test system have defined
extensible counters (which are usually the counters that are really wanted) and
the same extensible counters are not defined on the analysis system, settings
files may not work at all. To add to
the problem, the analysis system needs to not only have the same extensible
counters, but the extensible counters need to have the same counter IDs on both
the test system and the analysis system.
This means that applications that define extensible counters need to be
loaded in the same order on both the test system and the analysis system. No error is generated in using a settings
file that tries to display extensible counters, but the data from the log file
will display all zeroes if extensible counters and counter IDs do not
match.
Note, however, that Performance Monitor
can analyze log file data from a test system on a different system - even if
there is a mismatch in the extensible counters defined or a mismatch of the
extensible counter IDs - if it is done manually. That is to say, it works if a settings file is not used.
5. Exporting Chart Data
To overcome some of the problems of
Performance Monitor with regards to finding min and max values, getting the
average of an instantaneous counter, and plotting more than 100 x-axis points,
it is possible to export chart data to a tab or comma separated text file. Once the chart data is exported, it may be
imported into a spreadsheet or other statistical tool for analysis. The caution in doing this, is that at this
point some people will start to calculate averages in the spreadsheet without
regard to the type of counter the
data came from and trusting that sample
intervals were all identical.
The following example demonstrates the
problem of ignoring the type of counter that the performance data comes
from. In this example, the first column
after the date and time stamp is "Memory Page Reads/sec." This counter is of the generic type:
difference. Doing a simple arithmetic
mean of the eight samples in this column produces an average of 34.1335 which
is consistent with the rounded value of 34.134 that Performance Monitor gives
when it combines the eight samples into one sample (this is done through relogging the log file). For this counter it is OK to do averaging of
the exported chart data if sample intervals are identical.
Doing an average of the eight samples
in the "PhysicalDisk Avg. Disk Bytes/Read" column gives an average of
2048, which is wrong. The correct
average is 4096. Looking at the
samples, one might conclude that if the samples that were zero were thrown out
(there were no disk reads during those samples) then the rest of the samples
could be averaged without any problems.
This method, however, will only give the correct answer some of the
time! The column labeled
"PhysicalDisk Avg. Disk Bytes/Writes" has all non-zero samples, yet a
simple arithmetic mean done on the eight samples yields the wrong answer. This is due to the fact that the values for
the Disk Bytes/Read and Disk Bytes/Write counters are both calculated values
and some information is lost in deriving their values. Stated another way, there is no way to be
sure that the denominators of the ratios that are being averaged all have the
same value.
