Don't wait until your system slows to a crawl before dealing with performance issues. By monitoring CPU utilization
for critical applications, you can get warnings in advance of major problems.
This tip provides updated cumulative CPU utilization guideline values for monitoring your "interactive-like," response-time critical applications. Use a guideline value to alert you to possible response time degradation in the "near future" or a possible cause for "longer than normal" response times. The guideline values increase as the number of processors available in the system or partition (LPAR configuration) increases.
Recall that guidelines are just that -- guidelines. There may be instances when response time is very acceptable at values higher than guideline values shown. Alternatively, there may be other instances when response time is not acceptable and the CPU utilization guidelines are not exceeded.
The table shown below is an updated set of guideline values for monitoring steady-state CPU utilization of "high priority jobs" running under OS/400 -- where a response time is an important consideration. These guidelines apply to jobs/threads that perform work based upon input data and receiving a response. This work includes jobs doing 5250 transactions, Web serving, Domino server functions, and SQL-oriented database file operations.
No. of processors Guideline values for per system/partition "high priority jobs/threads" 1 70 2 76 4 81 8 85 12 87 18 90 24 91 32 93
You need to determine, for your application environment, at which OS/400 job run priority value(s) the jobs/threads run that you want to monitor. The guideline for a job/thread at any given priority value then includes the cumulative utilization of all jobs/threads active at equal to or higher priority compared to the priority of the selected job(s)/thread(s). You are then monitoring the cumulative CPU utilization of all jobs/threads running at equal to or higher priority to that value. For example, assume you want to monitor HTTP serving functions. Find out the run priority for the associated HTTP jobs/threads running under OS/400. Taking HTTP configuration defaults, this would be OS/400 job/thread RUNPTY 25. By using a RUNPTY value of 25 you include the other OS/400 functions that run jobs and threads at RUNPTY values 0 (iSeries microcode) through 25.
Under OS/400, key high priority functions/applications with default RUNPTY values less than 25 include the following:
- 5250 jobs. These jobs default priority value is 20.
- QSQSRVR jobs that perform Call Level Interface DB2 UDB SQL functions used by IBM functions such as Management Central and typically by Java-based applications performing SQL. These jobs/threads default to running at priority value 10.
- QZDASOINIT jobs performing ODBC/JDBC SQL functions requested by client workstations. These jobs/threads default to running at priority 20.
- Domino for iSeries server jobs performing various Domino functions. These jobs default to running at priority 20.
- QRWTSRVR DDM Target jobs performing DDM requests from a DDM source application. These jobs default to running at priority 20.
You can get an approximation of the cumulative CPU utilization according to job type run priority values from the Resource Utilization Expansion section of the System Report. That report is one of the functions available with the Manager feature of the Performance Tools for iSeries, 5722-PT1, licensed program. Performance data used by this report is collected using OS/400 Collection Services.
Starting with V5R1, iSeries Navigator (Operations Navigator) Management Central System monitor or Job monitor functions can assist you in real-time monitoring of CPU utilization values for the various jobs/servers you wish to monitor.
Note that that there is no specific real-time "high priority work" monitor function provided by OS/400 through V5R2. Here are some suggested techniques, all or some of which may be useful in your environment:
- Use Management Central System Monitor metric CPU Utilization (Interactive Jobs). This monitors average CPU utilization for all active jobs performing 5250 I/O operations. The percent is relative to the total processor capacity (processor CPW) of the system/partition. This does not include work performed by high-priority jobs not performing 5250 I/O operations. For example, CPU utilization (Interactive Jobs) would not include CPU consumed by the QSQSRVR, QZDASOINIT, Domino, or HTTP-related jobs.
- Use Management Central Job Monitor and select job types Interactive, Batch Immediate, and Prestart. Depending on your runtime environment, this may cover only the jobs you are interested in. For example QSQSRVR, QZDASOINIT, and QRWTSRVR are prestarted jobs. The jobs that do most of the Domino server and HTTP server work run as Batch Immediate ("BCI") jobs. Note, there may be CPU consumed by other batch immediate and prestarted jobs in your environment.
These "high priority CPU utilization" guideline values cannot be directly applied to Interactive Feature utilization. A complete discussion of a set of guideline values for the Interactive Feature utilization is beyond the scope of this tip.
However, here is some information that may assist you with setting up your own guideline values for the Interactive Feature utilization:
- Each system/partition with a low interactive CPW capacity with a single 5250 job consuming high CPU utilization over an extended period is considered an administrative job and may use "more CPU" than you may expect with a low interactive CPW capacity.
- Interactive Feature utilization applies when there are two or more 5250 jobs consuming high CPU utilization over an extended period of time. Interactive Feature utilization varies, based on the time interval of consistently high CPU utilization by jobs/threads performing 5250 input and output operations and the higher the interactive CPW capacity for the partition/system.
- In general, the Interactive Feature utilization is computed by the microcode, and the algorithm can be characterized as slow to anger (high 5250 jobs CPU utilization over several minutes time span) and quick to forgive (lowered 5250 job/thread CPU utilization over a "short" time span). At this time, the general recommended guideline value for Interactive Feature utilization is a value of 70%. As with the high priority CPU utilization guideline, there may be times of acceptable response time when Interactive Feature utilization approaches 100%, and there may be times of degraded response time when less than 70% of the Interactive Feature is being utilized over an extended period of time.
- As the percent of Interactive CPW rating approaches 50% or more of the partition's (or total system) Processor CPW rating, guideline values higher than 70% may be applied. Additional information requires further investigation.
This tip assumes V5R1 or later. If you are running V4R5, you can use Management Central System monitor support, but not Job monitor support (new in V5R1). V4R5 or later Management Central System monitor support includes monitoring Interactive Feature utilization. V4R5 or later OS/400 Collection Services collects Interactive Feature utilization. After associated performance database files (QAPMcccc) have been generated, the Manager feature of Performance Tools for iSeries, 5722-PT1 can produce System and Component reports that contain Interactive Feature utilization values.
Managing OS/400 with Operations Navigator V5R1 Volume 1: Overview and more, SG24-6226 provides overview information on Collection Services and detailed information on using a System or Job monitor. This Redpiece can be found at http://www.ibm.com/redbooks. Search for this document by number.
iSeries Information Center at http://www.ibm.com/eserver/iseries/infocenter. Select Systems Management -> Performance.
About the author: Jim Cook has worked at IBM Rochester for over 30 years in the areas of communications, performance and general OS/400 system interfaces. He has spent the last 10 years in the iSeries International Technical Support Center.
- Identify interactive job using CPU UTILIZATION window in OpsNav
Search400.com member Karl Baker gives you step-by-step instructions for identifying an interactive job consuming the most CPU.
- How to remedy poor response times This user says he has a new 270 running 5.1 that's approximately twice the size of his old system. He isn't running anymore than he was before, but he's having serious performance problems. On occasion his response times on interactive jobs has been five minutes. Site expert Tim Granatir lends a hand.
- Replace Performance Monitor commands in OS/400 V5R1
When IBM released OS/400 V5R1, it replaced OS/400's Performance Monitor feature with a new Collection Services function that is configured through the Management Central capability of iSeries Operations Navigator. That means many shops migrating to V5R1 must learn to put aside the old green-screen commands that start, stop, end, add and change performance collection data under the Performance Monitor and opt instead for the mostly graphical configuration capabilities that come with Collection Services. Unfortunately, there's a problem with this setup. Joe Hertvik tells you what it is and provides a workaround for it.