NA ISSW Technical Interchange 2014

IBM WebSphere Application Server

Performance Data and Tools
Session Number: <AP19>
Gary Hunt – Focused Technology Practice

© 2014 IBM Corporation

WebSphere Performance Data and Tools NA ISSW TI 2014

“Pay no attention to the man behind the curtain”

2 © 2014 IBM Corporation

WebSphere Performance Data and Tools NA ISSW TI 2014

Presentation Agenda and Objectives

• WebSphere Performance Data
 Hardware Performance Data
 Operating System Performance Data
 JVM Performance Data
 Web Server Performance Data
 Application Server Performance Data
 Messaging Performance Data
 Database Performance Data
• WebSphere Performance Tools
 CPU Utilization Tools
 HTTP Session Tools
 Web Server Access Log
 Garbage Collection Tools
 Thread Dump Tools
 Messaging Tools
 PMI Tools

After completing this presentation, you should be able to:

• List the performance data that is important to monitoring WebSphere based applications
• List the performance tools that are useful in monitoring performance data that describe the behavior of
WebSphere based applications

3 © 2014 IBM Corporation

 NA ISSW TI 2014

WebSphere Performance Data

4 © 2014 IBM Corporation

WebSphere Performance Data and Tools NA ISSW TI 2014

Introduction
• High level performance data areas
 Hardware
 Operating System
 JVM
 Web Servers
 Application Servers
 Messaging
 Databases

5 © 2014 IBM Corporation

WebSphere Performance Data and Tools NA ISSW TI 2014

Hardware Performance Data

• Number of physical processor
 Virtual and logical processors are operating system concepts
• Amount of physical memory
 Maximum swap size is an operating system concept
• Total and free disk space
• I/O channel utilization
• Network interface card utilization
• Note:
 CPU Utilization and memory utilization are Operating System level concepts

6 © 2014 IBM Corporation

WebSphere Performance Data and Tools NA ISSW TI 2014

Operating System Performance Data

• System configuration
 Number of virtual and logical processors
 Logical processors per physical processor
• CPU utilization
 Whole partition
 Per process
• Physical and virtual memory utilization
 Maximum swap space size and swap space utilization

7 © 2014 IBM Corporation

WebSphere Performance Data and Tools NA ISSW TI 2014

JVM Performance Data

• Heap utilization as in current heap size versus maximum heap size
 Current nursery size versus maximum nursery size
 Current tenured size versus maximum tenured size
• Live object utilization
 Current size of live versus current heap size
 Current nursery used versus current nursery size
 Recent minimum tenured used versus current tenured size
• Garbage collection (GC) overhead
 Time in GC versus Time between GCs
• Level of heap fragmentation as in “Free before GC”
• Average and peak size of requested objects
• Sources of JVM performance data
 JVM configuration such as maximum heap size are Java command line parameters
 Statistics regarding garbage collection behavior are available via verbose garbage collection output

8 © 2014 IBM Corporation

WebSphere Performance Data and Tools NA ISSW TI 2014

Web Server Performance Data

• History of incoming HTTP Requests
 Size of request
 Time of request
 Time to respond
 URL of target server and content
• Source of Web Server Performance Data
 Web server access log

9 © 2014 IBM Corporation

WebSphere Performance Data and Tools NA ISSW TI 2014

Application Server Performance Data

• Data source utilization levels
 Current number of database connections per data source versus Maximum number of database connections
per data source
• Caching
 Maximum number of cache entries
 Cache hit rate
• Thread pool utilization levels
 Current number of active threads per thread pool versus Maximum number of threads per thread pool
• Source of Application Server Performance Data
 Specialized tools are required to monitor data sources, caching, and thread pools

10 © 2014 IBM Corporation

WebSphere Performance Data and Tools NA ISSW TI 2014

Messaging Performance Data

• Names of queues
• Location (Server) for queues
• Queue utilization levels
 Current queue depth versus Maximum queue depth
• Average and peak message sizes
• Current and peak queue depths
 Queue depth trends
• Message Arrival Rates
• Sources of messaging performance data
 WebSphere administrative console for WebSphere Default Messaging (System Integration Bus)
 Specialized tools for other messaging providers including MQ

11 © 2014 IBM Corporation

WebSphere Performance Data and Tools NA ISSW TI 2014

Database Performance Data

• Names (and counts) of databases and schemas
• Database connection utilization levels
 Current number of database connections versus Maximum number of database connections
• Longest running SQL queries
• Buffer utilization levels
• Sources of database performance data
 Specialized tools are required for database performance monitoring
 Typically provided by the database product

12 © 2014 IBM Corporation

 NA ISSW TI 2014

WebSphere Performance Tools

13 © 2014 IBM Corporation

WebSphere Performance Data and Tools NA ISSW TI 2014

Introduction
• High-end monitoring tools can be very useful
 IBM Tivoli for Composite Application Management (ITCAM) Suite
 Other non-IBM products such as Wily Introscope
• Much tuning/debugging can be performed with freely available tools
 Are already available or can be installed easily
 Focus on the problem at hand instead of the tool(s)
• Many tuning and problem determination tools can be downloaded using the IBM Support Assistant
(ISA)

14 © 2014 IBM Corporation

WebSphere Performance Data and Tools NA ISSW TI 2014

Aggregated CPU Utilization – “vmstat”

• “vmstat” is a lightweight monitoring tool for key system resources
 Especially useful to monitor CPU usage
• Available on all major UNIX operating systems and Linux distributions
• Use “mpstat” on multi-processor AIX systems to monitor per processor
statistics
• Command Format:
 vmstat [-options] [interval [count]]
• Where interval is the number of seconds between measurements and count is the
number of measurements performed
• Time stamps for each line of data are useful
 “vmstat” on AIX supports the “-t” switch which adds time stamps to each line
 Scripts can be developed to add time stamps to “vmstat” output

15 © 2014 IBM Corporation

WebSphere Performance Data and Tools NA ISSW TI 2014

Aggregated CPU Utilization – “vmstat” (continued)

• To print system resource usage every 10 seconds, run the following
command:

# vmstat 10
procs -----------memory---------- ---swap-- -----io---- -system-- -----cpu------
r b swpd free buff cache si so bi bo in cs us sy id wa st
4 0 99864 60000 34404 451260 0 2 40 32 269 927 6 7 86 1 0
0 0 99848 60572 34416 451524 0 0 0 87 264 1514 17 9 74 1 0
0 0 99828 60356 34428 451728 6 0 6 50 263 1479 13 6 81 1 0
10 0 103020 51880 34372 446612 0 302 0 376 267 1259 12 35 53 0 0
2 0 111644 53328 26968 443712 6 844 7 880 265 1396 24 72 4 0 0
40 0 111636 53884 26992 442916 0 0 0 80 262 1460 17 10 69 5 0
0 0 111612 52724 27008 443412 3 0 3 74 264 1576 13 7 80 0 0
5 0 111568 51532 27020 444420 6 0 6 70 266 1614 20 17 63 0 0

• Output from a SUSE Linux system

16 © 2014 IBM Corporation

WebSphere Performance Data and Tools NA ISSW TI 2014

Aggregated CPU Utilization – Key Metrics from “vmstat”

• Aggregated CPU Utilization
 AIX: for micro-partitions on shared processor pools, “vmstat” includes number of physical processors assigned
to the LPAR (“pc”) and percent of entitlement (“ec”)
• Run Queue Depth
 Indicates level of threading in running processes
• Page in/Page Out (Swapping) Statistics
 The JVM heap should not be swapped out

17 © 2014 IBM Corporation

WebSphere Performance Data and Tools NA ISSW TI 2014

Graphing “vmstat -t” (AIX) CPU Utilization Data

Parser and Graphing Template for AIX “vmstat –t” Command Output
https://w3-03.ibm.com/tools/cm/iram/oslc/assets/A8D83AE2-81EF-E387-F6AE-FBC374537720/1.0

18 © 2014 IBM Corporation

WebSphere Performance Data and Tools NA ISSW TI 2014

Additional CPU Utilization Tools

• Aggregated and Per Process CPU Utilization
 AIX: “topas”
 Linux including zLinux: “top”
• also reports per process memory utilization

• Whole “Frame” CPU Utilization

 AIX: “lparstat”
“lparstat” Usage - http://publib.boulder.ibm.com/infocenter/aix/v7r1/index.jsp?topic=%2Fcom.ibm.aix.cmds%2Fdoc
%2Faixcmds3%2Flparstat.htm

19 © 2014 IBM Corporation

WebSphere Performance Data and Tools NA ISSW TI 2014

Graphing “top -b” (Linux) CPU Utilization Data

Parser and Graphing Template for Linux “top –b” Command Output
https://w3-03.ibm.com/tools/cm/iram/oslc/assets/9C1EEA1F-5EA8-FF33-A005-42F75366C72F/1.0

20 © 2014 IBM Corporation

WebSphere Performance Data and Tools NA ISSW TI 2014

Analysis and monitoring tools: nmon

• Free and very powerful tool to monitor AIX and Linux systems
 Can be run in interactive mode and non-interactive mode
 Monitors most operating system resources including
• CPU
• Memory
• Network I/O
• Disk I/O
 Very useful on platforms that support hardware virtualization
• Monitors LPAR resources on AIX system p machines
 Non-interactive mode probably most useful during performance test runs
• Monitors and logs to “.csv” file
• Use Excel Analyzer spreadsheet to import the “.csv” file and generate graphs

21 © 2014 IBM Corporation

WebSphere Performance Data and Tools NA ISSW TI 2014

Nmon output of CPU and memory statistics

• “nmon cm” provides CPU Utilization and Memory Statistics

22 © 2014 IBM Corporation

WebSphere Performance Data and Tools NA ISSW TI 2014

Nmon output of kernal statistics and top processes

• “nmon kt” provides Kernel statistics and top processes

23 © 2014 IBM Corporation

WebSphere Performance Data and Tools NA ISSW TI 2014

Web server performance tools: IBM HTTP Server “server-status”

• “server-status” allows easy monitoring of a web server through a browser
• Enable by adding the following lines to “httpd.conf” file

<Location /server-status>
SetHandler server-status
</Location>
• After a server restart, the server-status is available
 http://yourhost/server-status
• Optionally, you can specify a refresh interval
 For example 10 seconds http://yourhost/server-status?refresh=10
• Can optionally monitor extended status
 Provides more detailed child level statistics at some performance cost

24 © 2014 IBM Corporation

WebSphere Performance Data and Tools NA ISSW TI 2014

Web server performance tools: Web server access logs

• Apache response time logging is a powerful tool
 Used to obtain the actual response times at the server side
 Does not depend on the speed of the network or any other unknowns such as proxy servers
• Enable using standard Apache Web server directives in the “httpd.conf” file

LogFormat "%t \"%r\" Response time (us): %D" responsetime

CustomLog logs/response.log responsetime
• Logs response times accurately in μs (micro-seconds)
 Almost no overhead so can be used in production
 Use various tools to analyze response time characteristics

25 © 2014 IBM Corporation

WebSphere Performance Data and Tools NA ISSW TI 2014

Web server performance tools: Web server access logs

• Analysis of Apache response time logs can be very powerful
 Use of standard tools such as Microsoft Excel
 Use various graphing options
• Scatter, Logarithmic. Histogram, etc

26 © 2014 IBM Corporation

WebSphere Performance Data and Tools NA ISSW TI 2014

JVM performance tools: Verbose garbage collection

• The best way to analyze garbage collection behavior is to enable and collect verbose GC data and graph it
 Garbage Collection time data
• The IBM Support Assistant includes a tool called Garbage Collection and Memory Visualizer (GCMV) which will
graph verbose GC
• GCMV uses a powerful statistical analysis engine which provides guidance in the following areas:
• Memory leak detection
• Detect Java heap exhaustion and memory leaks
• Detect native heap (malloc) exhaustion and memory leaks
• Optimize garbage collection performance
• Determine garbage collection overhead
• Detect long or frequent garbage collection cycles and recommend configuration changes to address these issues
• Recommend optimal garbage collection policy
• Fine tuning of Java heap size
• Determine peak and average memory usage
• Recommend Java heap settings

• Alternative Techniques
 PMAT:
• https://www.ibm.com/developerworks/community/groups/service/html/communityview?communityUuid=22d56091-
3a7b-4497-b36e-634b51838e11
 Verbose GC Parser and Graphing Templates
• https://w3-03.ibm.com/tools/cm/iram/oslc/assets/%7B4FD70C73-20D2-EE7F-BD3E-4907BEF2AAC8%7D/1.0

27 © 2014 IBM Corporation

WebSphere Performance Data and Tools NA ISSW TI 2014

JVM performance tools: Garbage Collection Memory Visualizer (GCMV)

The VGC Data menu allows you

to choose what data to display.

Pre-defined
Templates
allow you to plot
different
metrics.
The Axes panel
supports
customized
units.

The Line plot tab

shows the data
visualization.

28 © 2014 IBM Corporation

WebSphere Performance Data and tools NA ISSW TI 2014

JVM performance tools: Thread dumps

• Thread dumps provide a snapshot of the running JVM
• Useful when:
 CPU utilization is not as expected (lower/higher)
 Throughput has hit a plateau
• Thread dumps produce a stack trace of each thread’s activity
• Things to keep in mind:
 A thread dump is a snapshot
• Take several, giving the JVM time to recover after each
• The overhead of producing a thread dump is similar to the overhead of a garbage collection cycle
 They vary in format and detail among platforms
• Also, an egg-hunt to find where each platform writes the dump file
• The IBM JVM running WebSphere will document where the thread dump was written in the WebSphere
“native_stderr.log” file

29 © 2014 IBM Corporation

WebSphere Performance Data and Tools NA ISSW TI 2014

JVM performance tools: Thread dumps

• Analyzing thread dumps with a plain editor can be difficult; use a thread dump viewer to help
• IBM Thread and Monitor Dump Analyzer for Java
 This tool simplifies the process of analyzing a thread dump by presenting a list of the threads that existed at
the time the thread dump was triggered
 It simplifies the process of examining the Java call stack from the list of threads.
 The tool also provides information regarding locking
 Shows the threads that own locks
 Shows the thread(s) that are blocked waiting on a lock.
 https://www.ibm.com/developerworks/community/groups/service/html/communityview?
communityUuid=2245aa39-fa5c-4475-b891-14c205f7333c
• Additional Techniques
 Thread Dump Parser and Call Stack Correlator
• https://w3-03.ibm.com/tools/cm/iram/oslc/assets/7689645E-DAFA-FA6D-D179-5127294D7E04/1.0

30 © 2014 IBM Corporation

WebSphere Performance Data and Tools NA ISSW TI 2014

JVM performance tools: Thread and Monitor Dump Analyzer for Java

• Take care with the thread statuses reported by the JVM in the thread dump
 Runnable threads may have a status of “Waiting on Condition”
 Thread status vs “Active/Idle” as reported via WebSphere PMI
• A priori knowledge of the threading model for the application is required
31 © 2014 IBM Corporation
WebSphere Performance Data and Tools NA ISSW TI 2014

Messaging Performance Tools

• MQ Series
 MQ Explorer can be used to monitor MQ hosted queues
 MQ Explorer comes with the MQ Series product
• WebSphere default messaging (Service Integration Bus)
 SIBus Explorer has similar functionality to MQ Explorer and is available on IBM Alphaworks
 Various aspects of SIBus status can be monitored through the WebSphere Administrator Console and with
Tivoli Performance Viewer
• Functions available in both tools
 Provide a list of queues
 Report current queue depths
 Manage queue contents

32 © 2014 IBM Corporation

WebSphere Performance Data and Tools NA ISSW TI 2014

Performance Data: Performance Monitoring Infrastructure (PMI)

• When PMI is enabled for an application server, monitoring of individual WebSphere components
can be enabled or disabled dynamically.
• WebSphere PMI provides four predefined statistic sets that can be used to enable a set of statistics
 Basic: Statistics specified in J2EE 1.4 as well as top statistics like CPU usage and live HTTP sessions are enabled.
Basic level of monitoring is the default
 Extended: Basic statistics plus key statistics from various WebSphere components like Workload Management
(WLM) and dynamic caching.
 All: All statistics are enabled
 Custom: Allows the user to enable or disable statistics selectively
• PMI levels can be set via the WebSphere administrative console or the wsadmin command

33 © 2014 IBM Corporation

WebSphere Performance Data and Tools NA ISSW TI 2014

Performance Tools: Tivoli Performance Viewer (TPV)

• Tivoli Performance Viewer is part of the WebSphere administrative console
 Allows graphical monitoring using PMI of:
 WebSphere managed resources
 Thread pools
 Connection pools
 Connection factories
 Messaging engine
 JVM heap usage
• Response times
 JSPs and servlets
 EJBs

34 © 2014 IBM Corporation

WebSphere Performance Data and Tools NA ISSW TI 2014

Performance Tools: Tivoli Performance Viewer (TPV)

• Chart view of TPV monitoring DayTrader servlets

35 © 2014 IBM Corporation

WebSphere Performance Data and Tools NA ISSW TI 2014

Performance Tools: Request Metrics

• Request metrics is a facility that enables tracking of individual JEE requests recording the processing
time in each of the major WebSphere Application Server components
 Data is either recorded in WAS log files or sent to ARM agents (or both)
 Provides a hierarchical view of each request by response time against the following application server
components
• Web server plug-in (for requests that arrive at the web server)
• Proxy server
• Web Container
• EJB Container
• JDBC calls
• Java Connector Architecture (JCA)
• Web services
• JMS interactions
• Service Integration Bus (SIBus)
• Portlet Container
• Asynchronous Beans

36 © 2014 IBM Corporation

WebSphere Performance Data and Tools NA ISSW TI 2014

Performance Tools: Request Metrics

• Collecting performance data at all of the instrumented application components can have an adverse
effect on performance.
• The impact of monitoring can be minimized by using filtering to limit the amount of monitoring per
transaction.
• Request Metrics provides filtering against the following transaction details:
 Source IP
 URI
 EJB method name
 JMS parameters
 Web services parameters

37 © 2014 IBM Corporation

WebSphere Performance Data and Tools NA ISSW TI 2014

A final word on performance data and tools

• The tools discussed in this presentation are only a small subset of the tools used by people to
monitor WebSphere performance
 Investigate all available tools and use tooling that makes your team the most productive
• Use the right tool at the right time
 Use the tool you are familiar with
 Use the tool that does not require extensive setup
 Use the tool that is cost effective

38 © 2014 IBM Corporation

 NA ISSW TI 2014

Legal Disclaimer

• © IBM Corporation 2014. All Rights Reserved.

• The information contained in this publication is provided for informational purposes only. While efforts were made to verify the completeness and accuracy of the information contained
in this publication, it is provided AS IS without warranty of any kind, express or implied. In addition, this information is based on IBM’s current product plans and strategy, which are
subject to change by IBM without notice. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, this publication or any other materials. Nothing
contained in this publication is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and
conditions of the applicable license agreement governing the use of IBM software.

• References in this presentation to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. Product release dates and/or
capabilities referenced in this presentation may change at any time at IBM’s sole discretion based on market opportunities or other factors, and are not intended to be a commitment to
future product or feature availability in any way. Nothing contained in these materials is intended to, nor shall have the effect of, stating or implying that any activities undertaken by
you will result in any specific sales, revenue growth or other results.

• If the text contains performance statistics or references to benchmarks, insert the following language; otherwise delete:
Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will
experience will vary depending upon many factors, including considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage
configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve results similar to those stated here.

39 © 2014 IBM Corporation