© 2017, IBM Corporation Liberty z/OS Good Practices
WebSphere Liberty z/OS
Operations, Monitoring, Problem Determination
(Some sections still under construction)
1
© 2017, IBM Corporation Liberty z/OS Good Practices
Objective of this Presentation
Provide an understanding of operational good practices
Provide an understanding of monitoring alternatives
Provide an understanding of problem determination practices
This unit is still under construction, and will likely have
content added over time as we come to understand the
good practices that are emerging in this topic space.
2
© 2017, IBM Corporation Liberty z/OS Good Practices
Operations
A review of good practices related to operating your Liberty z/OS servers
3
© 2017, IBM Corporation Liberty z/OS Good Practices
(Content Under Design and Construction)
We are still in the process of designing and
constructing the content for this section.
4
© 2017, IBM Corporation Liberty z/OS Good Practices
Monitoring
A review of monitoring your Liberty z/OS servers
5
© 2017, IBM Corporation Liberty z/OS Good Practices
Monitor What?
1. The z/OS Address Space
• This involves using z/OS tools such as the SMF Type 30 record, or RMF
z/OS Started Task 1 • Liberty z/OS and its WLM classification function can assist with RMF monitoring by
classifying application activity more granularly
3 Application Activity
2. The Java Virtual Machine
• This involves monitoring the heap, garbage collection, and thread activity
• There are a number of tools that can assist with this.
Java Virtual Machine 2
3. Application Activity in the JVM
• The goal here is to understand better how the work requests are received and
processed by the application.
4 • For HTTP, Liberty z/OS has the SMF 120.11 record to assist
Clients
4. Availability and Performance
• The focus here is on adherence to Service Level Agreements with respect to up-
time, response time, and throughput.
6
© 2017, IBM Corporation Liberty z/OS Good Practices
SMF 30 and RMF Type 72 Monitoring
Use SMF Type 30 - Common Address Space Work
Liberty z/OS • Monitors activity at the address space level (CPU, I/O, etc.)
Server • Not granular; no insights into application activity
Liberty z/OS WLM Support and Use RMF Reporting
• The Liberty z/OS WLM support provides a way to assign a WLM transaction class (TC)
SMF 30
and associate a WLM report class to work
• You can control the granularity of the report class assignment:
SMF 72 + RMF • Least granular = wildcard the request filter to capture all requests in one TC
• More granular = identify specific application URIs and assign to different TCs
The SMF 120.11 v2 activity • Enable with this the zosWLM-1.0 feature
record for Liberty z/OS is
covered on a later chart • Requires the Angel Process and server READ to BBG.AUTHMOD.BBGZSAFM.ZOSWLM
7
© 2017, IBM Corporation Liberty z/OS Good Practices
Enabling Verbose Garbage Collection
1. Enabling verbose GC for a server
• Verbose GC is enabled with a line in the jvm.options file: -verbose:gc
Liberty z/OS
Server 2. Directing output
• Output goes to STDERR
• Output may be directed to JES, or to a file location
• If JES, consider spool partitioning:
//STDERR DD SYSOUT=*,FREE=CLOSE,SPIN=(UNALLOC,1M)
server.xml
• If file location, can specify in JCL proc (a commented-out DD statement has example), or
jvm.options you can add a separate line in jvm.options: -Xverbosegclog:/<path>/<file>
1
3. Performing analysis
Output • Any tool capable of consuming the verbose GC output may be used
Location • One such tool is IBM Support Assistant (free of charge)
2 Link http://www-01.ibm.com/support/docview.wss?uid=swg27039279
Analysis
3 Standard function of Java SDK
8
© 2017, IBM Corporation Liberty z/OS Good Practices
High-Level: Monitoring the JVM
Liberty z/OS Server
1. The Liberty z/OS server JVM
Java Virtual Machine 1 • The focus is on the JVM and the activity there
• If you want z/OS information, you can use existing tools (RMF)
Applications
2. Real-time data collection agent
• Something in the JVM must collect data and make it available
2 Data Collection Agent • Example: IBM Health Center API (comes with IBM SDK for z/OS)
• Example: Liberty "monitor-1.0" management beans
• Example: "OMEGAMON for JVM on z/OS" agent
3 • Goal of agent: collect data but be as lightweight as possible
Data Viewing Client 4 3. Data collected
• Garbage collection stats, thread usage, locking stats, etc.
4. Data viewing client
• Typically some form of viewing client is available.
9
© 2017, IBM Corporation Liberty z/OS Good Practices
IBM Health Center 1. The Liberty z/OS server
• The IBM Health Center agent can be enabled in any JVM that operates with the IBM Java SDK
• This implies any Liberty z/OS server: standalone, controller, member
Liberty z/OS Server
Standalone, collective 1 2. The Health Center agent
member, or a controller • This is provided with the IBM Java SDK
• It is configured with JVM options (jvm.options file)
Health Center Agent
3. The Health Center client
2 • This is a separately downloaded and installed Eclipse-based client (included with IBM Support Assistant)
• You configure it to access the target server with host:port
4. Data collected:
• Classes: Information about classes being loaded
• CPU: Processor usage for the application and the system on which it is running
• Environment: Details of the configuration and system of the monitored application
• Garbage Collection: Information about the Java heap and pause times
3
• I/O: Information about I/O activities that take place
• Locking: Information about contention on inflated locks
• Method Trace: Information about method use over time
4 • Native Memory: Information about the native memory usage
• Profiling: Provides a sampling profile of Java methods including call paths
• Threads: Information about the live threads of the monitored JVM
Agent provided with IBM Java SDK; client available for download no fee
Link https://www.ibm.com/support/knowledgecenter/en/SS3KLZ/com.ibm.java.diagnostics.healthcenter.doc/homepage/plugin-homepage-hc.html
10
© 2017, IBM Corporation Liberty z/OS Good Practices
IBM OMEGAMON for JVM on z/OS
JVM JVM
(agent) (Agent)
1. JVMs in your z/OS environment
• Any online JVM on z/OS (WAS, Liberty, CICS, IMS, DB2, standalone JVM)
1
JVM JVM
(Agent) ( -- )
2. Discovery of JVMs
• The OMEGAMON for JVM on z/OS agent for the LPAR has the ability to scan and detect JVMs
JVM JVM that are operating. That provides a view of what's active in your LPAR environment.
( -- ) (Agent) • Captures the jobname, ASID, subsystem type and basic details of the JVM.
OMEGAMON 3. Data collection
for JVM Agent
2 • The IBM Health Center agent collects data within the JVM; the OMEGAMON for JVM agent
collects additional information related to address space and CPU used.
• Data: GC statistics, thread details, lock details, GP, zIIP, zIIP offload to GP
3 • Also maintain historical data
4. Data viewing client
• Both a 3270 client as well as an Eclipse-based graphical client (Tivoli Enterprise Portal)
Client Display JVM
Eclipse information in a
consolidated view
3270
4
Very powerful enterprise JVM monitor. Licensed fee product.
Link http://www-03.ibm.com/software/products/en/omegamon-jvm-monitor
11
© 2017, IBM Corporation Liberty z/OS Good Practices
Server Monitor Function in AdminCenter
1. The Liberty z/OS server
Liberty z/OS Server
Standalone or a • Either a standalone server or a collective controller
1
collective controller
2. The adminCenter-1.0 feature
adminCenter-1.0 2 • Enables the AdminCenter
3. Browser access with security requirements satisfied
• ID has access to the "administrator" role (either basic or in SAF)
• Garbage collection stats, thread usage, locking stats, etc.
4. Data provided
• If monitor-1.0 feature not present, then: Used Heap Memory, Loaded
3 Classes, Active JVM Threads, CPU Usage
Browser • If monitor-1.0 feature present, then add (depending on what applications
are in the server): Active Sessions, Active Liberty Threads, Average
Data 4 Response Time, Average Wait Time, Request Count, Used Connections
Provided as a feature of Liberty (all platforms)
12 Link http://www.ibm.com/support/knowledgecenter/SSAW57_liberty/com.ibm.websphere.wlp.core.doc/ae/twlp_ui_explore_monitor.html#twlp_ui_explore_monitor__openchart
© 2017, IBM Corporation Liberty z/OS Good Practices
The "monitor-1.0" Feature
1. The Liberty z/OS server
Liberty z/OS Server
Standalone, collective
• The monitor-1.0 feature can be enabled in a standalone server, or a
1
member, or a controller collective controller or member.
monitor-1.0 2
2. The monitor-1.0 feature
• This enables the feature and provides a set of management beans that
collect and report statistics
3. Accessing the data
• You can access the data through the AdminCenter, or using a JMX client
4. Management beans:
3
Admin JMX
Center Client
Data 4
Provided as a feature of Liberty (all platforms)
Link http://www.ibm.com/support/knowledgecenter/SS7K4U_liberty/com.ibm.websphere.wlp.zseries.doc/ae/twlp_mon.html
13
© 2017, IBM Corporation Liberty z/OS Good Practices
The SMF 120.11 Version 2 Record
SMF is widely-used activity recording and analysis tool of z/OS
The SMF 120.11 Version 2 record for Liberty z/OS was made available in the
Liberty z/OS 16.0.0.2 fixpack. The record captures information about HTTP requests.
Server Information includes: server information, request information, WLM
classification information, network data information, optional user section.
1. Enabling SMF 120.11 Version 2 recording
1 server.xml • Provide the zosRequestLogging-1.0 feature to the featureManager list
• Allow the server ID (or group) READ to the BPX.SMF FACILITY class in SAF
2. Data output
SMF Output • The data is written to standard SMF buffers, then out to SMF data sets.
• The data sets are dumped for analysis
2
3. Analysis
Analysis • SMF records are formatted in a documented layout, so any analysis tool may be used
• If you use a vendor analysis tool, check with the vendor to see if the 120.11 v2 record support
3 is in place.
14 Link https://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP102655
© 2017, IBM Corporation Liberty z/OS Good Practices
Problem Determination
A review of performing effective problem determination of Liberty z/OS servers
15
© 2017, IBM Corporation Liberty z/OS Good Practices
(Content Under Design and Construction)
We are still in the process of designing and
constructing the content for this section.
16
