PowerVM Monitoring and Performance Management Tools

PowerVM Monitoring and Performance
management tools
—
Jose Luis Ortega
IBM - Certified Advanced Technical Expert - CATE V2 - V3
IBM - High-End Disk for Open Systems V2
IBM Certified Database Administrator - DB2 V9 & V10 DBA
IBM Certified System Administrator - Informix Dynamic Server
TOGAF 9 Foundation & Certified
PMI - Project Management
Master in Project Management
2018 IBM Systems
Technical University
April 30 - May 4
Orlando
Please note
IBM’s statements regarding its plans, directions, and intent are subject to change
or withdrawal without notice and at IBM’s sole discretion.
Information regarding potential future products is intended to outline our general

product direction and it should not be relied on in making a purchasing decision.
The information mentioned regarding potential future products is not a commitment, promise,
or legal obligation to deliver any material, code or functionality. Information about potential
future products may not be incorporated into any contract.
The development, release, and timing of any future features or functionality described for our
products remains at our sole discretion.
Performance is based on measurements and projections using standard IBM benchmarks in

a controlled environment. The actual throughput or performance that any user will
experience will vary depending upon many factors, including considerations such as the
amount of multiprogramming in the user’s job stream, the I/O configuration, the storage
configuration, and the workload processed. Therefore, no assurance can be given that an
individual user will achieve results similar to those stated here.
© Copyright IBM Corporation 2018 2

Please
complete
the session
survey!
Contents
PowerVM monitoring and performance management tools

HMC tools
Performance reports with topas and nmon
Optimizers
Monitoring tools for PowerVM

Abstract
Learn about the different virtualization management tools

that can be used to manage and monitor a POWER
virtualized environment. This talk does not give detailed
descriptions of all tools. Rather, you explore a list of
common tools with several usage examples.

Objectives
After this lecture, you should be able to:
• Enable the HMC to gather and view performance and capacity utilization Data
from the managed servers
• Describe how to generate historical performance reports using topas and
nmon
• Learn how to use and enable DSO (Dynamic System Optimizer), DPO
(Dynamic Platform Optimizer), and PowerVP to analyze performance
• List the basic virtualization management and monitoring functions of the
PowerVC application
• Learn about some free monitoring tools for PowerVM

Virtualization monitoring and performance management tools
Agenda: HMC tools HMC tools

Performance reports with topas and nmon
Optimizers

PowerVM monitoring tools
overview There are many tools which provide performance metrics. Each one of them at different
layers and purposes.
Monitoring strategy is VIOS and AIX tools:

important in any
• topas, nmon, entstat, lparstat, mpstat, and so on
computing
environment.
HMC:
• Utilization data and performance and capacity monitoring
Optimizers:
• Dynamic platform optimizer (DPO), Dynamic System Optimizer (DSO) and
PowerVP.
Enterprise tools:
• Tivoli monitoring, PowerVC
Freeware tools
© Copyright IBM Corporation 2018

HMC utilization data
• Available in HMC Classic GUI
• The HMC collects utilization data for memory and processor resources.
• First, you have to enable the collection:
• Select server  operations  utilization data  Change sampling rate.
• Select a collection interval (Recommended: 1 minute)

• Allow some time before start checking metrics.

Viewing utilization data from
HMC • To view utilization:
• select the server  operations  utilization data  view  select interval: snapshot,
Hourly, Daily, Monthly
• Select a utilization sample.

HMC utilization The first panel shows overall system utilization:
data
Choose a resource from the View menu to see more detail:

HMC utilization data examples
Shared processor pool
Partition utilization
Shared memory pool
Physical processor pool utilization:

HMC GUI HMC V8R8.1.0 added the new Performance and Capacity Monitoring (PCM)
function.
performance – Provides reports of a server’s current resource (CPU, Memory, IO) utilization
and capacity and historical data.
monitoring – When managing a server using VIOS 2.2.3.3 or later and Firmware 780 or
later, it also provides a view of the PowerVM and I/O utilization.
Data collection must be enabled for each managed system before the PCM task
can display information.
HMC Management > Change Performance

Monitoring Settings: Set number of days to retain
historical performance data.
Enable for specific managed

systems.

HMC PCM: Server
overview
Select managed system and

run the Performance task to
view information.

HMC PCM: Processor utilization trend
Processor utilization trend Memory utilization trend

HMC PCM: Network and storage utilization trend
Network utilization trend Storage utilization trend

Agenda: Performance reports HMC tools

with topas and nmon Performance reports with topas and nmon
Optimizers

AIX monitoring: topas recording and reporting:
topas reports – topasrec can be used to record local system data and cross LPAR data in binary format.
(1 of 2) Start recording:
# topasrec –L –o <output file> (local data)
# topasrec –C –o <output file> (cross partition data)
– Use smitty Stop_Recording to stop topas recording.
Listing running recording:

# topasrec –l
Pid User Fmt Start_time Path Status
245850 root bin 12:58:23,Oct06,2009 /usr/lpp/perfagent/
Running
Summary report:
# topasout –R summary <output file>
Detailed report:
# topasout –R detailed /etc/perf/topas_cec.080816
AIX monitoring:
topas reports SMIT panels are available to manage topas recordings.
(2 of 2) The following options are provided:
– Start or stop persistent recording (24x7)
– Choose the type of recording
• Binary/nmon style local recording
• CEC recording (cross partitions data)
– List available and completed recordings
– Generate reports on the completed recording

Example: Generate a report
with topasout Summary output of a CEC recording:
topasout -R summary topas_cec_cec_180222_1758.topas
The topasout
command or SMIT
panels (smit topas)
can be used to
convert recordings
into various text- Detailed output for a CEC recording:
based reports. topasout -R detailed topas_cec_cec_180222_1758.topas

Real time Cross-partition monitoring
with topas
• Analysis the whole machine reporting of the Logical Partitions (LPAR)
The Cross-Partition
view, also known as • If you are using a Virtual I/O Server (VIOS) and logged on as the padmin
CEC mode or whole user, type: topas -cecdisp
machine view.
Just type: topas -C

AIX monitoring: nmon:
nmon reports Installed by default in AIX
Ideal for benchmarks, proof-of-concepts, and problem analysis
Allows high-resolution recordings to be made while in monitoring mode
– Records samples at the interactive monitoring rate
– Start nmon, then use [ and ] (brackets) to start and end a recording.

AIX monitoring: nmon reports
nmon analyzer:
nmon analyzer:
 Excel spread sheet to graph nmon output
 Post processing tool to analyze performance
 Analyze files produced by topasout in nmon
format
 Cross partitions statistics cannot be analyzed
nmon consolidator: nmon consolidator:
 Excel preadsheet to graph lots of nmon output
from different machines/LPARs
 Need to use nmon consolidator if you want to
get a report for the entire machine.

Agenda: Advisors and HMC tools

Optimizers Performance reports with topas and nmon
Optimizers

Dynamic Optimization strategies:
System - Cache Affinity
Optimizer for - Memory Affinity
AIX – DSO: - Large page optimization (Stand-alone program or part of AIX EE)
- Data stream Prefetch Optimization (Stand-alone program or part of AIX EE)
Works to improve the memory and cache affinity for processes
– Affects the resource set (RSET) or scheduler resource allocation domain

(SRAD) where a process is running
– When a workload is running, DSO detects the process and applies each kind
of optimization strategy in turn to the workload to determine the best tuning
approach.
Best workloads are:
– Multi-threaded with longer-running threads (minutes rather than seconds)

– On larger servers with lots of processors and memory

Dynamic Supported since POWER7 AIX 7.1 TL1 – POWER8 – AIX 7.1 TL1 SP7
System – Active Memory Sharing, Live Partition Mobility, and disabling Enhanced
Affinity not supported
Optimizer for
– Supported with Active Memory Expansion
AIX – DSO:
– Check configuration:
# oslevel -s
7100-01-02-1150
# lslpp -L | grep -i optimi
bos.aso 7.1.1.2 C F Active System Optimizer
# lsconf | grep ^Proc
Processor Type: PowerPC_POWER7
Processor Implementation Mode: POWER 7
Processor Version: PV_7_Compat
Processor Clock Speed: 3000 MHz

Dynamic DSO runs as the aso daemon and uses the asoo command for
configuration.
System
– Start the kernel service: startsrc –s aso
Optimizer for
– Then activate ASO: asoo –op aso_active=1
AIX – DSO:
– View with lssrc –s aso
– asoo –a shows current status (enabled or disabled)
– asoo –h aso_active shows help information
Logs activity to /var/log/aso/*

– aso.log shows status and some reasons for status (such as hibernation)
– aso_process.log shows details such as processes modified and actions taken

Dynamic Once DSO/ASO starts, it keeps monitoring the system utilization every minute.
When a workload is running, DSO detects the process and applies each kind of
System optimization strategy in turn to the workload to determine the best tuning
approach. If the performance results are not as expected, DSO reverses its
Optimizer for actions immediately.
AIX – DSO:
More information on Redbook: IBM Power Systems Performance Guide: Implementing and Optimizing

Dynamic DSO optimizes threads within an OS instance (LPAR).
Platform DPO optimizes LPAR placement within a frame to improve memory
Optimizer - and processor affinity after partitions are up and running.
DPO Performs memory and processor relocations to optimize the layout

based on the server’s topology
Suboptimal placement can result from:
– DLPAR, LPM, Suspend and Resume, CHARM, or inability to determine optimal
placement
Benefits:
– Dynamic changes; no power or re-activations required
– Improves performance
• Great for cloud or highly virtualized environments
– Allows for dynamic optimization after mobility

Dynamic DPO is managed from the HMC.
Platform – Use the lsmemopt and optmem commands
Optimizer - – optmem can be an HMC scheduled operation

DPO The DPO operation uses the CPU resources of the LPAR it is
optimizing.
DPO is available on newer hardware, firmware, and operating systems
– Available for some POWER7 and POWER8 servers running firmware 760 and
higher
– HMC V7R780 or higher
– AIX 6.1 TL8, AIX 7.1 TL2, or later
– IBM i 7.1 TR6 or later
– Linux: partial support in RHEL7 and SLES12
– VIOS 2.2.2.0

Dynamic Query the current system score:
> lsmemopt -m msname -o currscore
Platform
Optimizer - curr_sys_score=89 A score of 100 is optimal.
Query the LPAR scores:

DPO
> lsmemopt -m msname -o currscore -r lpar
Determine the predicted system score:
> lsmemopt -m msname -o calcscore -r sys
curr_sys_score=100,predicted_sys_score=100,requested_lpar_i
ds=none,protected_lpar_ids=none
Optimize partitions:
> optmem –m msname –o start –t affinity

PowerVP Provides a graphical overview of hardware resource activity with high-
level views of:
– Node/socket activity
– Dedicated cores and shared processor pool cores
– Virtual processors assigned to cores
– Partition memory assigned to DIMMs
– Memory bus activity
– I/O bus activity
– Storage and network activity by partition
– System and partition Cycles-Per-Instruction
Provides visual indications when resource consumptions pass

thresholds
Provides visual understanding of how resources are being consumed
by the partitions
PowerVP
Not intended to replace traditional performance management products
Provides some detailed views if partition-level collectors are installed
A system-wide collector is required per managed system.
Provides real-time monitoring and historical data
Supported on POWER7 systems with either 770 or 780 firmware
(depending on the model)

PowerVP and DPO - Example
PowerVP allows you to see the DPO cache After running DPO
affinity improvements
Before running DPO

Agenda: Monitoring tools for HMC tools

PowerVM Performance reports with topas and nmon
Optimizers

Power Virtualization Center (PowerVC) - Functions
– Virtual image management and deployment
• Monitoring and provisioning of resources
• Dynamic resizing
• Image capture and deploy new VMs
– Resource pooling
• Manage pools of resource across multiple servers
– Dynamic VM relocation
• VM mobility including policy-based placement
– Infrastructure for cloud solutions
• VM management tool for IBM Cloud Manager with OpenStack
– Replacement for VMControl
• Easy to install and intuitive to use

Power Virtualization Center (PowerVC) - Functions
PowerVC manages:
- Hosts
- Virtual machines
- Storage
- Networks

IBM Tivoli Monitoring
Consolidated monitoring of physical
and virtual resources:

IBM Tivoli Four different System p agents are available for monitoring:
Monitoring Virtual I/O Server Premium Agent: provides mapping of storage and network
resources to the client LPAR, and provides storage and network utilization
statistics.
CEC Agent: The CEC agent provides overall CPU and memory utilization of the
frame for monitored partitions and provides CPU and memory utilization by
LPAR.
AIX Premium Agent: Detailed statistics about resource consumption for the
logical partition.
HMC Agent: The HMC agent provides health and availability of the HMC.
License for ITM:

• IBM Tivoli Monitoring can be ordered as a separate product.
• Included in AIX Enterprise Edition.

Freeware
Ganglia is an open source cluster performance monitoring tool:
monitoring
– Includes features like shared processor LPARs, entitlement, physical CPU
tools usage, and so on

Freeware
Lpar2rrd - CPU Cross Partition Graphs from HMC data :
monitoring
tools – LPAR CPU statistics and documentation tool for IBM POWER servers
– Alerting possibilities
– Produces historical CPU utilization graphs of LPARs and shared CPU usage
– Collects complete physical (HW) and logical configuration of all managed
systems and their LPARs including changes in their state and configuration

Freeware
monitoring
tools
Lpar2rrd – functions:

Summary
After completed this lecture, you should be able to:
• Enable the HMC to gather and view performance and capacity utilization Data
from the managed servers
• Describe how to generate historical performance reports using topas and
nmon
• Learn how to use and enable DSO (Dynamic System Optimizer), DPO
(Dynamic Platform Optimizer), and PowerVP to gain performance insight
• List the basic virtualization management and monitoring functions of the
PowerVC application
• Learn about some free monitoring tools for PowerVM

Thank you
Jose Luis Ortega
IBM - Certified Advanced Technical Expert - CATE V2 - V3

IBM - High-End Disk for Open Systems V2
IBM Certified Database Administrator - DB2 V9 & V10 DBA
IBM Certified System Administrator - Informix Dynamic Server
TOGAF 9 Foundation & Certified
PMI - Project Management
Master in Project Management
—
joseortegad@yahoo.com
+57-300-804-41-70

Notices and disclaimers
© 2018 International Business Machines Corporation. No part of this Performance data contained herein was generally obtained in a controlled,
document may be reproduced or transmitted in any form without isolated environments. Customer examples are presented as illustrations of
written permission from IBM. how those
U.S. Government Users Restricted Rights — use, duplication or customers have used IBM products and the results they may have
disclosure restricted by GSA ADP Schedule Contract with IBM. achieved. Actual performance, cost, savings or other results in other
operating environments may vary.
Information in these presentations (including information relating to
products that have not yet been announced by IBM) has been reviewed References in this document to IBM products, programs, or services does
for accuracy as of the date of initial publication and could include not imply that IBM intends to make such products, programs or services
unintentional technical or typographical errors. IBM shall have no available in all countries in which IBM operates or does business.
responsibility to update this information. This document is distributed
“as is” without any warranty, either express or implied. In no event, Workshops, sessions and associated materials may have been prepared by
shall IBM be liable for any damage arising from the use of this independent session speakers, and do not necessarily reflect the views of
information, including but not limited to, loss of data, business IBM. All materials and discussions are provided for informational purposes
interruption, loss of profit or loss of opportunity. IBM products and only, and are neither intended to, nor shall constitute legal or other guidance
services are warranted per the terms and conditions of the agreements or advice to any individual participant or their specific situation.
under which they are provided.
It is the customer’s responsibility to insure its own compliance with legal
IBM products are manufactured from new parts or new and used parts. requirements and to obtain advice of competent legal counsel as to
In some cases, a product may not be new and may have been previously the identification and interpretation of any relevant laws and regulatory
installed. Regardless, our warranty terms apply.” requirements that may affect the customer’s business and any actions the
customer may need to take to comply with such laws. IBM does not provide
Any statements regarding IBM's future direction, intent or product legal advice or represent or warrant that its services or products will ensure
plans are subject to change or withdrawal without notice. that the customer follows any law.

Notices and disclaimers
continued
Information concerning non-IBM products was obtained from the IBM, the IBM logo, ibm.com and [names of other referenced IBM
suppliers of those products, their published announcements or other products and services used in the presentation] are trademarks of
publicly available sources. IBM has not tested those products about this International Business Machines Corporation, registered in many
publication and cannot confirm the accuracy of performance, compatibility jurisdictions worldwide. Other product and service names might
or any other claims related to non-IBM products. Questions on the be trademarks of IBM or other companies. A current list of IBM
capabilities of non-IBM products should be addressed to the suppliers of trademarks is available on the Web at "Copyright and trademark
those products. IBM does not warrant the quality of any third-party information" at: www.ibm.com/legal/copytrade.shtml.
products, or the ability of any such third-party products to
interoperate with IBM’s products. IBM expressly disclaims all .
warranties, expressed or implied, including but not limited to, the
implied warranties of merchantability and fitness for a purpose.
The provision of the information contained herein is not intended to, and
does not, grant any right or license under any IBM patents, copyrights,
trademarks or other intellectual property right.


PowerVM Monitoring and Performance Management Tools

Uploaded by

Copyright:

Available Formats

PowerVM Monitoring and Performance Management Tools

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

PowerVM Monitoring and Performance Management Tools

Uploaded by

Copyright:

Available Formats

PowerVM Monitoring and Performance

Information regarding potential future products is intended to outline our general

Performance is based on measurements and projections using standard IBM benchmarks in

© Copyright IBM Corporation 2018 2

PowerVM monitoring and performance management tools

© Copyright IBM Corporation 2018 4

Learn about the different virtualization management tools

© Copyright IBM Corporation 2018 5

© Copyright IBM Corporation 2018 6

Agenda: HMC tools HMC tools

© Copyright IBM Corporation 2018 7

Monitoring strategy is VIOS and AIX tools:

© Copyright IBM Corporation 2018

• Select a collection interval (Recommended: 1 minute)

© Copyright IBM Corporation 2018

• Select a utilization sample.

© Copyright IBM Corporation 2018

Choose a resource from the View menu to see more detail:

© Copyright IBM Corporation 2018 11

Shared memory pool

Physical processor pool utilization:

© Copyright IBM Corporation 2018

HMC Management > Change Performance

Enable for specific managed

© Copyright IBM Corporation 2018 13

Select managed system and

© Copyright IBM Corporation 2018

© Copyright IBM Corporation 2018 15

© Copyright IBM Corporation 2018 16

Agenda: Performance reports HMC tools

© Copyright IBM Corporation 2018

# topasrec –C –o <output file> (cross partition data)

– Use smitty Stop_Recording to stop topas recording.

Listing running recording:

© Copyright IBM Corporation 2018 19

© Copyright IBM Corporation 2018 20

Just type: topas -C

© Copyright IBM Corporation 2018 21

© Copyright IBM Corporation 2018 22

© Copyright IBM Corporation 2018 23

Agenda: Advisors and HMC tools

© Copyright IBM Corporation 2018 24

System - Cache Affinity

Optimizer for - Memory Affinity

– Affects the resource set (RSET) or scheduler resource allocation domain

– Multi-threaded with longer-running threads (minutes rather than seconds)

© Copyright IBM Corporation 2018 25

# lslpp -L | grep -i optimi

bos.aso 7.1.1.2 C F Active System Optimizer

# lsconf | grep ^Proc

Processor Type: PowerPC_POWER7

Processor Implementation Mode: POWER 7

Processor Version: PV_7_Compat

Processor Clock Speed: 3000 MHz

© Copyright IBM Corporation 2018 26

Logs activity to /var/log/aso/*

© Copyright IBM Corporation 2018 27

© Copyright IBM Corporation 2018 28

DPO Performs memory and processor relocations to optimize the layout