SG 248426

Front cover
Implementing IBM VM
Recovery Manager
for IBM Power Systems
Dino Quintero
Jose Martin Abeleira
Adriano Almeida
Bernhard Buehler
Primitivo Cervantes
Stuart Cunliffe
Jes Kiran
Byron Martinez Martinez
Antony Steel
Oscar Humberto Torres
Stefan Velica
Redbooks
IBM Redbooks
Implementing IBM VM Recovery Manager for IBM

Power Systems
October 2019
SG24-8426-00
Note: Before using this information and the product it supports, read the information in “Notices” on
page vii.
First Edition (October 2019)
This edition applies to:

Virtual I/O Server (VIOS) V3.1.0.11
AIX 7.2.2.1
Hardware Management Console (HMC) V9 R1 920
KSYS V1.3.0.1
© Copyright International Business Machines Corporation 2019. All rights reserved.

Note to U.S. Government Users Restricted Rights -- Use, duplication or disclosure restricted by GSA ADP Schedule
Contract with IBM Corp.
Contents
Notices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii
Trademarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viii
Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix
Authors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix
Now you can become a published author, too! . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi
Comments welcome. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xii
Stay connected to IBM Redbooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xii
Chapter 1. Introduction to high availability and disaster recovery . . . . . . . . . . . . . . . . 1

1.1 General high availability and disaster recovery overview . . . . . . . . . . . . . . . . . . . . . . . . 2
1.1.1 General concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.1.2 Outage-related differences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.1.3 High availability and continuous availability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.1.4 Disaster recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
1.2 Restart technologies and clustering methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
1.2.1 Live Partition Mobility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
1.2.2 Simplified Remote Restart . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
1.2.3 IBM PowerHA SystemMirror . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
1.2.4 Comparison of solutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
1.3 VM Recovery Manager HA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
1.4 IBM VM Recovery Manager DR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
1.4.1 Implementing IP change scripts on AIX . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
1.4.2 Implementing IP change scripts on Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
Chapter 2. IBM VM Recovery Manager High Availability components . . . . . . . . . . . . 39

2.1 Overview and capabilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
2.1.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
2.1.2 Capabilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
2.2 Solution components and detailed architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
2.2.1 Simplified component model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
2.2.2 Refined component model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
2.2.3 KSYS test cluster example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
2.2.4 KSYS as a Resource Monitoring and Control subsystem. . . . . . . . . . . . . . . . . . . 52
2.2.5 KSYS daemon restart exercise . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
2.2.6 KSYS subsystem functional modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
2.2.7 Host monitor, Health Status Database, and related VIOS internals . . . . . . . . . . 114
2.2.8 VM Agent architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164
2.2.9 HVNCP protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199
2.3 Communication flows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 202
2.3.1 Discovery of a VM that is enabled for monitoring . . . . . . . . . . . . . . . . . . . . . . . . 203
2.3.2 Failure Detection Engine: QuickQuery and NeedsAttention flows . . . . . . . . . . . 234
2.3.3 Application Reporting flow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 262
Chapter 3. Planning and deploying IBM VM Recovery Manager High Availability for IBM
Power Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265
3.1 VM Recovery Manager HA requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 266
3.1.1 Software requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 266
3.1.2 Firmware requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 267
© Copyright IBM Corp. 2019. All rights reserved. iii

3.1.3 Installation and configuration requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . 267
3.1.4 Host group requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 268
3.1.5 HMC requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 268
3.1.6 Network requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269
3.1.7 GUI requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 270
3.2 VM Recovery Manager HA coexistence with other products . . . . . . . . . . . . . . . . . . . 271
3.2.1 IBM Power Virtualization Center . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 271
3.2.2 IBM PowerHA SystemMirror Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 271
3.2.3 PowerVM NovaLink . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 271
3.3 VM Recovery Manager HA restrictions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272
3.3.1 Migration restriction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272
3.3.2 VM Agent restrictions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272
3.3.3 GUI restrictions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 273
3.3.4 VM Recovery Manager HA filesets and structure . . . . . . . . . . . . . . . . . . . . . . . . 273
3.4 Installing the VM Recovery Manager HA solution . . . . . . . . . . . . . . . . . . . . . . . . . . . . 274
3.4.1 Installing the VIOS interim fix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275
3.4.2 Installing the KSYS software. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277
3.4.3 Installing VM Agents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 279
3.5 Configuring VM Recovery Manager HA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 281
3.5.1 Initializing the KSYS cluster . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283
3.5.2 Adding HMCs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 284
3.5.3 Adding hosts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 285
3.5.4 Creating host groups. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 286
3.5.5 Discovering and verifying the KSYS configuration . . . . . . . . . . . . . . . . . . . . . . . 289
3.6 Setting up HA policies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 297
3.6.1 HA monitoring policies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 297
3.6.2 Environment-level policies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 298
3.6.3 VM-level policies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 305
3.7 Setting up the VM Agent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 309
3.7.1 The ksysvmmgr command and utility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 310
3.7.2 Application Management Engine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 312
3.8 Setting contacts for event notification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 318
3.9 Uninstalling VM Recovery Manager HA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 320
3.9.1 Uninstalling the KSYS software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 320
3.9.2 Uninstalling VM Agents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 321
Chapter 4. IBM VM Recovery Manager High Availability GUI deployment . . . . . . . . 323

4.1 Installing the GUI filesets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 324
4.2 Configuring and discovering the VM Recovery Manager HA infrastructure by using the GUI
only . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 327
4.3 VM Recovery Manager HA dashboard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 335
4.3.1 KSYS cluster features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 335
4.4 Relevant move tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 345
4.4.1 VM Live Partition Mobility move test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 345
4.4.2 VM restart move test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 349
4.4.3 Host LPM move test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 353
Chapter 5. Test scenarios . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 357

5.1 Test environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 358
5.2 Linux virtual machine failures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 360
5.3 AIX VM failures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 363
5.4 Host failures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 366
Appendix A. Additional material . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 371
iv Implementing IBM VM Recovery Manager for IBM Power Systems

Locating the web material . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 371
Using the web material. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 371
System requirements for downloading the web material . . . . . . . . . . . . . . . . . . . . . . . 371
Downloading and extracting the web material . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 372
Related publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 373

IBM Redbooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 373
Online resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 373
Help from IBM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 373
Contents v
vi Implementing IBM VM Recovery Manager for IBM Power Systems
Notices
This information was developed for products and services offered in the US. This material might be available
from IBM in other languages. However, you may be required to own a copy of the product or product version in
that language in order to access it.
IBM may not offer the products, services, or features discussed in this document in other countries. Consult
your local IBM representative for information on the products and services currently available in your area. Any
reference to an IBM product, program, or service is not intended to state or imply that only that IBM product,
program, or service may be used. Any functionally equivalent product, program, or service that does not
infringe any IBM intellectual property right may be used instead. However, it is the user’s responsibility to
evaluate and verify the operation of any non-IBM product, program, or service.
IBM may have patents or pending patent applications covering subject matter described in this document. The
furnishing of this document does not grant you any license to these patents. You can send license inquiries, in
writing, to:
IBM Director of Licensing, IBM Corporation, North Castle Drive, MD-NC119, Armonk, NY 10504-1785, US
INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION “AS IS”

WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED
TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A
PARTICULAR PURPOSE. Some jurisdictions do not allow disclaimer of express or implied warranties in
certain transactions, therefore, this statement may not apply to you.
This information could include technical inaccuracies or typographical errors. Changes are periodically made
to the information herein; these changes will be incorporated in new editions of the publication. IBM may make
improvements and/or changes in the product(s) and/or the program(s) described in this publication at any time
without notice.
Any references in this information to non-IBM websites are provided for convenience only and do not in any
manner serve as an endorsement of those websites. The materials at those websites are not part of the
materials for this IBM product and use of those websites is at your own risk.
IBM may use or distribute any of the information you provide in any way it believes appropriate without
incurring any obligation to you.
The performance data and client examples cited are presented for illustrative purposes only. Actual
performance results may vary depending on specific configurations and operating conditions.
Information concerning non-IBM products was obtained from the suppliers of those products, their published
announcements or other publicly available sources. IBM has not tested those products and cannot confirm the
accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the
capabilities of non-IBM products should be addressed to the suppliers of those products.
Statements regarding IBM’s future direction or intent are subject to change or withdrawal without notice, and
represent goals and objectives only.
This information contains examples of data and reports used in daily business operations. To illustrate them
as completely as possible, the examples include the names of individuals, companies, brands, and products.
All of these names are fictitious and any similarity to actual people or business enterprises is entirely
coincidental.
COPYRIGHT LICENSE:
This information contains sample application programs in source language, which illustrate programming
techniques on various operating platforms. You may copy, modify, and distribute these sample programs in
any form without payment to IBM, for the purposes of developing, using, marketing or distributing application
programs conforming to the application programming interface for the operating platform for which the sample
programs are written. These examples have not been thoroughly tested under all conditions. IBM, therefore,
cannot guarantee or imply reliability, serviceability, or function of these programs. The sample programs are
provided “AS IS”, without warranty of any kind. IBM shall not be liable for any damages arising out of your use
of the sample programs.
© Copyright IBM Corp. 2019. All rights reserved. vii

Trademarks
IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business Machines
Corporation, registered in many jurisdictions worldwide. Other product and service names might be
trademarks of IBM or other companies. A current list of IBM trademarks is available on the web at “Copyright
and trademark information” at http://www.ibm.com/legal/copytrade.shtml
The following terms are trademarks or registered trademarks of International Business Machines Corporation,
and might also be trademarks or registered trademarks in other countries.
AIX® Parallel Sysplex® Redbooks®
DB2® POWER® Redbooks (logo) ®
GDPS® POWER8® Storwize®
IBM® POWER9™ System Storage™
IBM Cloud™ PowerHA® SystemMirror®
IBM Spectrum® PowerVM®
The following terms are trademarks of other companies:
The registered trademark Linux® is used pursuant to a sublicense from the Linux Foundation, the exclusive
licensee of Linus Torvalds, owner of the mark on a worldwide basis.
Windows, and the Windows logo are trademarks of Microsoft Corporation in the United States, other
countries, or both.
Red Hat, are trademarks or registered trademarks of Red Hat, Inc. or its subsidiaries in the United States and
other countries.
UNIX is a registered trademark of The Open Group in the United States and other countries.
Other company, product, or service names may be trademarks or service marks of others.
viii Implementing IBM VM Recovery Manager for IBM Power Systems

Preface
This IBM® Redbooks® publication describes the IBM VM Recovery Manager for Power
Systems, and addresses topics to help answer customers' complex high availability (HA) and
disaster recovery (DR) requirements for IBM AIX® and Linux on IBM Power Systems servers
to help maximize systems' availability and resources, and provide technical documentation to
transfer the how-to skills to users and support teams.
The IBM VM Recovery Manager for Power Systems product is an easy to use and
economical HA and DR solution. Automation software, installation services, and
remote-based software support help you streamline the process of recovery, which raises
availability and recovery testing, and maintains a state-of-the-art HA and DR solution. Built-in
functions and IBM Support can decrease the need for expert-level skills and shorten your
recovery time objective (RTO), improve your recovery point objective (RPO), optimize
backups, and better manage growing data volumes.
This book examines the IBM VM Recovery Manager solution, tools, documentation, and other
resources that are available to help technical teams develop, implement, and support
business resilience solutions in IBM VM Recovery Manager for IBM Power Systems
environments.
This publication targets technical professionals (consultants, technical support staff, IT

Architects, and IT Specialists) who are responsible for providing HA and DR solutions and
support for IBM Power Systems.
Authors
This book was produced by a team of specialists from around the world working at IBM
Redbooks, Austin Center.
Dino Quintero is an IT Management Consultant and an IBM Level 3 Senior Certified IT

Specialist with IBM Redbooks in Poughkeepsie, New York. Dino shares his technical
computing passion and expertise by leading teams developing technical content in the areas
of enterprise continuous availability (CA); enterprise systems management;
high-performance computing (HPC); cloud computing; artificial intelligence (AI), which
includes machine learning (ML) and deep learning (DL); and cognitive solutions. He also is a
Certified Open Group Distinguished IT Specialist. Dino holds a Master of Computing
Information Systems degree and a Bachelor of Science degree in Computer Science from
Marist.
Jose Martin Abeleira is a Senior Systems and Storage Administrator at DGI Uruguay. He is
a former IBMer, a prior IBM Redbooks author, a Certified Consulting IT Specialist, and an IBM
Certified Systems Expert Enterprise Technical Support for AIX and Linux in Montevideo,
Uruguay. He worked with IBM for 8 years and has 15 years of AIX experience. He holds an
Information Systems degree from Universidad Ort Uruguay. His areas of expertise include
IBM Power Systems, AIX, UNIX, and Linux; Live Partition Mobility (LPM), IBM PowerHA®
SystemMirror®, storage area network (SAN), and Storage on IBM DS line; and V7000,
HITACHI HUSVM, and G200/G400.
© Copyright IBM Corp. 2019. All rights reserved. ix

Adriano Almeida is a Senior Power Systems Consultant from IBM Systems Lab Services in
Brazil. He has worked at IBM for 20 years. His areas of expertise include IBM AIX,
PowerVM®, IBM Power Virtualization Center (IBM PowerVC), IBM PowerHA SystemMirror,
Linux, IBM Cloud™ Private, and SAP HANA on IBM Power Systems. He is an IBM Certified
Expert IT Specialist and IBM Certified Advanced Technical Expert on IBM Power Systems.
He has worked extensively on PowerHA SystemMirror, PowerVM, and Linux and AIX projects,
performing health checking, performance analysis and consulting on IBM Power Systems
environments, and technical project leadership. He holds a degree in computing technology
from the Faculdade de Tecnologia em Processamento de Dados do Litoral (FTPDL). He is a
coauthor of Exploiting IBM PowerHA SystemMirror V6.1 for AIX Enterprise Edition,
SG24-7841-01 and IBM PowerVM Best Practices, SG24-8062.
Bernhard Buehler is an IT Specialist in Germany. He works for IBM Systems Lab Services in
Nice, France. He has worked at IBM for 37 years and has 28 years of experience in AIX and
the availability field. His areas of expertise include AIX, Linux, PowerHA SystemMirror, HA
architecture, script programming, and AIX security. He is a co-author of several IBM
Redbooks publications. He is also a co-author of several courses in the IBM AIX curriculum.
Primitivo Cervantes is a Senior Certified IT Specialist at IBM US and provides HA and DR

services to clients on hardware and software components for AIX and Linux. He holds a
Bachelor of Science degree in Electrical Engineering from California State University, Long
Beach. Primitivo has worked for IBM for over 30 years with most of those years in the HA and
DR fields.
Stuart Cunliffe is a senior IBM Systems Consultant with IBM UK. He has worked for
IBM since graduating from Leeds Metropolitan University in 1995, and has held roles in
IBM Demonstration Group, IBM Global Technologies Services (GTS) System Outsourcing,
eBusiness hosting, and ITS. He currently works for IBM System Group Lab Services where
he specializes in IBM Power Systems, helping customers gain the most out of their Power
Systems infrastructure with solutions that offer PowerHA SystemMirror, PowerVM, IBM
PowerVC, AIX, Linux, IBM Cloud Private, IBM Cloud Automation Manager, and IBM DevOps.
Jes Kiran is a Development Architect for IBM VM Recovery Manager for HA and DR
products. He has worked in the IT industry for the last 18 years, and has experience in the
HA, DR, cloud, and virtualization areas. He is an expert in the Power, IBM Storage, and AIX
platforms.
Byron Martinez Martinez is an IT Specialist at IBM Colombia who provides services to

clients for hardware and software components for UNIX -like systems. He holds a degree in
electronics engineering from the National University of Colombia. During the last 4 years, he
has worked as deployment professional for IBM Power Systems. His areas of expertise
include Power Systems, AIX, PowerLinux, UNIX -based operating systems, PowerVM
Virtualization, LPM, IBM Hardware Management Console (HMC), NIM Server, IBM
Spectrum® Scale (formerly GPFS), IBM PowerHA SystemMirror, and SAN and Storage on
Brocade Communications Systems and IBM Storwize® storage systems.
Antony Steel is a Senior IT Specialist in Singapore. He has had over 25 years of field
experience in AIX, performance tuning, clustering, and HA. He worked for IBM for 19 years in
Australia and Singapore, and is now CTO for Systemethix in Singapore. He has co-authored
many IBM Redbooks publications about Logical Volume Manager (LVM) and PowerHA
SystemMirror, and helped prepare certification exams and runbooks for IBM Lab Services.
x Implementing IBM VM Recovery Manager for IBM Power Systems

Oscar Humberto Torres is an IBM Power Systems Consultant at IBM. He has been with IBM
since 2009. He has 16 years of experience in Power Systems and UNIX. He holds a degree in
systems engineering from Universidad Autonoma de Colombia. During the last 8 years, he
worked as a Power Systems Consultant deploying services and training courses. His areas of
expertise include Power Systems, SAP HANA on Power Systems, SUSE HA, AIX, LPM,
IBM Spectrum Scale (formerly GPFS), UNIX, IBM Cloud IBM PowerVC Manager, and IBM
PowerHA SystemMirror.
Stefan Velica is an IT Specialist who currently works for IBM GTS in Romania. He has 10
years of experience with IBM Power Systems. He is a Certified Specialist for IBM System p
Administration, HACMP for AIX, high-end and entry/midrange IBM DS Series, and Storage
Networking Solutions. His areas of expertise include IBM System Storage™, SAN, PowerVM,
AIX, and PowerHA SystemMirror. Stefan holds a bachelor’s degree in Electronics and
telecommunications engineering from the Polytechnic Institute of Bucharest.
Thanks to the following people for their contributions to this project:
Wade Wallace
IBM Redbooks, Austin Center
P I Ganesh, Denise Genty, Kam Lee, Luis Pizaña, Ravi Shankar, Thomas Weaver,
IBM Austin
Maria-Katharina Esser
IBM Germany
Luis Bolinches
IBM Finland
Javier Bazan Lazcano

IBM Argentina
Kelvin Inegbenuda
IBM West Africa
Ahmed (Mash) Mashhour

IBM Saudi Arabia
Shawn Bodily
Clear Technologies, an IBM Business Partner
Now you can become a published author, too!

Here’s an opportunity to spotlight your skills, grow your career, and become a published
author—all at the same time! Join an IBM Redbooks residency project and help write a book
in your area of expertise, while honing your experience using leading-edge technologies. Your
efforts will help to increase product acceptance and customer satisfaction, as you expand
your network of technical contacts and relationships. Residencies run from two to six weeks
in length, and you can participate either in person or as a remote resident working from your
home base.
Find out more about the residency program, browse the residency index, and apply online at:
ibm.com/redbooks/residencies.html
Preface xi
Comments welcome
Your comments are important to us!
We want our books to be as helpful as possible. Send us your comments about this book or
other IBM Redbooks publications in one of the following ways:
򐂰 Use the online Contact us review Redbooks form found at:
ibm.com/redbooks
򐂰 Send your comments in an email to:
redbooks@us.ibm.com
򐂰 Mail your comments to:
IBM Corporation, IBM Redbooks
Dept. HYTD Mail Station P099
2455 South Road
Poughkeepsie, NY 12601-5400
Stay connected to IBM Redbooks

򐂰 Find us on Facebook:
http://www.facebook.com/IBMRedbooks
򐂰 Follow us on Twitter:
http://twitter.com/ibmredbooks
򐂰 Look for us on LinkedIn:
http://www.linkedin.com/groups?home=&gid=2130806
򐂰 Explore new Redbooks publications, residencies, and workshops with the IBM Redbooks
weekly newsletter:
https://www.redbooks.ibm.com/Redbooks.nsf/subscribe?OpenForm
򐂰 Stay current on recent Redbooks publications with RSS Feeds:
http://www.redbooks.ibm.com/rss.html
xii Implementing IBM VM Recovery Manager for IBM Power Systems

1
Chapter 1. Introduction to high availability

and disaster recovery
This chapter describes high availability (HA) and disaster recovery (DR), including
technologies that are used with clusters to restart the servers. In addition, this chapter
introduces the IBM VM Recovery Manager for HA and DR.
The following topics are described in this chapter:

򐂰 General high availability and disaster recovery overview
򐂰 Restart technologies and clustering methods
򐂰 VM Recovery Manager HA
򐂰 IBM VM Recovery Manager DR
© Copyright IBM Corp. 2019. All rights reserved. 1

1.1 General high availability and disaster recovery overview
This section describes HA and DR topics as a high-level overview. Before providing details,
the terminology that is used throughout this publication is defined. These definitions might not
match the information that you find on the internet because sometimes different meanings
exist for the same phrase.
Availability The ability of a service component to perform its
required function at a stated instant or over a stated
period. It is expressed as the availability ratio, for
example, the proportion of time that the service is
available for use by the customers within the agreed
service hours.
Continuous availability (CA) The attribute of a system to deliver nondisruptive
service to the user 365 days a year, 24 hours a day
(assuming no planned or unplanned outages exist).
Figure 1-1 shows this relationship.
Note: In most cases when people talk about HA, they mean CA.
Figure 1-1 Continuous availability, HA, and continuous operation relationship
Continuous operations (CO) The ability of a system to continuously operate and

mask planned outages from users. It uses redundant
hardware and software components (often clustering)
along with nondisruptive maintenance and change
management procedures.
2 Implementing IBM VM Recovery Manager for IBM Power Systems

HA The ability of a system to provide access to
applications regardless of local failures, whether these
failures are in the business processes, the physical
facilities, or the IT hardware or software. The aim is to
mask unplanned outages from users.
DR The ability to continue processing with a minimal loss
of integrity of a data center at a different site if a
disaster destroys the primary site or otherwise renders
it inoperable. With a DR solution, processing resumes
at a different site and on different hardware.
Recovery time objective (RTO) The total time that you can allow for your systems to be
offline. How long can you afford to be without your
systems?
Recovery point objective (RPO) The point at which data is restored to if there is a
disaster. When it is recovered, how much data can you
afford to re-create?
Business recovery objective (BRO) The time within which business processes must be
recovered, along with the minimum staff, assets, and
services required within this time.
Network recovery objective (NRO) Indicates the maximum time that the network
operations can be down for a business. How long can
you afford to take to switch over the network?
1.1.1 General concepts

From a service perspective, three concepts exist:
Active/Inactive The service is running on one system. One or more physical backup
systems exist. Backup systems are not powered on or virtual system
definitions are not defined. Only a physical connection to the data
exists.
Active/Passive The service is running on one system at a time, and one or more
backup systems can take over the service. An active OS is running on
the backup systems.
Active/Active The same service is running concurrent in multiple systems. The
application has simultaneous access to the same data on all systems.
An active/active solution requires that the application is aware of the redundant components;
an active/passive and an active/inactive solution are exempt from this requirement.
From an operating system or system perspective, two concepts also exist:

򐂰 Internal-managed (In some documentation, it is called application-based HA or
application-based DR.)
򐂰 External-managed (In some documentation, it is called virtual machine (VM)-based HA or
VM-based DR.)
Chapter 1. Introduction to high availability and disaster recovery 3

Figure 1-2 illustrates these two concepts. Detailed differences are described in the following
sections.
Figure 1-2 Internal- and external-managed HA and DR
Internal-managed
In an internal-managed environment, the operating system or the application has information
that it is part of a cluster. In most cases. HA components are installed that check whether the
cluster partners, hardware, and software components are reachable or available.
Highlights of this environment include these items:

򐂰 Can be used on physical or virtual systems.
򐂰 Has a hot-standby topology.
򐂰 Is based on independent operating systems.
򐂰 Cluster logic is part of the OS or application.
򐂰 Rolling migration is possible.
򐂰 Supports active/passive and active/active architecture within the cluster.
External-managed
In an external-managed environment, the operating system has no information about whether
it is part of a cluster.
Highlights of this environment include these items:

򐂰 Requires a virtual system.
򐂰 Has a cold-standby topology.
򐂰 Is based on a single operating system.
򐂰 Cluster logic is outside of the OS or application.
򐂰 Supports only an active/inactive architecture within the cluster.
򐂰 Offers reduced license costs.
1.1.2 Outage-related differences

Independent of the selected architecture (internal- or external-managed), there are two main
forms of outages: planned outages and unplanned outages. In this section, we describe the
differences between internal and external-managed architectures regarding these two types
of outages.

Planned outages
Before looking at the differences between internal- and external-managed environments, we
look at the similarities between the two of them.
As shown in Figure 1-3, an update or upgrade of the underlying hardware or firmware (FW)
can be hidden from the user. The outage period is the time that it takes to move to the new
hardware and FW environment. In both cases, you can make the change on the backup
system while the application is still running on the production environment. From a planning
point of view, you need a second downtime to move back to the production environment.
Make the same hardware and FW changes on the production environment before you move
the application back to it.
In an internal-managed environment, the update of the operating system and application can
be hidden from the user because you have multiple independent installations. As described in
1.1.1, “General concepts” on page 3, the update can be made on the backup system while
the application is still running on the current environment, as shown in Figure 1-3.
In an external-managed environment, you need a longer downtime to do an update to the

operating systems or application because you have only one operating system and one
installation of the application. You cannot hide the whole time that is needed to make the
updates from the user, as shown in Figure 1-3.
Figure 1-3 Software and hardware maintenance-dependent action times
Unplanned outages
As listed in Figure 1-4 on page 6, the unexpected outage of the hardware, operating system,
or application can in some cases be covered in a similar way for both architectures. However,
if data corruption occurs, the outage can be different.
Because the data in both cases is available only once, the effect of data corruption is the
same. This is the worst case scenario because it requires a restore of your data from your
backup media. This restoration can take a long time, and your recovery point might be old.
The outage time also can differ based on the cause of the outage. An outage of one of the
components is detected much faster than data corruption on one of the components. The
outage detection time does not differ much regarding the selected architecture.

In an internal-managed environment, data corruption in the operating system or the
application code can be partly hidden from the user. What is visible is the fallover time that is
needed to restart the application on the backup system, as shown in Figure 1-4.
In an external-managed environment, data corruption in the operating system or the

application code cannot be hidden from the user. A simple restart does not help in this case.
This situation requires a restore of your operating system, application code, or both. This task
requires more time than a restart, as shown in Figure 1-4.
Figure 1-4 Failure-dependent action times
1.1.3 High availability and continuous availability

In most cases, when the topic is about HA, the topic is really about CA. In this section, all
references to HA are about CA.
Measuring availability
In many cases, people measure availability in percentage (%) numbers. You might find
comments that are related to the three nines or four nines of HA. Table 1-1 shows what these
percentage values mean in hours or minutes.
Table 1-1 Availability measuring

Measurement Availability % Downtime/year Downtime/month Downtime/week
One nine 90 36.5 days 72 hours 16.8 hours
Two nines 99 3.65 days 7.2 hours 1.68 hours
Three nines 99.9 8.76 hours 43.2 min. 10.1 min.
Four nines 99.99 52.56 min. 4.32 min. 1.01 min.
Five nines 99.999 5.26 min. 25.9 sec. 6.05 sec.
Six nines 99.9999 31.55 sec.a 2.59 sec. 0.605 sec.

a. https://en.wikipedia.org/wiki/High_availability

Do these percentage numbers help for HA planning? Here is a fictional situation in which a
customer has the following guidelines:
1. It is acceptable if the system goes offline for 1 hour every day.
2. If the system is down for 8 hours continuously, the company will be bankrupt.
What does this mean in terms of the annual availability percentages?

For guideline 1 In the worst case scenario, you might have up to 365 hours of outage but
happy customers because you have an annual availability percentage of
95.833%.
For guideline 2 In this case, if you have a single outage extending to and beyond 8 hours,
then the customers are unhappy even though the annual availability
percentage is 99.909%.
Planing for availability

The two important items to consider when planning for HA and DR solutions include RTO and
RPO.
RPO means how much data may be lost without a major business impact or how far can you
go back in your backups to get consistent data.
Figure 1-5 illustrates the three major RTO components:

Outage How long until the outage is recognized?
Fallover How long until fallover to the backup system occurs?
Restart How long until the service is restarted on the same or different hardware?
Depending on the selected solution and the used application, these items have different
values.
Figure 1-5 RPO and RTO content for high availability

Other considerations for high availability
Several extra items must be considered when planning for a highly available environment, as
shown in Figure 1-6.
Figure 1-6 Other considerations for continuous availability
The following list includes but is not limited to the items that are represented in Figure 1-6:
򐂰 People
The knowledge and experience of the system administrators managing the environment is
important to the stability and usability of the availability solution.
򐂰 Data
An important aspect is that critical data must be redundant (by using a RAID 1, 5, or 6
setup) and backups must exist.
򐂰 Hardware
The hardware must be able to handle the expected workload because a slow responding
system is as bad as a non-existing one.
򐂰 Software
The software (application) must be able to automatically recover after a system crash.
򐂰 Environment
The location of your data center is important, and it must not be too close to the coastline
or a river due to the high risk of flooding. Additionally, electrical power support must be
redundant.

򐂰 Networking
Also important is configuring an internal redundant network that is combined with a
redundant internet connection.
򐂰 Systems management
Here, especially good change management control is important. You also must have
organized incident and problem management.
1.1.4 Disaster recovery

This section describes some components of DR from an IT perspective.
Recovery plan: DR is a small component of the overall business recovery plan.
The main purpose of DR is to have a defined and possibly automated procedure for recovery
from a major business impact, such as an outage of the whole data center as the result of an
earthquake, flooding, or storm.
Planing for disaster recovery

From a risk assessment perspective, you have the same challenges as described in
“Measuring availability” on page 6.
RTO and RPO for DR are normally different from the RPO and RTO values for availability.
Note: RPO means how much data can be lost without suffering a major impact to the
business or how far back in time that you must go to get consistent data. The RPO for DR
is greater than the RPO for availability. The worse case is the time between the disaster
and when the last successful backup was completed.
Figure 1-7 on page 10 shows the RTO summary of the four major components:
Outage How long until the outage is recognized.
Prepare to repair How long until a decision is made whether a DR situation must be
declared.
Note: Declaring a disaster is often a management decision.
Repair or fallover How long until fallover to the backup system occurs, or how long until
the original system is repaired or reinstalled.
Minimum service How long until full service is available on the same or different
hardware.

Depending on the selected solution and the application that are used, these items have
different values.
Figure 1-7 RPO and RTO content for disaster recovery
As for availability, a DR solution must also address the items that are described in “Other
considerations for high availability” on page 8.
Other considerations for disaster recovery

In addition to the considerations that are listed in “Other considerations for high availability” on
page 8, check the distance between your data centers.
In a few publications, the recommended minimum distance is listed as 5 km. However, this is
not sufficient when you think about major natural events such as earthquakes, hurricanes,
volcano eruptions, or other disasters. In such cases, even 100 km cannot be sufficient.
1.2 Restart technologies and clustering methods

Many technologies and solutions are available for IBM Power Systems to increase logical
partition (LPAR) availability and application availability. In this section, we cover many of these
options and highlight which solution is best suited to which environment.
1.2.1 Live Partition Mobility

Live Partition Mobility (LPM) is a component of the PowerVM Enterprise Edition hardware
feature that you can use to move AIX, IBM i, and Linux LPARs from one system to another
one. The mobility process transfers the system environment, which includes the processor
state, memory, attached virtual devices, and connected users.
Active partition mobility moves AIX, IBM i, and Linux LPARs that are running, including the
operating system and applications, from one system to another. The LPAR and the
applications running on that migrated LPAR do not need to be shut down.
Inactive partition mobility moves a powered-off AIX, IBM i, or Linux LPAR from one system to
another.

Partition mobility provides systems management flexibility and is designed to improve system
availability. For example:
򐂰 You can avoid planned outages for hardware or FW maintenance by migrating LPARs to
another server and then performing the maintenance. Partition mobility can help because
you can use it to work around scheduled maintenance activities.
򐂰 You can avoid downtime for a server upgrade by migrating LPARs to another server and
then performing the upgrade to continue your work without disruption.
򐂰 If a server indicates a potential failure, you can migrate its LPARs to another server before
the failure occurs. Partition mobility can help avoid unplanned downtime.
򐂰 You can consolidate workloads running on several small, underused servers onto a single
large server.
򐂰 You can move workloads from server to server to optimize resource use and workload
performance within your computing environment. With active partition mobility, you can
manage workloads with minimal downtime.
򐂰 For some systems, you can move applications from one server to an upgraded server by
using IBM PowerVM Editions LPM or the AIX Live Application Mobility software without
affecting availability of the applications.
However, while partition mobility provides many benefits, it does not provide the following
functions:
򐂰 Partition mobility does not provide automatic workload balancing.
򐂰 Partition mobility does not provide a bridge to new functions. LPARs must be restarted and
possibly reinstalled to take advantage of new features.
When an LPAR is moved by using LPM, a profile is automatically created on the target server
that matches the profile on the source server. The partition’s memory is then copied
asynchronously from the source system to the target server, which creates a clone of a
running partition. Memory pages that changed on the partition (“dirty” pages) are recopied.
When a threshold is reached that indicates that enough memory pages were successfully
copied to the target server, the LPAR on that target server becomes active, and any remaining
memory pages are copied synchronously. The original source LPAR is then automatically
removed.
Because the Hardware Management Console (HMC) always migrates the last activated
profile, an inactive LPAR that has never been activated cannot be migrated. For inactive
partition mobility, you can either select the partition state that is defined in the hypervisor or
select the configuration data that is defined in the last activated profile on the source server.
There are many prerequisites that must be met before an LPAR can be classified as LPM
ready. You must verify that the source and destination systems are configured correctly so
that you can successfully migrate the mobile partition from the source system to the
destination system. You must verify the configuration of the source and destination servers,
the HMC, the Virtual I/O Server (VIOS) LPARs, the mobile partition, the virtual storage
configuration, and the virtual network configuration.
For more information about preparing for partition mobility, see IBM Knowledge Center.
In addition to the minimum required FW, HMC versions, and VIOS versions, the high-level
prerequisites for LPM include:
򐂰 The LPAR to be moved can be in an active or inactive state.
򐂰 The source and target systems must be in an active state.
򐂰 The source and target systems VIOSs that provide the I/O for the LPAR must be active.

򐂰 The source and the target systems must have a VIOS that is marked as the mover service
partition (MSP).
򐂰 The source and target mover server partitions must be active.
򐂰 Resource Monitoring and Control (RMC) network communication must exist between the
mobile partitions, service processor, and the MSPs.
򐂰 The logical memory block (LMB) size must be the same on the source and the target
system.
򐂰 The target system must have enough resources (processors and memory) to host the
partition.
򐂰 The target system VIOSs must provide the networks that are required for the LPAR.
򐂰 The source and the target system VIOSs must both be able to see all the LPARs storage
logical unit numbers (LUNs).
򐂰 All physical I/O devices must be removed from the partition before it can be moved.
򐂰 For IBM i partitions, the restricted I/O partition cannot be set.
You can use the HMC command miglpar or GUI to validate whether an LPAR is LPM-capable
without performing the actual migration. For more information, see IBM Knowledge Center.
Initiating LPM can be done by using the HMC GUI, the HMC command-line interface (CLI),
IBM Power Virtualization Center (IBM PowerVC), IBM Lab Services LPM Automation Toolkit,
and IBM VM Recovery Manager. However, there are a few restrictions:
򐂰 The HMC GUI can initiate only a single LPM migration at a time.
򐂰 The HMC CLI can initiate only a single LPM migration at a time, unless scripted.
򐂰 IBM PowerVC can initiate only a single LPM migration at a time, unless frame evacuation
is used.
򐂰 IBM PowerVC cannot migrate inactive LPARs.
You can confirm whether an IBM Power Systems server is licensed for LPM by going to the
HMC and clicking Resources → All Systems → System Name → Licensed Capabilities.
Figure 1-8 shows a Power Systems server that is LPM-capable.
Figure 1-8 PowerVM Live Partition Mobility Capable

1.2.2 Simplified Remote Restart
Simplified Remote Restart (SRR) is a component of the PowerVM Enterprise Edition
hardware feature that can restart AIX, IBM i, and Linux LPARs on a different physical server if
the original server is no longer active. If the original physical server (source) has an error that
causes an outage, you can restart the LPARs on another (target) server.
If the source server has a physical fault that requires fixing, then you can use SRR to recover
the key LPARs in a more timely manner. Also, if the source server is restarting, it might take
longer to start the server, in which case you can use SRR for faster reprovisioning of the
partition. This operation can complete quicker compared to restarting the server that failed
and then restarting the partition.
SSR (available from IBM POWER8® processor-based systems that use FW level 8.2.0 or
later and HMC version 8.2.0 or later) removes the need to use a reserved storage device that
is assigned to each partition. As a best practice, use SSR.
Here are the characteristics of SSR:

򐂰 SSR is not a suspend and resume or migration operation of the partition that preserves
the active running state of the partition. During the remote restart operation, the LPAR is
shut down and then restarted on a different system.
򐂰 SSR preserves the resource configuration of the partition. If processors, memory, or I/O
are added or removed while the partition is running, the remote restart operation activates
the partition with the most recent configuration.
When an LPAR is restarted by using SRR, a new profile is automatically created on the target
server that matches the profile on the source server. That new profile is then mapped to the
storage LUNs that were being used by the original partition (the partition is inactive). The new
profile on the target server is then activated and the partition is again active. When the source
server becomes active, you must remove the old profile to ensure that the partition is not
accidentally restarted on that server (it restarts automatically). The automatic cleanup runs
without the force option, which means that if a failure occurs during the cleanup (for example,
RMC communications with the VIOS fails), the LPAR is left on the original source server and
its status marked as Source Side Cleanup Failed.
As with LPM, there are many prerequisites that must be met in order for SRR to work.
However, if LPM is not working for an LPAR, then SRR does not work.
Other than the minimum required FW, HMC versions, and VIOS versions, the high-level SRR
prerequisites include:
򐂰 The source system must be in a state of Initializing, Power Off, Powering Off, No
connection, Error, or Error - Dump in progress.
򐂰 The source systems VIOSs that provide the I/O for the LPAR must be inactive.
򐂰 The target system must be in an active state.
򐂰 The target systems VIOSs that provide the I/O for the LPAR must be active.
򐂰 The LPAR that will be restarted must be in an inactive state.
򐂰 The LMB size is the same on the source and the target system.
򐂰 The target system must have enough resources (processors and memory) to host the
partition.
򐂰 The target system VIOSs must be able to provide the networks that are required for the
LPAR.

򐂰 The source and the target systems VIOSs must both be able to see all the LPARs storage
LUNs.
򐂰 All physical I/O devices must be removed from the partition before it can be moved.
򐂰 For IBM i partitions, the restricted I/O partition cannot be set.
In order for an LPAR to perform a remote restart operation, a flag must be set against it by
using the HMC. It is possible to view and set this flag dynamically by using the HMC.
With HMC V9 R1.910.0, many SRR enhancements were announced:

򐂰 Remote Restart a partition with reduced or minimum CPU/Memory on the Target system
When a system outage occurs and you want to restart all the partitions on another system
to reduce the downtime, there might be scenarios where the target system does not have
enough capacity to host all the partitions, but some partitions can be run with reduced
resources (for example, development or test workloads versus production workloads). You
can now restart a partition with reduced resources on target system.
򐂰 Remote Restart by choosing different Virtual Switch on Target system
You can validate or perform the remote restart operation for an LPAR when you want to
start the LPAR with a different virtual switch on the target server than the virtual switch that
the LPAR was assigned on the source server.
򐂰 Remote Restart the partition without powering on the partition on Target system
With this option, you can prevent an LPAR from being started during the remote restart
operation. You can use this option in cases where you want to check the configuration on
the target system before powering on the partition. All the other steps that are performed
during a remote restart operation are performed in this case except for powering on the
partition.
򐂰 Remote Restart the partition for test purposes when the source managed system is in the
Operating or Standby state
Remote restart is supported when a system has failed. If you want to validate whether a
remote restart operation works in case of a system failure, you can always use the validate
option. However, if you want to go one step further and test restarting the partition on
another system, you can do that by using the test option that is introduced with HMC V9
R1.910.0. The source partition must be in the shutdown state to use the test option for
remote restart.
򐂰 Display the partition configuration information
HMC collects and persists the configuration data that is required for restarting a partition.
You can now use the lsrrstartlpar command to view the persisted configuration
information of all the LPARs that support SRR.
򐂰 Remote Restart by way of the Rest API
You can also perform the remote restart operation by using the Representational State
Transfer (REST) API:
https://<<HMCIP>>:12443/rest/api/uom/ManagedSystem/<ManagedSystem_UUID>LogicalP
artition/<<PARTITION_UUID>>/do/RemoteRestart
All the options and overrides are added to the REST API as well.

You can initiate SRR by using the HMC CLI, REST APIs, IBM PowerVC, IBM Lab Services
LPM and SRR Automation Toolkit, and IBM VM Recovery Manager. However, there are
limitations:
򐂰 The HMC GUI cannot be used to initiate partition restarts with SRR.
򐂰 The HMC CLI (rrstartlpar) can initiate only a single partition restart with SRR.
򐂰 Only up to 32 concurrent SRR operations can occur for a destination server (including
concurrent rrstartlpar commands).
򐂰 IBM PowerVC can initiate a single partition restart or a whole frame restart, but not a
subset of SRR-capable partitions.
򐂰 IBM PowerVC can optionally be configured to automatically launch SRR for all SRR
capable partitions on a failed server.
You can confirm whether an IBM POWER® server is licensed for SRR by opening the HMC
and selecting Resources → All Systems → System Name → Licensed Capabilities, as
Figure 1-9 PowerVM Simplified Remote Restart Capability
1.2.3 IBM PowerHA SystemMirror

IBM PowerHA SystemMirror for AIX, IBM i, and Linux is a separate licensed product that
provides HA clusters on IBM Power Systems servers. A PowerHA cluster must contain a
minimum of two LPARs (called nodes) that communicate with each other by using heartbeats
and keepalive packets. The cluster contains many resources, such as IP addresses, shared
storage, and application scripts, that are grouped to form a resource group.
If PowerHA detects an event within the cluster, it automatically acts to ensure that the
resource group is placed on the most appropriate node in the cluster to ensure availability. A
correctly configured PowerHA cluster after setup requires no manual intervention to protect
against a single point of failure, such as failures of physical servers, nodes, applications,
adapters, cables, ports, network switches, and storage area network (SAN) switches. The
cluster can also be controlled manually if the resource groups must be balanced across the
clusters or moved for planned outages.
PowerHA Enterprise Edition also provides cross-site clustering where shared storage might
not be an option but SAN-based replication is available. In this environment, PowerHA uses
the remote copy facilities of the storage controllers to ensure that the nodes at each site have
access to the same data, but on different storage devices. It is possible to combine both local
and remote nodes within a PowerHA cluster to provide local HA and cross-site DR.

Clusters within PowerHA can be configured in many ways:
򐂰 Active/Passive: One node in the cluster runs the resource group, and its partners are in
standby mode waiting to take on the resources when required. The passive nodes in the
cluster must be running in order for them to participate in the cluster. Although the standby
nodes do not have to have the full processor and memory allocation when in passive
mode, they must still be active.
򐂰 Active/Active: All nodes in the cluster are running a resource group but are also the
standby node for another resource group in the cluster. Many resources groups can be
configured within a cluster, so how they are spread out across the nodes and in which
order they move is highly configurable.
򐂰 Concurrent: All nodes in the cluster run the same resource group.
1.2.4 Comparison of solutions

In 1.2, “Restart technologies and clustering methods” on page 10, we described LPM, SRR,
and PowerHA, which are three different solutions that each have their own advantages and
disadvantages. In this section, we show where each solution fits in terms of providing HA or
DR for applications running on your IBM Power Systems servers. We also highlight where
solutions are not appropriate for your particular environment. IBM VM Recovery Manager HA
uses the functions of LPM and SRR to perform its work, so you can see how it fits into the
solutions.
Ease of use
Both LPM and SRR are relatively simple to configure and use. Although this book does not
describe how to enable VM to be LPM- and SRR-ready, you can do this task in a few simple
steps and verify your success online. Both LPM and SRR are included in IBM PowerVM
Enterprise Edition so no extra software is required. You can start LPM by using the HMC GUI,
the HMC CLI, IBM PowerVC, IBM Lab Service, LPM Automation Toolkit, or IBM VM Recovery
Manager. You can start SSR by using the HMC CLI, REST APIs, IBM PowerVC, IBM Lab
Services LPM and SRR Automation Toolkit, or IBM VM Recovery Manager.
IBM PowerHA is a more complex product to set up because it requires a license and you must
install the cluster code on all nodes in the cluster. It also requires an experienced
IBM PowerHA consultant to design and implement a successful cluster. After the cluster is
active, it must be administered by an experience PowerHA administrator.
VM Recovery Manager for HA is simpler to configure than IBM PowerHA, but relies on LPM
and SRR being configured to function correctly.
Availability, recovery times, and recovery points

LPM, SRR, VM Recovery Manager for HA, and standard PowerHA must see the same shared
storage to work successfully, so the recovery points must be the same. For VM Recovery
Manager for DR and cross-site PowerHA, storage replication is needed and the recovery
point depends on how storage replication is configured.
The key differentiators in this section are availability and recovery times as shown in Table 1-2
on page 17. You can see how each product differs in terms of storage requirements,
automation, server status, and outages.

Table 1-2 Solutions comparison matrix
Solution Storage Automated Source server Source VIOS VM/Application
Requirements failover status status outage
LPM Shared No Active Active No (if the VM is

active)
SRR Shared No Inactive Inactive Yes
VM Recovery Shared Yes Active or Inactive Active or Inactive Only if a

Manager HA server/VM
outage occurs.
PowerHA Shared Yes Active or Inactive Active or Inactive Yes

Standard
PowerHA EE Remote Copy Yes Active or Inactive Active or Inactive Yes
VM Recovery Remote Copy No Active or Inactive Active or Inactive Yes

Manager DR
Note: SRR can be automated by using IBM PowerVC.
If you start with LPM, the source and targets servers (and their VIOSs) must be running and
active. Although the VM that you want to move can be offline, the server it is hosted on must
be online. If this criteria is met, then using LPM on an active VM does not result in any
downtime. If that VM crashes, LPM does not automatically restart the VM or migrate it to
another server because manual intervention is required. Therefore, for applications and VMs
where a basic form of planned HA is required, using just LPM is sufficient. However, it is not
sufficient when automatic mobility is required or recovery from an offline source server.
As with LPM, SRR requires the destination server to see the same shared storage as the
source server to rebuild the VM. The key difference between SRR and LPM is that the source
server must be offline so that the rebuild is successful.
1.3 VM Recovery Manager HA

HA management is a critical feature of business continuity plans. Any downtime to the
software stack can result in loss of revenues and disruption of services. IBM VM Recovery
Manager HA for Power Systems is a HA solution that is easy to deploy and provides an
automated solution to recover the VMs (LPARs).
The VM Recovery Manager HA solution implements recovery of the VMs based on the VM
restart technology. The VM restart technology relies on an out-of-band monitoring and
management component that restarts the VMs on another server when the host infrastructure
fails. The VM restart technology is different from the conventional cluster-based technology
that deploys redundant hardware and software components for a near-real-time failover
operation when a component fails. The cluster-based HA solutions are commonly deployed to
protect critical workloads.
The VM Recovery Manager HA solution is ideal to ensure HA for many VMs. Additionally, the
VM Recovery Manager HA solution is easier to manage because it does not have clustering
complexities.
The VM Recovery Manager HA solution provides the capabilities that are described in the
following sections.

Host health monitoring
The VM Recovery Manager HA solution monitors hosts for any failures. If a host fails, the VMs
in the failed host are automatically restarted on other hosts. The VM Recovery Manager HA
solution uses the host monitor (HM) module of the VIOS partition in a host to monitor the
health of hosts.
VM and application health monitoring

The VM Recovery Manager HA solution monitors the VMs, its registered applications, and its
hosts, for any failures. If a VM or a critical application fails, the corresponding VMs are started
automatically on other hosts. The VM Recovery Manager HA solution uses the VM monitor
(VMM) agent, which must be installed in each VM to monitor the health of VMs and registered
applications.
Unplanned HA management
During an unplanned outage, when the VM Recovery Manager HA solution detects a failure
in the environment, the VMs are restarted automatically on other hosts. You can also change
the auto-restart policy to advisory mode. In advisory mode, failed VMs are not relocated
automatically, but instead email or text messages are sent to the administrator. An
administrator can use the interfaces to manually restart the VMs.
Planned HA management
During a planned outage, when you plan to update FW for a host, you can use the LPM
operation of the VM Recovery Manager HA solution to vacate a host by moving all the VMs
on the host to the remaining hosts in the group. After the upgrade operation is complete, you
can use the VM Recovery Manager HA solution to restore the VM to its original host in a
single operation.
Advanced HA policies
The VM Recovery Manager HA solution provides advanced policies to define relationships
between VMs, such as collocation and anticollocation of VMs, the priority in which the VMs
are restarted, and the capacity of VMs during failover operations.
GUI- and command-line interface-based management

You can use a GUI or CLI to manage the resources in the VM Recovery Manager HA solution.
For a GUI, you can install the UI server and then use the web browser to manage the
resources. Alternatively, the ksysmgr command and the ksysvmmgr command on KSYS LPAR
provide end-to-end HA management for all resources.
1.4 IBM VM Recovery Manager DR

DR of applications and services is a key component to provide continuous business services.
The IBM VM Recovery Manager DR for Power Systems solution is a DR solution that is easy
to deploy and provides automated operations to recover the production site. The VM
Recovery Manager DR solution is based on the IBM Geographically Dispersed Parallel
Sysplex® (IBM GDPS®) offering concept that optimizes the usage of resources. This solution
does not require you to deploy the backup VMs for DR. Thus, the VM Recovery Manager DR
solution reduces the software license and administrative costs.
Note: VM Recovery Manager DR now supports the Single-Root I/O Virtualization

(SR-IOV) Virtual Network Interface Controller (vNIC).

The following HA and DR models are commonly used by customers:
򐂰 Cluster-based technology
򐂰 VM restart-based technology
Clustered HA and DR solutions typically deploy redundant hardware and software

components to provide near-real-time failover when one or more components fail. The VM
restart-based HA and DR solution relies on an out-of-band monitoring and management
component that restarts the VMs on other hardware when the host infrastructure fails. The
VM Recovery Manager DR solution is based on the VM restart technology.
1.4.1 Implementing IP change scripts on AIX

When you move VMs by using VM Recovery Manager DR between sites, the default behavior
as it relates to networking is to duplicate the host name, IP address, netmask, and so on, at
the remote site so it looks just like the primary site VM. However, the customer network
environment can dictate that the IP address, host name, netmask, and so on, be different at
the DR site than the primary site. In this situation, we describe a methodology to change IP
addresses between sites by using sample scripts.
Introduction
Implementation of scripts for switching IP addresses when moving an IBM AIX VM between
sites can pose a challenge because AIX configures the IP address early in the startup
process. You can update the IP address after the AIX VM is running and operational at the
DR site, but that means the primary site IP address is active, even for a short period, at the
DR site. This situation can be problematic for network switches, custom applications that
depend on the network environment, and the application configuration.
One method to meet this challenge is by using the AIX ghostdev system flag. This flag is
primarily used for duplicating AIX images to multiple environments. With this flag, you can
make an AIX mksysb image of a VM and use that image to build more VMs on other systems.
When the AIX operation system comes upon this new VM, AIX detects that it is on another
system and automatically removes the host name and IP address configuration as it comes
up on the new host. In addition, the flag also removes the volume group (VG) information and
changes devices, such as host bus adapters (HBAs), to their default values. Many customers
customize device values for performance reasons and do not want the default values. We
address how to return these values back to their wanted settings.
Even though you do not use the ghostdev flag to make an AIX mksysb image, it is useful
because of its automatic system detection and its ability to prevent using the primary site IP
address at the DR site.
Here is the strategy that you use:

1. Gather the host name and IP address information for each VM at each host.
2. Install our sample scripts, which are described later.
3. Configure the AIX operating system by setting the ghostdev parameter to 1.
4. Customize the sample script with the appropriate host name, IP addresses, and so on, for
each particular host.
5. Run the data collection script, which collects data on the device configuration.
6. Move the VM from the primary site to the DR site.
7. Verify that the ghostdev parameter is operational.
8. Verify the DR scripts function by running the DR setup scripts.

9. Update /etc/inittab to automatically run the DR setup scripts.
10.Update cron to automatically run the data collection scripts.
Note: The sample scripts in the VM Recovery Manager DR product and in this publication
are provided AS-IS and without any IBM Support Services.
Example environment
In our example, we use AIX 7.2.2 in our AIX VMs, but any supported version of AIX works.
The environment consists of two sites, “sitert11” and “sitert12”. The AIX VMs are configured
such that when the VMs are in sitert11, they have a different IP address and subnet than on
site sitert12. We demonstrate by using one AIX VM. The VM has a host name of rt11006
when the VM is on host rt11, and a host name of rt11006b when the VM is on host rt12.
Table 1-3 shows the site, VM, and IP address information.
Table 1-3 VM Recovery Manager DR sample environment configuration

Component Site: sitert11 Site: sitert12
Host rt11 rt12
Host model 8286-42A 8286-42A
Host serial number 020607585 022100E5W
Host name rt11006 rt11006b
IP address 10.40.32.110 10.40.0.191
Netmask 255.255.254.0 255.255.254.0
Default gateway 10.40.32.1 10.40.0.1
DNS server 9.3.1.200 9.3.1.200
The goal here is to make sure that the scripts configure the VM such that when it is on system
rt11, it has the appropriate IP and network configuration for that system. Likewise, when the
VM is moved to system rt12, it must have the appropriate IP and network configuration for
that system.

Figure 1-10 and in Figure 1-11 show more details about this process.
Figure 1-10 AIX VM before the VM Recovery Manager DR move to the DR site
Figure 1-11 AIX VM after the VM Recovery Manager DR move to the DR site

The AIX ghostdev parameter
You use the AIX ghostdev parameter mainly for taking an image of an AIX operating system
by using something like the AIX mksysb facility, and then you use that mksysb image to build
more AIX VMs. AIX uses the ghostdev parameter to:
򐂰 Determine whether it is operating on the same hardware as previously.
򐂰 If it is a new AIX image, it clears the host name and IP address configuration.
򐂰 It also clears the VG information and disk drive information, and sets devices to their
default settings.
Without the ghostdev parameter, a copied AIX VM comes up with the previous host name, IP
address, and device configuration. Because those devices no longer exist on the copied VM,
they are in Defined state. We call these devices ghost devices, hence the name of the
ghostdev parameter.
You use the ghostdev setting because you want to prevent the primary site IP addresses from
becoming active, even for a short period, at the DR site. However, the clearing of the VG
information and setting of devices to their default values means that you must use scripts to
return those values to their customize settings.
Gathering environment configuration data

The first step in this process is to gather data that is pertinent for the configuration of the VM
at each site. For each site, you need:
򐂰 The machine type and the serial number of the system (host) the VM moves from and to.
򐂰 The host name of the VM when it is at each host.
򐂰 The related network configuration of the VM when it is at each host:
– IP address
– Netmask
– Default gateway
– DNS server
The machine type and serial number of the host is obtained by running the AIX prtconf
command. The network parameters (for example, IP address, and so on) can probably be
acquired from your network administrator.
Example 1-1 shows an example of using prtconf from the AIX VM to gather system model
and serial number information.
Example 1-1 System model and serial number information

root@rt11006: /
# prtconf | grep -iE "system model|serial number"
System Model: IBM,8286-42A
Machine Serial Number: 2100E5W

Installing AIX sample scripts
Installing the VM Recovery Manager DR IP address change scripts involves the following
steps:
1. Downloading the scripts from the IBM Redbooks website.
The VM Recovery Manager DR scripts can be downloaded from the IBM Redbooks
website (see Appendix A, “Additional material” on page 371).
Use your browser to download the compressed file to your PC or workstation.
2. Copying the VM Recovery Manager DR sample scripts to the managed VMs.
From the location where you downloaded the VM Recovery Manager DR scripts, use your
favorite method (scp, sftp, and so on) to copy the scripts to each managed VM. These
scripts can be staged in any directory. It is a best practice to not put installation files in AIX
system directories, and your favorite non- AIX system directory can be used.
For our purposes, we assume that we downloaded the VM Recovery Manager DR script
to a directory named /software_install.
3. Installing the VM Recovery Manager DR scripts to the appropriate location.
After staging the VM Recovery Manager DR script to an installation directory (in our
example, /software_install), you install the scripts by running the AIX tar command. As
the root user, change to the / directory and run the following commands:
cd /
tar -xvf /software_install/VMR_DR_scripts.tar
The scripts must be unpacked and installed in the /usr/local/DR directory. Several
subdirectories and scripts are created under that /usr/local/DR directory.
Verify that the following files are created under the /usr/local/DR directory:
data_collection.ksh
setup_dr.cfg
setup_dr.ksh
setup_dr_HBAs.ksh
setup_dr_etc_exports.ksh
setup_dr_ethernet.ksh
setup_dr_hadiskhbs.ksh
setup_dr_hostname_ip.ksh
setup_dr_hosts.cfg
setup_dr_reboot.ksh
setup_dr_vgs.ksh
setup_mkitab.ksh
Verify that the following directories are also created:
/usr/local/DR/data
/usr/local/DR/data_default

Description of the VM Recovery Manager DR scripts
Table 1-4 describes the DR scripts and their functions.
Table 1-4 DR scripts and their functions

Script name Script function
data_collection.ksh Collects system device data at the primary site.

This data can be used by other scripts to
customize the system devices when the VM
moves between sites.
setup_dr.cfg Parameter configuration file. Do not customize

this file.
setup_dr.ksh The main script that runs at the DR site to

configure the DR IP addresses. This script calls
the other scripts as necessary.
setup_dr_HBAs.ksh Uses the data that collected to customize the

HBAs at the DR site.
setup_dr_etc_exports.ksh Uses the data that is collected to customize the

NFS exports at the DR site.
setup_dr_ethernet.ksh Uses the data that is collected to customize more

IP addresses at the DR site. It is non-functioning
and is provided as a shell.
setup_dr_hadiskhbs.ksh Uses data that is collected to customize PowerHA

cluster storage devices. This script is
non-functional and is included only as a shell.
setup_dr_hostname_ip.ksh Uses the configuration file information to set up

the DR IP address and network information.
setup_dr_hosts.cfg This file is customized by the user. It is the main

configuration file that contains the host model,
serial number information, and the IP address
and network configuration for each host.
setup_dr_reboot.ksh Restarts the VM as necessary and defined by the

other DR scripts. The VM must be restarted after
it first comes up at the DR site.
setup_dr_vgs.ksh Uses the data that is collected to import the

non-rootvg VGs as necessary.
Updating the AIX ghostdev parameter

On the AIX VM that will be moved from the primary site to the DR site, you want to customize
the ghostdev parameter. By default, the AIX ghostdev parameter is not active. It can be
displayed by running the following command:
root@rt11006: /
# # lsattr -El sys0 | grep ghostdev
ghostdev 0 Recreate ODM devices on system change / modify PVID True
To change the ghostdev parameter and make it active on the system, change the value from 0
to 1 by running the following command:
chdev -l sys0 -a ghostdev=1

Review that the ghostdev parameter has changed by running the following command:
root@rt11006: /
# lsattr -El sys0 | grep ghostdev
ghostdev 1 Recreate ODM devices on system change / modify PVID True
Customizing the AIX sample scripts

The only script file that must be customized is setup_dr_hosts.cfg. The file contains a pair of
stanzas that reflect the parameters for the primary site and the DR site. The file can contain
as many stanzas as necessary if they follow the same format. For our example, we customize
the IP address entries as shown:
[HOST_INFO]
IBM_MODEL="8286-42A"
IBM_SERIAL="0607585"
HOSTNAME=rt11006
IP_ADDRESS=10.40.32.110
NETMASK=255.255.254.0
HOSTNAME_ADAPTER=$HOSTNAME_ADAPTER
NAMESERVER=9.3.1.200
DNSDOMAIN=aus.stglabs.ibm.com
DEFAULT_ROUTE=10.40.32.1
ETC_EXPORTS=${DR_DATA_DIR}/${HOSTNAME}.exports.txt
export IBM_MODEL
export IBM_SERIAL
export HOSTNAME
export IP_ADDRESS
export NETMASK
export HOSTNAME_ADAPTER
export NAMESERVER
export DNSDOMAIN
export DEFAULT_ROUTE
export ETC_EXPORTS
[/HOST_INFO]
and
[HOST_INFO]
IBM_MODEL="8286-42A"
IBM_SERIAL="2100E5W"
HOSTNAME=rt11006a
IP_ADDRESS=10.40.0.191
NETMASK=255.255.254.0
HOSTNAME_ADAPTER=$HOSTNAME_ADAPTER
NAMESERVER=9.3.1.200
DNSDOMAIN=aus.stglabs.ibm.com
DEFAULT_ROUTE=10.40.0.1
ETC_EXPORTS=${DR_DATA_DIR}/${HOSTNAME}.exports.txt
export HOSTNAME
export IP_ADDRESS
export NETMASK
export HOSTNAME_ADAPTER
export NAMESERVER
export DNSDOMAIN
export DEFAULT_ROUTE
export ETC_EXPORTS
[/HOST_INFO]

Running the AIX sample scripts
After configuring the AIX IP configurations, you run the scripts and verify the functions:
1. Run the data_collection.ksh script.
The initial process involves collecting the system data while the VM is at the primary site.
The data collection data files can later be used to customize the VM when it is operational
at the DR site. The data that is collected is mainly system device and storage
configuration.
a. To collect the system configuration, run the following script:
/usr/local/DR/data_collection.ksh
b. The script produces no output but does create text files that contain the device and
storage information. Theses files are in the /usr/local/DR/data and
/usr/local/DR/data_default directories. To verify that the script ran successfully, run
the following command:
/usr/local/DR/data_collection.ksh;echo $?
A successful execution of this script has a result of 0.
2. Initial AIX configuration verification
Using the configuration that we described, log in to the AIX VM rt11006 and review the
current configuration by running prtconf:
a. Run prtconf to verify the mode and serial number of the current system:
root:rt11006: / >
# prtconf | grep -iE "System Model|Machine Serial"
Machine Serial Number: 0607585
b. Run hostname to verify the VM host name:
root:rt11006: / >
# hostname
rt11006
c. Run netstat to verify VM network routing table:
root:rt11006: / >
# netstat -rn
Routing tables
Destination Gateway Flags Refs Use If Exp Groups
Route Tree for Protocol Family 2 (Internet):

default 10.40.32.1 UG 1 15667 en0 - -
10.40.32.0 10.40.32.110 UHSb 0 0 en0 - -
=>
10.40.32/23 10.40.32.110 U 1 1 en0 - -
10.40.32.110 127.0.0.1 UGHS 0 1 lo0 - -
10.40.33.255 10.40.32.110 UHSb 0 1 en0 - -
127/8 127.0.0.1 U 2 117 lo0 - -
Route Tree for Protocol Family 24 (Internet v6):

::1%1 ::1%1 UH 0 9181 lo0 - -
d. Run ifconfig to verify VM IP address configuration:
root:rt11006: / >
# ifconfig -a

en0:
flags=1e084863,814c0<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUP
RT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),LARGESEND,CHAIN>
inet 10.40.32.110 netmask 0xfffffe00 broadcast 10.40.33.255
tcp_sendspace 262144 tcp_recvspace 262144 rfc1323 1
lo0:
flags=e08084b,c0<UP,BROADCAST,LOOPBACK,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64B
IT,LARGESEND,CHAIN>
inet 127.0.0.1 netmask 0xff000000 broadcast 127.255.255.255
inet6 ::1%1/64
3. Verify that the AIX LPAR is ready to be moved.
a. For this example, only a single AIX VM is needed. The VM that is moved in this case is
rt11006. For this reason, unmanage all VMs except the one to be moved by running
the following commands:
ksysmgr unmanage vm rt11007
...
b. To verify that the environment is ready, run a ksysmgr discovery and verification
process:
root:ksys7003rb.aus.stglabs.ibm.com: / >
# ksysmgr discover site sitert11 verify=yes
Host_group Default_HG was modified
Running discovery on entire site, this may take few minutes...
Storage state synchronization has started for Host_group Default_HG
Storage state synchronization has completed for Host_group
Default_HG
Discovery has started for VM rt11006
Configuration information retrieval started for VM rt11006
Configuration information retrieval completed for VM rt11006
Discovery has finished for sitert11
1 out of 1 managed VMs have been successfully discovered
Site verification started for sitert11

Default_HG verification has started
rt12-8286-42A-2100E5W verification has started
rt11006 verification has started
rt11006 verification has completed
Default_HG verification has completed
Verification has finished for sitert11
1 out of 1 VMs have been successfully verified
Lastly, the VM is checked and shown to be READY_TO_MOVE:
# ksysmgr q vm rt11006
VM:
Name: rt11006
UUID: 793E27BD-0113-4A26-93FE-F8DDD5D9B3F1
Host: rt11-8286-42A-0607585
State: READY_TO_MOVE
Priority: Medium
skip_resource_check: No

skip_power_on: No
memory_capacity: none
cpu_capacity: none
Dr Test State: INIT
4. Move the VM to the DR site.
After you verify that the VM is ready to move, run ksysmgr to move the host group to the
DR site:
# ksysmgr move host_group Default_HG to=sitert12
You are initiating a failover across sites
Do you wish to proceed? [y|n]
y
Host_group move started for Default_HG to sitert12
Shutdown on Default_HG Host_group has started for VM rt11006
Shutdown on Default_HG Host_group has completed for VM rt11006
Restart on sitert12 Host_group has started for VM rt11006
Restart on sitert12 Host_group has completed for VM rt11006
Move has completed for VM rt11006
Configuration cleanup started on Default_HG Host_group for VM rt11006
Rediscovering configuration for VM rt11006 on sitert12
Done rediscovering configuration for VM rt11006 on sitert12
Host_group move completed for Host_group Default_HG to sitert12
1 out of 1 VMs have been successfully moved for Default_HG to sitert12
5. Log in to AIX VM and verify the IP address configuration.
Verify the VM IP address configuration. A successful move shows that the IP address and
host name changed to the wanted combination for the DR site.
Note: If the DR scripts are successfully configured, the AIX VM restarts after the initial
configuration is performed by the AIX DR scripts.
a. Run prtconf to verify the mode and serial number of the current system:
root:rt11006a: / >
# prtconf | grep -iE "System Model|Machine Serial"
Machine Serial Number: 2100E5W
b. Run hostname to verify the VM host name:
root:rt11006a: / >
# hostname
rt11006a
c. Run netstat to verify VM network routing table:
root:rt11006a: / >
# netstat -rn
Routing tables
Destination Gateway Flags Refs Use If Exp Groups
Route Tree for Protocol Family 2 (Internet):

default 10.40.0.1 UG 1 37 en0 - -
10.40.0.0 10.40.0.191 UHSb 0 0 en0 - -
=>
10.40/23 10.40.0.191 U 1 0 en0 - -

10.40.0.191 127.0.0.1 UGHS 0 2 lo0 - -
10.40.1.255 10.40.0.191 UHSb 0 0 en0 - -
127/8 127.0.0.1 U 2 72 lo0 - -
Route Tree for Protocol Family 24 (Internet v6):

::1%1 ::1%1 UH 0 0 lo0 - -
d. Run ifconfig to verify the VM IP address configuration:
root:rt11006a: / >
# ifconfig -a
en0:
flags=1e084863,814c0<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUP
RT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),LARGESEND,CHAIN>
lo0:
flags=e08084b,c0<UP,BROADCAST,LOOPBACK,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64B
IT,LARGESEND,CHAIN>
inet 127.0.0.1 netmask 0xff000000 broadcast 127.255.255.255
inet6 ::1%1/64
1.4.2 Implementing IP change scripts on Linux

As shown in 1.4.1, “Implementing IP change scripts on AIX” on page 19, in the situation
where the network configuration is different between the customer’s primary and DR sites,
and where the same network IP addresses cannot be used across sites, we include a set of
sample scripts to address this requirement for the Linux VMs.
Introduction
The implementation of scripts for switching IP addresses when moving a VM between sites is
specific to Linux implementations because Linux typically uses the systemd processes for
starting up resources. Using systemd for starting Linux system processes has many
advantages due to its ability to start processes in parallel. When trying to implement a
process that depends on other processes, you must understand how and where to insert your
process.
We do not describe in detail what systemd does or how it is configured in this book, but we
provide enough information to our scripts that change the IP addresses depending on which
site the VM is.
Systemd consists of multiple components, of which a unit is one. Systemd manages units and
can enable and disable them. Units are described by unit files, which are contained in the
/etc/systemd/system directory. There are many types of units that systemd manages, and
here you focus on the .service unit. The .service unit describes how to manage an
application on the system. The .service unit describes how to run an application or script on
the system. This unit updates the IP address of the VM.

You create a service by creating systemd unit files. This service is run before the network
configuration is performed on the VM. Use the following procedure:
1. Create a systemd service that does the following tasks:
– Starts before the network configuration service.
– Runs a script that copies the correct information to the network configuration files as
part of its processing.
2. Create the script that is called by the service, which does the following tasks:
– Uses a system identifier (SID), which typically is a system serial number to determine
the appropriate IP address for the VM.
– Copies the correct information to the network configuration files.
Note: For purposes of brevity, error checking is not included in the scripts in this chapter,
but it is a best practice to include error checking in your scripts to determine whether the
commands ran successfully.
In addition, the sample scripts in the IBM VM Recovery Manager DR product and in this
publication are provided as is and without any IBM Support services.
Environment configuration data

The first step in this process is to gather data that is pertinent for the configuration of the VM
at each site. For each site, you need the following information:
򐂰 The serial number of the system (host) that the VM moves from and to.
򐂰 The host name of the VM when it is at each host.
򐂰 The related network configuration of the VM when it is at each host:
– IP address and associate
– Netmask
– Default gateway
– DNS server
The serial number of the host is obtained from the file /proc/device-tree/system-id and can
be displayed on a Linux CLI by running cat /proc/device-tree/system-id. The network
parameters (for example, IP address, and so on) can be acquired from your network
administrator.
Example environment
In our example hardware and software setup, we use Red Hat Enterprise Linux (RHEL)
server release 7.6 (Maipo). The environment consists of two sites, sitert11 and sitert12.
The Linux VMs are configured such that when the VMs are in sitert11, they have a different IP
address and subnet than on site sitert12. We demonstrate that setup by using one Linux VM.
The VM has a host name of rt11009 when the VM is on host rt11, and it has a host name of
rt11009b when the VM is on host rt12.

For information about the site, VM, and IP address information, see Table 1-5.
Table 1-5 IBM VM Recovery Manager DR sample environment configuration

Component Site sitert11 Site sitert12
Host rt11 rt12
Host serial number 020607585 022100E5W

IP address 10.40.32.113 10.40.0.194
Netmask 255.255.254.0 255.255.254.0
Default gateway 10.40.32.1 10.40.0.1
DNS server 9.3.1.200 9.3.1.200
The goal here is to make sure that the scripts configure the VM such that when it is on system
rt11that it has the appropriate IP and network configuration for that system. Likewise, when
the VM is moved to system rt12, it must have the appropriate IP and network configuration for
that system. For more information, see Figure 1-12 and Figure 1-13 on page 32.
Figure 1-12 Linux VM before the VM Recovery Manager DR move to the DR site

Figure 1-13 Linux VM before the VM Recovery DR move to the DR site
Creating copies of the system files

In our example, we first install the Linux VM on site sitert11 and host rt11 and configured the
host name as rt11009 with IP address 10.40.32.113. These settings are the appropriate IP
address and host name for this site and host.
We then create copies of system files by using the serial number as part of the name to
identify which network file applies to which host. In our case, we use serial number identifiers
“020607585” for host rt11 and “022100E5W” for host rt12.
You must use the appropriate identifiers for your hosts:

򐂰 For the system file, use /etc/sysconfig/network-scripts/ifcfg-net0 and complete the
following steps:
a. Create a copy by using the following commands:
cd /etc/sysconfig/network-scripts
cp -p ifcfg-net0 020607585.ifcfg-net0
This file, 202607585.ifcfg-net0, corresponds to the network information that is needed
for host rt11.
b. Create another copy by using the following commands:
cd /etc/sysconfig/network-scripts
cp -p ifcfg-net0 022100E5W.ifcfg-net0
In our example, we update 022100E5W.ifcfg-net0 by using data that corresponds to host
rt12.

򐂰 For the system file, use /etc/hostname and complete the following steps:
a. Create a copy by using the following command:
cp -p /etc/hostname /etc/020607585.hostname
This file corresponds to the host name information that is needed for host rt11
b. Create a copy of /etc/hostname by using the following command:
cp -p /etc/hostname /etc/022100E5W.hostname
In our example, we updated /etc/022100E5W.hostname with the data that corresponds to
host rt12.
In total, we have all of the files that are shown in Table 1-6 and Table 1-7.
Table 1-6 Standard and customized ifcfg-net0 files

File type File name
Standard network file /etc/sysconfig/network-scripts/ifcfg-net0
Customized for host rt11 /etc/sysconfig/network-scripts/020607585.i

fcfg-net0
Customized for host rt12 /etc/sysconfig/network-scripts/022100E5W.i

fcfg-net0
Table 1-7 Standard and customized host name files

File type File name
Standard host name file /etc/hostname
Customized for host rt11 /etc/020607585.hostname
Customized for host rt12 /etc/022100E5W.hostname
Contents of the customized system files

In “Creating copies of the system files” on page 32, we created copies of Linux system files.
We customize those files with parameters that are appropriate to the host where the VM will
move to.

For the contents on these customized files, see Table 1-8 and Table 1-9.
Table 1-8 Contents of customized Linux host name files

File type Host name file for rt11 Host name file for rt12
Host name file /etc/020607585.hostname /etc/022100E5W.hostname
Contents rt11009.ausprv.stglabs.ibm.com rt11009b.ausprv.stglabs.ibm.com
Table 1-9 Contents of customized Linux network configuration files

File type Host name file for rt11 Host name file for rt12
Contents NAME="net0" NAME="net0"

HWADDR="52:bd:17:03:aa:0b" HWADDR="52:bd:17:03:aa:0b"
DEVICE="net0" DEVICE="net0"
ONBOOT=yes ONBOOT=yes
NETBOOT=yes NETBOOT=yes
UUID="0c7aade5-93e1-4e01-a6f7-ec5 UUID="0c7aade5-93e1-4e01-a6f7-ec5
454eba555" 454eba555"
IPV6INIT=yes IPV6INIT=yes
BOOTPROTO=none BOOTPROTO=none
IPADDR="10.40.32.113" IPADDR="10.40.0.194"
NETMASK="255.255.254.0" NETMASK="255.255.254.0"
GATEWAY="10.40.32.1" GATEWAY="10.40.0.1"
TYPE=Ethernet TYPE=Ethernet
DNS1="9.3.1.200" DNS1="9.3.1.200"
Creating the script to copy appropriate customized file to the system file
Create the script that copies the appropriate customized files to the system files. This script is
named /usr/sbin/vmr_dr_ip_conf.sh. This file uses the Bash shell, and its full content is as
follows:
#!/bin/bash
#
SERIAL=$(/usr/bin/cat /proc/device-tree/system-id | /usr/bin/awk -F, '{print $2}')
cp -p /etc/sysconfig/network-scripts/$SERIAL.ifcfg-net0
/etc/sysconfig/network-scripts/ifcfg-net0
cp -p /etc/$SERIAL.hostname /etc/hostname
This file copies each of the customized system files to the appropriate standard system file
depending on which host (by serial number) the VM is operational.
Creating the systemd unit file

For our sample environment, we create and configure the systemd unit file that runs the
vmr_dr_ip_conf.sh as /etc/systemd/system/vmr_dr_ip_conf.service. We create this file
and use the vi editor to add the following content:
[Unit]
Description=Set IP conf based on system serial
Before=network-pre.target
Wants=network-pre.target
DefaultDependencies=no
Requires=local-fs.target
After=local-fs.target

[Service]
Type=oneshot
ExecStart=/usr/sbin/vmr_dr_ip_conf.sh
RemainAfterExit=yes
[Install]
WantedBy=network.target
Note following items about the vmr_dr_ip_conf.service file contents:

򐂰 The Before, Wants, and WantedBy stanzas tell systemd to run this service before the
network services start.
򐂰 The ExecStart stanza tells systemd to run the /usr/sbin/vmr_dr_ip_conf.sh script.
By using this file, the vmr_dr_ip_conf service runs before the network services start, and it
customizes the system files with the appropriate parameters for the host on which it is
running.
Enabling the vmr_dr_ip_conf service

To enable the vmr_dr_ip_conf service, run the following command:
systemctl enable vmr_dr_ip_conf
Here is the command output:

systemctl enable vmr_dr_ip_conf
Created symlink from
/etc/systemd/system/network.target.wants/vmr_dr_ip_conf.service to
/etc/systemd/system/vmr_dr_ip_conf.service.
Verifying that the vmr_dr_ip_conf service is operational

To verify that the vmr_dr_ip_conf service is operational, run the systemctl command. Verify
that the vmr_dr_ip_conf service is “loaded” and “active”.
(0) root @ rt11009: /root
# systemctl | grep vmr
vmr_dr_ip_conf.service loaded active exited
Set IP conf based on system serial
Verifying that the IP address scripts change the IP addresses during the
DR event
Now that the DR scripts are configured and the vmr_dr_ip_conf systemd service is set up and
operational, the environment is ready to be tested by moving the Linux VM to the DR site.
Test the environment by performing the following steps:

1. Do the initial Linux configuration verification.
Using the configuration as noted in the previous sections, log in to a Linux VM (rt11009)
and review the current configuration:
a. Verify that the vmr_dr_ip_conf service is operational:
# systemctl | grep vmr

vmr_dr_ip_conf.service loaded active
exited Set IP conf based on system serial
b. Verify the system serial number the VM is on by looking at the system-id file:
# cat /proc/device-tree/system-id
IBM,020607585
c. Verify the host name of the VM by running the hostname command:
# hostname
rt11009.ausprv.stglabs.ibm.com
d. Verify the IP address configuration by running the ifconfig command:
# ifconfig -a
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 128 bytes 11584 (11.3 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 128 bytes 11584 (11.3 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
net0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500

inet 10.40.32.113 netmask 255.255.254.0 broadcast 10.40.33.255
inet6 2002:903:15f:180:50bd:17ff:fe03:aa0b prefixlen 64 scopeid
0x0<global>
inet6 fe80::50bd:17ff:fe03:aa0b prefixlen 64 scopeid 0x20<link>
ether 52:bd:17:03:aa:0b txqueuelen 1000 (Ethernet)
RX packets 566757 bytes 37564748 (35.8 MiB)
device interrupt 19
e. Verify the routing configuration by using the netstat command:
# netstat -rn
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt
Iface
0.0.0.0 10.40.32.1 0.0.0.0 UG 0 0 0
net0
10.40.32.0 0.0.0.0 255.255.254.0 U 0 0 0
net0
2. Verify that the Linux LPAR is ready to be moved.
For this example, only a single Linux VM is needed. The VM that is moved in this case is
rt11009. For this reason, we unmanage all VMs except the one to be moved:
...

a. To verify that the environment is ready, perform a discovery and verification process:
# ksysmgr discover site sitert11 verify=yes
Host_group Default_HG was modified
Running discovery on entire site, this may take few minutes...
Storage state synchronization has started for Host_group Default_HG
Storage state synchronization has completed for Host_group
Default_HG
Discovery has finished for sitert11
Site verification started for sitert11

Default_HG verification has started
rt12-8286-42A-2100E5W verification has started
Default_HG verification has completed
Verification has finished for sitert11
b. Lastly, the VM is checked and shown to be READY_TO_MOVE:
# ksysmgr q vm rt11009
VM:
Name: rt11009
UUID: 1CE6E32D-9E9E-4A59-9EFE-7275CA457DA2
Host: rt11-8286-42A-0607585
Priority: Medium
skip_resource_check: No
skip_power_on: No
memory_capacity: none
cpu_capacity: none
Dr Test State: INIT
3. Move the VM to the DR site.
After the VM is verified to be ready to move, run ksysmgr to move the host group to the DR
site:
# ksysmgr move host_group Default_HG to=sitert12
You are initiating a failover across sites
y
Host_group move started for Default_HG to sitert12
Shutdown on Default_HG Host_group has started for VM rt11009
Shutdown on Default_HG Host_group has completed for VM rt11009
Restart on sitert12 Host_group has started for VM rt11009
Restart on sitert12 Host_group has completed for VM rt11009
Move has completed for VM rt11009
Configuration cleanup started on Default_HG Host_group for VM rt11009
Rediscovering configuration for VM rt11009 on sitert12
Done rediscovering configuration for VM rt11009 on sitert12

Host_group move completed for Host_group Default_HG to sitert12
1 out of 1 VMs have been successfully moved for Default_HG to sitert12
4. At the DR site, log in to the Linux VM and verify the IP address configuration.
After the VM is moved to the DR site, the VM IP address configuration is verified. The IP
address and host name change to the wanted combination.
a. Verify the system serial number that the VM is on by looking at the system-id file:
(0) root @ rt11009b: /root
# cat /proc/device-tree/system-id
IBM,022100E5W
b. Verify the host name of the VM by running the hostname command:
# hostname
rt11009b.ausprv.stglabs.ibm.com
c. Verify the IP address configuration by running the ifconfig command:
# ifconfig -a
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 229 bytes 23056 (22.5 KiB)
net0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500

inet 10.40.0.194 netmask 255.255.254.0 broadcast 10.40.1.255
inet6 2002:903:15f:180:50bd:17ff:fe03:aa0b prefixlen 64 scopeid
0x0<global>
inet6 fe80::50bd:17ff:fe03:aa0b prefixlen 64 scopeid 0x20<link>
ether 52:bd:17:03:aa:0b txqueuelen 1000 (Ethernet)
RX packets 44760 bytes 2993269 (2.8 MiB)
device interrupt 19
d. Verify the routing configuration by running the netstat command:
# netstat -rn
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt
Iface
0.0.0.0 10.40.0.1 0.0.0.0 UG 0 0 0
net0
10.40.0.0 0.0.0.0 255.255.254.0 U 0 0 0
net0
The DR scripts are verified and the VM can be moved to its home site.

2
Chapter 2. IBM VM Recovery Manager High

Availability components
This chapter introduces the IBM VM Recovery Manager High Availability (HA) product from
an architectural perspective. The chapter groups its various topics into the following sections:
򐂰 Overview and capabilities
򐂰 Solution components and detailed architecture
򐂰 Communication flows
Note: When a specific concept, term, or abbreviation is used for the first time, it is
emphasized in italics and the surrounding statement can be considered as its definition.

2.1 Overview and capabilities
This section presents an architecture overview and the key capabilities of the VM Recovery
Manager HA product. Section 2.1.1, “Overview” on page 40 introduces the main idea of the
solution, and 2.1.2, “Capabilities” on page 41 lists the relevant use cases that are covered by
the tool.
2.1.1 Overview
VM Recovery Manager HA can be deployed across an existing PowerVM virtualized
infrastructure that consists of Hardware Management Consoles (HMCs), managed systems,
Virtual I/O Servers (VIOSs), and virtual machines (VMs). Figure 2-1 presents a generic
architecture overview diagram for the VM Recovery Manager HA solution. The term host is
used from now on for the physical POWER managed systems. More POWER hosts that are
in the same data center and sharing common network connectivity and storage access
configuration can be placed together in a host group so that if one host has an issue, the
affected VMs are relocated on neighboring healthy hosts in the group.
Figure 2-1 VM Recovery Manager HA architecture overview
The core component of the solution is the KSYS controller, which is a piece of software
running on a dedicated AIX logical partition (LPAR). The KSYS controller acts as an
orchestrator by monitoring continuously the managed infrastructure. When a failure is
detected, it makes VM relocation decisions and acts by restarting VMs on different hosts.
Configurable user policies are accounted for during relocation. The limit of 24 VIOSs results
in up to 12 managed systems per host group for typical dual-VIOS configurations. Up to four
host groups are supported by a KSYS controller in large environments. If needed, you can
also deploy multiple KSYS controller independent instances, each responsible for its separate
group of managed systems.

2.1.2 Capabilities
With VM Recovery Manager HA, you have easier HA management of virtualized PowerVM
environments. We restate here a bit more of its core capabilities that were introduced in 1.3,
“VM Recovery Manager HA” on page 17.
Host monitoring
Managed hosts are monitored continuously for failures. When the KSYS controller concludes
that a host failure happened, all the VMs previously running on the failed host are
automatically restarted on neighboring healthy hosts. The decision that a host failure
happened is made by assessing the health state of the VIOSs running on the host. Both
VIOSs must be declared unhealthy based on specific logic to conclude that a host failure
occurred.
For more information about the specific logic, see “Host relocation decision” on page 95.
Virtual machine monitoring

The KSYS framework also monitors for possible failures at the VM level. If the failure
detection component of the framework concludes that only a VM failed but the host is okay,
then only the failed VM is restarted on a different host. A piece of software that is called the
VM Agent must be installed in the VM so that a heartbeat message with the VM state is sent
periodically to the KSYS controller. The lack of the heartbeat messages is the primary factor
in deciding the VM relocation, but other factors are also considered when deciding.
For more information about the decision process, see “VM relocation decision” on page 96.
Application monitoring
An application management framework, component of the installed VM Agent monitors the
application by using a user-provided monitoring script. This local framework can restart the
application by using user-provided stop and start scripts over a user-defined number of times.
But, if this repeated local application restart does not solve the problem, then it is considered
a permanent application failure and the KSYS controller can take over.
The KSYS controller monitors all applications that are configured as critical by the user. When
such a critical application faces a permanent application failure, the KSYS controller restarts
the hosting VM. KSYS first restarts the VM on the same host so that the VM Agent can regain
control for a new sequence of local application restart cycles. If the permanent application
failure state is reached again, the local VM restarts and the inner application restart cycle is
repeated for a limited pre-configured number of times. When the pre-configured restart limit
count is reached, then the VM is restarted on a different host.
For more information about the application management framework and critical application
management topics, see 3.7.2, “Application Management Engine” on page 312.
Unplanned high availability management

During an unplanned outage, if the VM Recovery Manager HA framework detects a failure
and the restart policy is set to auto mode, then the affected VMs are restarted automatically
on the other hosts of the host group. If the restart policy is set to advisory mode, then the
failed VMs are not relocated automatically but registered users are notified. The user can
then check and decide whether to do a manual restart of the affected VMs by using the KSYS
interface.
Chapter 2. IBM VM Recovery Manager High Availability components 41

Planned HA management
When planned maintenance downtime is required (power circuits intervention, firmware (FW)
update, and so on), you can use the Live Partition Mobility (LPM) option of the VM Recovery
Manager HA to evacuate the host by migrating all its VMs to neighboring hosts in the group.
When maintenance activities are complete, you can use the same VM Recovery Manager HA
LPM option to bring the VMs back to their original host in a single operation.
Advanced high availability policies

Collocation and anticollocation relationships can be configured for managed VMs. Various
policies are available, such as priority-based relocation, priority for the order in which VMs are
restarted, host exclusion (blacklist) at VM relocation, and flexible capacity adjustment plus
best fit relocation of VMs.
For more information about the HA policies, see 3.6, “Setting up HA policies” on page 297.
GUI- and command-line interface-based management

You can use CLI tools on the KSYS controller and at the VM level. The KSYS controller CLI
(ksysmgr) and the VM CLI (ksysvmmgr) installed together with the VM Agent on each managed
VM provides full configuration and status management capabilities that cover every aspect of
the solution. A GUI that is called the GUI server is also available as a separate installable
piece of software for easier management through a web browser.
For more information, see Chapter 4, “IBM VM Recovery Manager High Availability GUI
deployment” on page 323.
2.2 Solution components and detailed architecture

The VM Recovery Manager HA solution consists of several key components that
communicate and work together to provide a HA environment for systems across the same
data center.
2.2.1 Simplified component model

We start with the simplified model in Figure 2-2 on page 43, where we place the components
that are deployed during the initial setup into the existing PowerVM infrastructure system
context, which consists of classical HMCs, hosts, VIOSs, and VMs.

Figure 2-2 Simplified component model
Figure 2-2 shows that from a high-level installation and deployment perspective, the product
consists of three primary components:
򐂰 KSYS controller
򐂰 VM Agent
򐂰 GUI server
KSYS controller
The KSYS controller acts as an overall orchestrator for the environment. You can refer to the
KSYS controller as the KSYS subsystem or KSYS. The registered infrastructure is monitored,
and if a failure is detected, the KSYS analyzes the situation, notifies the administrator, and
can automatically relocate the affected VMs to other healthy hosts. The KSYS interacts with
the registered HMCs, hosts, and VIOSs to collect configuration and status information about
their managed resources.
The KSYS controller runs on an AIX LPAR and can be separated from the managed
infrastructure to avoid simultaneous KSYS and managed infrastructure outage. The KSYS
remains operational even when parts of the managed infrastructure fail. Ensure that the AIX
instance running the KSYS subsystem is monitored by local monitoring and incident
management solutions. You can periodically check the KSYS subsystem health either by
using the CLI or the VM Recovery Manager HA GUI dashboard.

Virtual machine agent
When VM and application monitoring is needed, you must install the VM Agent filesets. VM
Agent filesets come bundled with the KSYS controller filesets. The VM Agent reacts to the
following possible VM and application-level issues:
VM failures If the operating system of a VM is not working correctly or if the VM
stopped working because of an error, the VM is restarted on another
host within the host group. In normal operation, the KSYS controller
receives the heartbeat from the VM Agent and nothing happens.
When the heartbeat from the VM does not arrive, the KSYS controller
reacts, enters a decision-making process, and restarts the VM.
Application failures Optionally, you can register the applications in the VM Agent to enable
application monitoring. The VM Agent checks the health of the
registered applications by running periodically an application-specific
user-provided monitoring script. It restarts the application locally if the
script returns an error. For critical applications, if the local restart does
not succeed, the KSYS is notified and can do a local VM restart in the
same host and even VM relocation on a different host.
GUI server
The administrator can operate the tool either directly on the KSYS controller by using CLI
commands in terminal sessions or the web browser through the GUI server. GUI-based
access offers a user-friendly management alternative to the CLI approach, and it can act as a
single point of control for multiple KSYS instances within the environment. The GUI server
topic is covered in Chapter 4, “IBM VM Recovery Manager High Availability GUI deployment”
on page 323.
2.2.2 Refined component model

We now further refine the component model that is described in 2.2.1, “Simplified component
model” on page 42 by providing more insights about the key subsystems that are involved,
the connectivity topology, and some basic communication flows that are used to exchange
data with each other. With this augmented model, which is shown in Figure 2-3, we can cover
redundancy and security aspects, which are mandatory for an enterprise solution.
Figure 2-3 Refined component model

The product is also designed to minimize the setup effort after the software installation
registers the HMCs with the correct credentials to the KSYS. This design enables visibility of
all hosts, VIOSs, and VMs that the HMCs manage. You define the host groups by deciding
which managed systems are a part of each host group. All VIOSs and VMs on a host that are
added in a host group are registered and managed by default. You can manually exclude
some VMs so they remain unmanaged.
After this preliminary host group setup is complete, run a simple host group discovery
command. The host group discovery scans and collects the required input and performs all
the required settings. Allow the discovery to complete and do not manually intervene while it
is running. Access the HMC and VIOS only if errors are detected after the discovery process.
Consider an initial discovery that successfully completed for a host group. New components
show up or are activated, as shown in Figure 2-3 on page 44, in addition to those in 2.2.1,
“Simplified component model” on page 42. We introduce these new components as follows:
򐂰 Host monitor
򐂰 Shared Storage Pool cluster and Health Status Database
򐂰 Virtual switch
򐂰 VM monitor and Application Monitoring Framework
Host monitor
The KSYS accesses the HMCs and activates the host monitor (HM) daemon on each VIOS.
The HM code is included as part of the VIOS V3.1 enhancements, so any VIOS 3.1 or later
installation has it. To see whether any specific update must be applied, see IBM Knowledge
Center. The HM daemon is activated on a VIOS when the user performs the first KSYS
discovery operation for the host group containing the host where the VIOS is. The HM is
ready to communicate with the KSYS controller through the HMC and VIOS.
A key role of the HM is to monitor the heartbeat coming from the VMs running on the host and
to update KSYS subsystem. HM also intermediates communication between the VM Agent
and KSYS.
Shared Storage Pool cluster and Health Status Database

The first KSYS discovery operation that is performed by the user for a defined host group also
deploys a Shared Storage Pool (SSP) cluster across the VIOSs. All these VIOSs must be set
as MANAGED in the KSYS configuration, and they can share two disks of at least 10 GB each.
One disk is used as repository disk for the SSP cluster, and the other one, the HA disk, is
used to track the health state of the whole host group. If the SSP cluster is created by the
KSYS subsystem, no other party may use it. You must not use this SSP cluster for any other
purpose. If a user-defined SSP cluster exists on the involved VIOSs, then the KSYS
subsystem uses the existing SSP cluster.
A shared file system space and database are created on the shared disk for SSP
management. KSYS creates its own schema inside this SSP management database. Health
status information about VMs, HMs, and VIOSs of all hosts on the host group is then saved
and kept inside that schema. This database is called the Health Status Database (HSDB).
When the KSYS controller retrieves a piece of data by querying the HSDB, it uses a single
VIOS in the host group and does not need to interrogate each VIOS separately. KSYS polls
the HSDB every 20 seconds for health status updates. The retrieved updates are empty
under normal operating conditions. Only when something is not healthy is information
returned to KSYS.

The SSP cluster is also used for a second purpose. The underlying Cluster Aware AIX (CAA)
cluster performs its CAA heartbeats over the network and, when the network goes down, over
the repository disk. CAA Autonomic Health Advisor File System (AHAFS) event monitoring is
integrated with the SSP monitoring, and all this event monitoring framework is extended to
update the health database with the received information from the CAA. CAA heartbeat and
health monitoring aids KSYS to monitor for host failures.
This is the bare minimum HA management that you obtain if you initialize only KSYS,
configure the HMCs, and create the host group. KSYS initiates health management by using
the deployed SSP cluster and HSDB, and it is ready to determine any subsequent host
failure. This host monitoring is independent of any VM running on the host itself. A second
thing that you can do is to install the VM Agent. It is optional, but if you install and configure
this VM Agent on your VMs, you get some extra benefits, as explained “Virtual switch” and
“VM monitor and Application Monitoring Framework” on page 47.
Virtual switch
During a host group discovery, if at least one VM on a member host is enabled for monitoring,
then KSYS creates a virtual switch in each host hypervisor. One trunk Virtual Ethernet
adapter is created on each VIOS and connected to the new virtual switch. The trunk adapters
have different virtual LAN (VLAN) IDs (101 on one VIOS and 102 on the other VIOS). During
discovery, two standard Virtual Ethernet adapters are also created on each VM that is
enabled for monitoring. One Virtual Ethernet adapter has VLAN ID 101 and the other Virtual
Ethernet adapter has VLAN ID 102. The virtual switch is configured in Virtual Ethernet Port
Aggregator (VEPA) mode to isolate the VMs from each other. Each VM can communicate
only with the trunk ports on the VIOS side and cannot reach the other VMs through this VEPA
mode switch.
Figure 2-4 shows a virtual switch and related Virtual Ethernet adapters on a host with three
VMs, all enabled for monitoring at the KSYS level.
Figure 2-4 Hypervisor vSwitch

A private data link protocol is used for communication between HM and VM monitor (VMM)
through the virtual switch. There is no TCP/IP stack that is configured on these private
interfaces. They are used exclusively for two purposes:
򐂰 To transmit heartbeat messages from the VM Agent to the HMs and from there to the
KSYS subsystem.
򐂰 To provide two-way connectivity between the KSYS controller and the VM Agent for the
exchange of configuration and status updates about the monitored applications.
By using this protocol, you get a complete communication path between the KSYS controller
on one side and each VM Agent with its monitored applications on the other side.
Redundancy is provided by an intrinsic design that is based on double components at the
HMC and VIOS levels. Security is ensured by the virtual switch VEPA mode, which isolates
the VM private interfaces.
VM monitor and Application Monitoring Framework

The VM Agent has two subsystems, each providing a specific function:
VM monitor This subsystem sends periodically a heartbeat
message to the HMs in the underlying VIOSs.
Those heartbeats help you detect OS hangs or
fails. If the OS kernel crashes, for example, then
HMs no longer receive the heartbeat from VMM
and after some time update the HSDB. KSYS is
polling HSDB periodically and is eventually
informed about the VM missed heartbeat. This
event can result in a VM relocation decision or at
least a user notification about the issue, depending
on the way this behavior is configured by the
administrator.
Application Monitoring Framework This subsystem registers, starts, stops, and
monitors the applications. For the monitoring
framework, an application is defined by three
user-provided scripts: a monitoring script, a start
script, and a stop script. When you register an
application, you specify these three scripts in the
framework. The framework then checks the health
of the registered applications by running
periodically the provided monitoring script. It
restarts the application locally in case the script
returns with an error. For critical applications, if
that repeated local restart does not succeed, the
KSYS is notified and can decide on a local VM
restart in the same host or VM relocation on a
different host.
For more information about the VM Agent structure and functions, see 2.2.8, “VM Agent
architecture” on page 164.

2.2.3 KSYS test cluster example
In our KSYS test cluster, we deployed a simplistic two-HMC and two-host configuration. The
primary focus here is to illustrate the functions that are described in 2.2.2, “Refined
component model” on page 44. The detailed configuration is described in Chapter 3,
“Planning and deploying IBM VM Recovery Manager High Availability for IBM Power
Systems” on page 265.
Assuming that the KSYS cluster is successfully installed and initialized, then its state can be
Online, as shown in Example 2-1.
Example 2-1 KSYS cluster state immediately after initial deployment

# ksysmgr query ksyscluster
Name: rbRMHA
State: Online
Type: HA
# lsrpdomain
Name OpState RSCTActiveVersion MixedVersions TSPort GSPort
rbRMHA Online 3.2.4.1 No 12347 12348
# lssrc -s IBM.VMR
Subsystem Group PID Status
IBM.VMR rsct_rm 8454466 active
#
Immediately after registering an HMC, you can see the universally unique identifier (UUID)
for each of its managed systems, as shown in Example 2-2. For dual-HMC configurations,
managed systems appear once under each managing HMC.
Example 2-2 Host UUIDs

# ksysmgr query hmc
Name: rthmc6
Ip: 9.3.207.78
Login: hscroot
Managed Host List:
Host Name Uuid

========= ====
rt12-8286-42A-2100E5W 4715697c-a486-3cbb-8b5a-2637fa4db054
rt14-8286-42A-SN2100DEW 6e2f37f5-1824-347a-99fd-488f818086dd
============================================================================
Name: rthmc3
Ip: 9.3.18.159
Login: hscroot
Managed Host List:
Host Name Uuid

========= ====
rt13-8286-42A-21E0B2V 4462c37c-65c6-3614-b02a-aa09d752c2ee
rt11-8286-42A-0607585 d04d8a4a-99fa-3e4b-80bc-c9d9716bd8f8
============================================================================
#

The host must be explicitly added to be managed by the KSYS cluster. You can select the
hosts among the ones that are available at the registered HMCs level. When a host is added,
all its VIOS LPARs are registered by default and set to the MANAGED state.
Example 2-3 shows the VIOSs running on the hosts that we added in our test KSYS cluster.
Example 2-3 VIOS LPARs on registered hosts

# ksysmgr query host
Name: rt14-8286-42A-SN2100DEW
UUID: 6e2f37f5-1824-347a-99fd-488f818086dd
FspIp: Must run discovery first to populate
Host_group: No host_group defined
VIOS: rt14v1
rt14v2
HMCs: rthmc6
HA_monitor: enable
VM_failure_detection_speed: normal
Name: rt13-8286-42A-21E0B2V
UUID: 4462c37c-65c6-3614-b02a-aa09d752c2ee
VIOS: rt13v2
rt13v1
HMCs: rthmc3
HA_monitor: enable
When a host has more than two VIOSs, you must keep in the KSYS configuration only two of
the VIOSs to provide access to the shared storage resources. The other VIOSs must be
explicitly unregistered from the KSYS cluster instance, as suggested in Example 2-4.
Example 2-4 How to unmanage a VIOS LPAR

# ksysmgr unmanage vios -h
ksysmgr unmanage vios <viosname[,...]> | <viosuuid[,...]
unmanage => unman*, umg
vios => vios*
#
The host group of our two hosts is created, as shown in Example 2-5.
Example 2-5 rbHG host group immediately after creation

# ksysmgr query host_group
Name: rbHG
Hosts: rt13-8286-42A-21E0B2V
rt14-8286-42A-SN2100DEW
Memory_capacity: Priority Based Settings
high:100
medium:100
low:100
CPU_capacity: Priority Based Settings
high:100

medium:100
low:100
Skip_power_on: No
HA_monitor: enable
Restart_policy: auto
Host_failure_detection_time: 90
No SSP
#
When we installed and started the VM Agent on the VMs, we decided to keep them as
managed in our configuration. Then, we performed the initial host group discovery and
verification operations. Example 2-6 shows the output of the initial discovery. The SSP was
created, but there is nothing about the virtual switches and trunk adapters.
Example 2-6 SSP creation at initial host group discovery

# ksysmgr discover host_group rbHG
Running discovery on Host_group rbHG, this may take few minutes...
SSP creation started for Host_group rbHG
SSP creation completed for Host_group rbHG
...
Discovery for VM rt13001 is complete
...
Discovery has finished for rbHG
Successfully created dir:/var/ksys/snapshots
# ksysmgr verify host_group rbHG
Host_group verification started for rbHG
...
...
Verification has finished for rbHG
8 out of 8VMs have been successfully verified
#
The SSP cluster status on a VIOS node is available in Example 2-7.
Example 2-7 SSP cluster status

$ cluster -status
Cluster Name State
KSYS_rbRMHA_1 OK
Node Name MTM Partition Num State Pool State

rt13v2 8286-42A0221E0B2V 2 OK OK
rt13v1 8286-42A0221E0B2V 1 OK OK
rt14v1 8286-42A022100DEW 1 OK OK
rt14v2 8286-42A022100DEW 2 OK OK
$

We repeated the discovery and verify operations, and checked that the system and managed
VMs have the ha_monitor attribute enabled, as shown in Example 2-8. Now, trunk adapters
appear.
Example 2-8 Trunk adapter creation at host group discovery after enabling monitoring
# ksysmgr modify system ha_monitor=enable
KSYS ha_monitor has been updated
# ksysmgr modify vm
rt13001,rt13002,rt13005,rt13007,rt14001,rt14002,rt14005,rt14007 ha_monitor=enable
For VM rt13001 attribute(s) 'ha_monitor' was successfully modified.
...
# ksysmgr discover host_group rbHG verify=yes
Creating HA trunk adapter for VIOS rt14v2
Finished creating HA trunk adapter for VIOS rt14v2
Preparing VIOS in rt13-8286-42A-21E0B2V for HA management
VIOS in rt13-8286-42A-21E0B2V prepared for HA management
Preparing VIOS in rt14-8286-42A-SN2100DEW for HA management
VIOS in rt14-8286-42A-SN2100DEW prepared for HA management
...
Host_group verification started for rbHG

...
Verification has finished for rbHG
#
Example 2-9 shows the vSwitch and Virtual Ethernet adapter connectivity details of an AIX
VM and its underlying VIOSs after the new discovery operation.
Example 2-9 vSwitch Virtual Ethernet Adapter details: VM and VIOSs

# uname -uL;oslevel -s
IBM,0221E0B2V 3 rt13001
7200-03-02-1845
# lsdev -l ent*|while read ent bla; do echo; echo $ent; entstat -d $ent|grep -E
"^Port\ VLAN|Switch"; done
ent0
Port VLAN ID: 1

Switch ID: ETHERNET0
ent1
Port VLAN ID: 101
Switch ID: rbRMHA_VSWITCH
ent2
Port VLAN ID: 102
#
# uname -uL; /usr/ios/cli/ioscli ioslevel; oslevel -s

IBM,0221E0B2V 1 rt13v1
3.1.0.00
7200-03-02-1845
# entstat -d ent7|grep -E "^Port\ VLAN|Switch"
Port VLAN ID: 102
Switch Mode: VEPA
#
# uname -uL; /usr/ios/cli/ioscli ioslevel; oslevel -s

IBM,0221E0B2V 2 rt13v2
3.1.0.00
7200-03-02-1845
# entstat -d ent5|grep -E "^Port\ VLAN|Switch"
Port VLAN ID: 101
Switch Mode: VEPA
#
The VEPA mode of the switch ensures complete isolation between connected VM Virtual
Ethernet adapters. They have connectivity only with the VIOS trunk adapter.
2.2.4 KSYS as a Resource Monitoring and Control subsystem

The KSYS daemon is implemented as a Resource Monitoring and Control (RMC) resource
manager, and the ksysmgr CLI as an RMC client application. RMC is a subset function of
Reliable Scalable Cluster Technology (RSCT). Before going into detail about the KSYS
subsystem, we introduce some basic RMC concepts and terminology.
Resource Monitoring and Control concepts

RSCT is a set of software components that provide generic clustering services for AIX and
Linux platforms. It consists of various core subsystems, such as Group Services, Topology
Services, and System Registry.
RMC is a subsystem of RSCT that is a generalized framework for managing, monitoring, and
manipulating system resources. By system resources, we mean various physical or logical
entities that are represented in an abstracted way in the RMC framework. Any resource
abstraction, simply called resource, has attributes that describe configuration and state
characteristics of the modeled entity and actions to model behavior and changes, like in usual
object-oriented approaches. Resources with similar characteristics are instances of
predefined resource classes.

An example of a resource is IBM.MCP (Example 2-10), which keeps details about the HMC
managing the LPAR.
Example 2-10 IBM.MCP resource

# lsrsrc IBM.MCP
Resource Persistent Attributes for IBM.MCP
resource 1:
MNName = "9.3.18.181"
NodeID = 6958593213159554706
KeyToken = "vmhmc5"
IPAddresses = {"9.3.207.188"}
ConnectivityNames = {"9.3.18.181"}
HMCName = "7042CR9*213BDAD"
HMCIPAddr = "9.3.207.188"
HMCAddIPs = ""
HMCAddIPv6s = ""
ActivePeerDomain = "rbRMHA"
NodeNameList = {"ksys7004customer.aus.stglabs.ibm.com"}
#
On a typical AIX instance, the RMC subsystem is activated in a default stand-alone

configuration immediately after the operating system installation. It is started by the SRC
subsystem as the ctrmc daemon and can be used this way to manage and monitor the
resources of a single machine. Associated local daemons, called resource managers, act as a
software layer between resources and the RMC daemon. Resource managers map the
programmatic abstracted resources and classes that are defined and maintained within the
RMC framework to actual calls and commands handling their counterpart physical or logical
entity resources. Resource managers also maintain definitions and persistent attributes of
those abstract resource instances and classes they are responsible for in a collection of local
files, referred to as the local registry. A resource manager daemon can be started by the SRC
subsystem, by the RMC daemon on demand, or autostarted when RMC is brought online.
You can configure ctrmc daemons on multiple systems to communicate with each other in a
cluster configuration. With such a setup, you can manage and monitor resources of all
systems (nodes) in the cluster. There are two cluster setups that are available: peer domain
clusters and management domain clusters.
A peer domain is a set of nodes that have a consistent knowledge of the existence of each
other and of the resources that are shared among them. In a peer domain setup, RMC
activates and uses core RSCT subsystems such as Topology Services, Group Services, and
cluster security services subsystems.
A management domain is defined as a set of nodes whose resources can be managed and
monitored from one of the nodes, which is designated as the management control point
(MCP). All other nodes are considered to be managed nodes. Topology Services and Group
Services are not used in a management domain.
The registry component provides data persistency and consistency services. Persistent data
is stored in local registry files. In cluster configurations, data consistency is ensured by
maintaining the local registry replicas in sync on all cluster nodes.

The RMC framework provides APIs and also command utilities (mkrsrc, lsrsrc, chrsrc,
rmsrc, lsactdef, runact, and so on) to support client applications, resource managers, and
users to create and manipulate their resources and classes. Libraries and APIs are available
to support all these functions, as shown in Figure 2-5.
Figure 2-5 RMC framework
An RMC client application can call API subroutines directly or can use command utilities,
which are RMC clients themselves. A resource manager implements its own functions so that
the RMC daemon can pass requests from client applications.
A resource manager daemon can create and initialize its own specific in-memory constructs,
mainly Resource Control Points (RCPs) and Resource Class Control Points (RCCPs) to
handle its managed resources and resource classes in the registry. RCPs and RCCPs are
instantiated objects of a class type that are initialized by a constructor and expose their
functions by implemented methods. A resource manager itself might have a Resource
Manager Control Point (RMCP) object.
Resource managers use RSCT multi-page tracing capability for logging purposes. Code
inside resource managers can send trace records (relevant to an internal functional context or
decided by some criteria to be grouped) into a common logical trace file that is referred by a
base name. Multiple distinct base names can be used. The tracing facility writes these
records into a set of physical trace files that have the same base name and suffixes in the
form of .number.sp. These physical files are referred to as trace page files and are written in a
typical rotating manner.
When the last page file that is associated with a logical trace file is full, the trace records are
written to the first page file. The fixed number of these trace files limits the history information
that is available for debugging purposes. If needed, you can modify the number and size of
the page files. Additionally, you can activate the trace spooling feature to copy the completed
files to another directory on the system where you have more space that is available. Trace
spooling can be turned on or off. Manual or automated (crontab based or similar) cleanup
maintenance is needed to avoid file system full situations for the spooling repository.

Example 2-11 presents the ctrmc RMC daemon and the resource managers running on our
AIX instance before the KSYS cluster creation. The AIX instance also shows up as part of a
management domain. It is the management domain that is established between the
managing HMC and its managed LPARs with the HMC in MCP role.
Example 2-11 Default RMC subsystem

# lssrc -g rsct
ctrmc rsct 12517638 active
ctcas rsct inoperative
# lssrc -g rsct_rm
IBM.HostRM rsct_rm 15532476 active
IBM.ConfigRM rsct_rm 11796916 active
IBM.DRM rsct_rm 6881586 active
IBM.MgmtDomainRM rsct_rm 15466872 active
IBM.ServiceRM rsct_rm 10289656 active
IBM.ERRM rsct_rm inoperative
IBM.AuditRM rsct_rm inoperative
# rmcdomainstatus -s ctrmc
Management Domain Status: Management Control Points
I A 0x6091e3a8bed1be92 0001 9.3.207.188

#
We show the local AIX resource in Example 2-10 on page 53 that represents the HMC MCP.
This management domain setup supports HMC and LPAR operations, like graceful shutdown;
DLPAR; LPM; and others. In a dual-HMC configuration, there are two management domains.
For more information about the RMC framework, see the IBM Knowledge Center and A
Practical Guide for Resource Monitoring and Control (RMC), SG24-6615. These resources
cover the default AIX resource managers and associated resources and classes that you
have seen so far in the example outputs of this section.
Let us now examine how the RMC framework is used by VM Recovery Manager HA to
implement its KSYS controller function.
KSYS components in the RMC framework

Example 2-1 on page 48 showed that the KSYS cluster creation step configures the RMC
subsystem as part of an RSCT peer domain. Example 2-12 shows the following extra details:
򐂰 The peer domain has a single node.
򐂰 Group Services and Topology Services subsystems are activated.
򐂰 New resource managers are configured, including the IBM.VMR resource manager in which
we are interested.
Example 2-12 IBM.VMR RMC Resource Manager

# lssrc -g rsct
ctrmc rsct 12517638 active
ctcas rsct inoperative
# lsrpdomain
rbRMHA Online 3.2.4.1 No 12347 12348

# lsrpnode -i
Name OpState RSCTVersion NodeNum NodeID
ksys7004customer.aus.stglabs.ibm.com Online 3.2.4.1 1 cbd72d17633f31e3
# rmcdomainstatus -s ctrmc
Peer Domain Status
S S 0xcbd72d17633f31e3 0001 ksys7004customer.aus.stglabs.ibm.com
Management Domain Status: Management Control Points
I A 0x6091e3a8bed1be92 0001 9.3.207.188
# lssrc -a|grep ctha
cthats cthats 7012846 active
cthags cthags 9437456 active
cthagsglsm cthags inoperative
# lssrc -g rsct_rm
IBM.HostRM rsct_rm 15532476 active
IBM.ConfigRM rsct_rm 11796916 active
IBM.DRM rsct_rm 6881586 active
IBM.MgmtDomainRM rsct_rm 15466872 active
IBM.ServiceRM rsct_rm 10289656 active
IBM.StorageRM rsct_rm 12648774 active
IBM.ERRM rsct_rm inoperative
IBM.AuditRM rsct_rm inoperative
IBM.LPRM rsct_rm inoperative
#
KSYS daemon is implemented in the form of the IBM.VMR resource manager. Figure 2-6 on
page 57 shows the KSYS controller components that are interconnected within the RMC
framework:
򐂰 The ksysmgr utility
򐂰 IBM.VMR resource manager
򐂰 Resource and Class Registry

Figure 2-6 KSYS in an RMC framework
The ksysmgr utility

As introduced in 3.5, “Configuring VM Recovery Manager HA” on page 281, ksysmgr
represents internally the various entities it manages as objects of a predefined type CLASS,
on which an ACTION can be performed. These objects are, in fact, just RMC resources of a
specific RMC resource class, with predefined attributes and actions. Most of the ksysmgr
commands start RMC-provided utilities that are revealed in the ksysmgr log
/var/ksys/log/ksysmgr.log. The excerpt in Example 2-13 shows a sequence of actions that
were logged during our KSYS cluster creation by the ksysmgr add ksyscluster command.
Steps in the sequence start RMC utilities such as preprpnode, mkrpdomain, and lsrsrc.
Example 2-13 KSYS cluster creation excerpt from /var/ksys/log/ksysmgr.log

[15335792] !COMMAND!:"ksysmgr add ksyscluster rbRMHA ksysnodes=ksys7004customer
type=HA"
[15335792] add_ksyscluster()[52]: ENTERING
[15335792] Getting hostname for ksys7004customer
[15335792] known host ksys7004customer.aus.stglabs.ibm.com
...
[15335792] Adding node to current cluster configuration
[15335792] time:10/27/2018 13:26:16 -------------
[15335792] runSystemCommand()[794] : RUNNING
COMMAND:/usr/sbin/rsct/bin/preprpnode ksys7004customer > /dev/null 2>&1
[15335792] runSystemCommand()[800]: Unaltered return status: 0
[15335792] runSystemCommand()[827]: RC: 0
[15335792] time:10/27/2018 13:26:16 -------------

[15335792] runSystemCommand()[794] : RUNNING
COMMAND:/usr/sbin/rsct/bin/mkrpdomain rbRMHA $(/usr/bin/hostname) > /dev/null
2>&1;
[15335792] Ksyscluster has been created, please run: "ksysmgr verify ksyscluster
<ksysclustername>"
[15335792] time:10/27/2018 13:26:20 -------------
[15335792] runSystemCommand()[794] : RUNNING COMMAND:lssrc -s IBM.VMR | grep
IBM.VMR | awk '{print $3}' >/tmp/ksysmgr.tmp.YQW7ea; [ -s /tmp/ksysmgr.tmp.YQW7ea
]
add_ksyscluster()[478]: EXITING SUCCESSFULLY
IBM.VMR resource manager

The IBM.VMR resource manager is responsible for its own set of resources and classes, and
has specific attributes and actions. Example 2-14 starts by listing all resource classes that are
defined at the RMC level after the KSYS cluster creation. The second column lists the
associated resource manager of each class in the first column. Filtering by the IBM.VMR string
in the second column, we get 11 classes as the final output. They are the RMC classes that
are managed by the IBM.VMR resource manager.
Example 2-14 Resource classes of the IBM.VMR resource manager

# echo "class_name:mgr_name"; lsrsrc -c|grep -v class_name|while read cl; do echo
"$cl:\c"; lsrsrcdef -c $cl|grep mgr_name|awk '{ print $3}';done|grep IBM.VMR
class_name:mgr_name
"IBM.VMR_HMC":"IBM.VMR"
"IBM.VMR_CEC":"IBM.VMR"
"IBM.VMR_LPAR":"IBM.VMR"
"IBM.VMR_VIOS":"IBM.VMR"
"IBM.VMR_SSP":"IBM.VMR"
"IBM.VMR_SITE":"IBM.VMR"
"IBM.VMR_SA":"IBM.VMR"
"IBM.VMR_DP":"IBM.VMR"
"IBM.VMR_DG":"IBM.VMR"
"IBM.VMR_HG":"IBM.VMR"
"IBM.VMR_APP":"IBM.VMR"
#
The RMC resource manager IBM.VMR implements its log files by using the common RSCT
trace facility. Example 2-15 shows the default trace setup for the IBM.VMR resource manager.
Example 2-15 Default tracing setup for the IBM.VMR resource manager
# lssrc -ls IBM.VMR|grep -p trace
Information about trace levels:
...
_VMR Errors=255 Info=1 LPAR=4 Verify=4 KREST=5 KRI=4
FDE=4 FDELong=4 CONFLICT=4 MSG=4 Debug=0
/var/ct/3EmzyyOpynuvLXYdEkdZSO/log/mc/IBM.VMR/trace.1.sp -> spooling not enabled
/var/ct/rbRMHA/log/mc/IBM.VMR/trace.conflict.1.sp -> spooling not enabled
/var/ct/rbRMHA/log/mc/IBM.VMR/trace.fde.1.sp -> spooling not enabled
/var/ct/rbRMHA/log/mc/IBM.VMR/trace.fdelong.1.sp -> spooling not enabled
/var/ct/rbRMHA/log/mc/IBM.VMR/trace.krest.1.sp -> spooling not enabled

/var/ct/rbRMHA/log/mc/IBM.VMR/trace.krestlong.1.sp -> spooling not enabled
/var/ct/rbRMHA/log/mc/IBM.VMR/trace.ksys.1.sp -> spooling not enabled
/var/ct/rbRMHA/log/mc/IBM.VMR/trace.msg.1.sp -> spooling not enabled
/var/ct/rbRMHA/log/mc/IBM.VMR/trace.user.1.sp -> spooling not enabled
#
There are nine distinct logical trace files with a specific base name, each with its
preconfigured number of physical page trace files. The product documentation mentions five
of them (user, ksys, fde, krest, and krestlong), which are used for problem determination.
The user and ksys files contain both information about the functions that are run for each
operation, but ksys provides more details. When investigating a problem, start by checking
the user log file and, if needed, go to ksys. Similarly, trace.fdelong.*.sp and
trace.fde.*.sp are the detailed and the abridged trace files for the Failure Detection Engine
(FDE), which is a core functional module that has its own dedicated trace files.
The KSYS daemon communicates with HMCs and VIOSs in the infrastructure by using a
wrapper library for the HMC Representational State Transfer (REST) API that is called
libkrest. Inputs, outputs, and errors of libkrest calls are traced by dedicated high-level
(trace.krest.*.sp) and detailed trace (trace.krestlong.*.sp) files.
Typical trace file records have as their third field the pthread ID (pTID) of the thread logging
the record. Example 2-16 shows sample records taken from the trace.user.*.sp files (after
these are formatted by the rpttr -odict /var/ct/rbRMHA/log/mc/IBM.VMR/trace.user*
command).
Example 2-16 User trace file excerpt

...
[00] 10/27/18 T( 1) ____ 13:29:45.936221 ******************* Trace Started - Pid
= 8454466 **********************
[00] 10/27/18 T(304) _VMR 13:29:45.964261 [ INFO,VMRDaemon,] Start LPAR category
logging
...
[00] 10/27/18 T(203) _VMR 14:29:50.997883 [ INFO]: SCHEDULER initiated QUICK
discoverVerify Started
...
[00] 10/27/18 T(809) _VMR 14:47:09.403713 DEBUG VMR_HG.C[13572]: Policymap
VmtoCECMapTable:
...
[00] 10/27/18 T(9293) _VMR 15:58:58.094910 DEBUG VMR_HG.C[7580]: User triggered
operation: migrate
...
[00] 10/27/18 T(396) _VMR 15:59:03.095091 DEBUG VMR_HG.C[13348]:
-------------POLICYMAP EXE START------------------------
...
[01] 11/10/18 T(90a) _VMR 02:05:50.840332 DEBUG VMR_LPAR.C[4220]: do_recover_LPAR
STARTED for VM: rt13001(4E395E99-FFFB-4345-8067-2D5296BA7D91) action_id: validate
...

To get a first contact with the structure of the KSYS daemon, look for these thread IDs in the
daemon thread stack snapshot that was taken, as shown in Example 2-17. The daemon had
142 threads at the moment the procstack command ran. Of them, 128 of them had the
threadHandler() function called followed by TaskQueue::getTaskFromQueue() as distinctive
sequence among the nested calls in each thread stack. Because it is not helpful to list them
all, we selected one (pTID 2314) to represent all 128 of them in the output of the last
command, prockstack 8454466. The output is further shortened for easier reading.
Example 2-17 KSYS daemon thread stack

# procstack 8454466|grep -c tid#
142
# procstack 8454466|grep -c "TaskQueue::getTaskFromQueue"
129
# procstack 8454466|grep -c "eventHandler"
1
# procstack 8454466|grep -c "threadHandler"
128
# procstack 8454466
8454466: /opt/rsct/bin/IBM.VMRd
---------- tid# 15532537 (pthread ID: 1) ----------
...
0xd28f9438 rsct_base::CDaemon::handleSRC(int)(0x3019cab0, 0x0) + 0x278
0xd27a7b8c rsct_rmf3v::RMDaemon::handleSRC(int)(0x3019cab0, 0x0) + 0x10c
0x1003e810 main() + 0x230
0x10000208 __start() + 0x68
---------- tid# 23069115 (pthread ID: 37523) ----------
...
---------- tid# 34996731 (pthread ID: 38296) ----------
...
0x10735d58 FDEthread(void*)(0x3024faa8) + 0x2af8
0xd0566f64 _pthread_body(??) + 0xe4
---------- tid# 29557123 (pthread ID: 2314) ----------
...
0x103f4d88 TaskQueue::getTaskFromQueue(Task**)(0x310711c0, 0x3108dad0) + 0xe8
0x103f4fbc threadHandler(void*)(0x310701a0) + 0xbc
---------- tid# 20840705 (pthread ID: 36495) ----------
...
...<<127x threadHandler/TaskQueue threads as pthread ID 2314 above>>
...
---------- tid# 35389853 (pthread ID: 918) ----------
...
0x103f0a34 TaskQueue::getTaskFromQueue(Task**,int*)(0x321f7760, 0x32219ac0,
0x32219ad0) + 0x134
0x103f1dcc eventHandler(void*)(0x321f7760) + 0x18c
---------- tid# 22937869 (pthread ID: 1800) ----------
...
---------- tid# 15466975 (pthread ID: 515) ----------
...
0xd279a3c8 rsct_rmf3v::RMSchedule::run(void*)(0x301ff3d8, 0x0) + 0x648
0xd28fad78 rsct_base::CRunnable::threadMain()(0x301ff3d8) + 0x318
0xd28fcd94 rsct_base::stubCRunnable(void*)(0x301ff3d8) + 0x14
---------- tid# 10027361 (pthread ID: 258) ----------

...
#
We also analyzed the ksys trace files looking for the most active threads during the daemon
startup. The result of this analysis also influenced the selection of the threads that are kept as
relevant for the procstack output, as shown in Example 2-17 on page 60. The analysis is
detailed in 2.2.5, “KSYS daemon restart exercise” on page 62. The field T(90a) of the last
record in Example 2-16 on page 59 means that the record itself is written by the thread with
pTID 2314, as shown in Example 2-17 on page 60, with the 90a value in parentheses being
the hexadecimal representation of decimal 2314.
Example 2-18 shows the kernel thread ID (TID) and the user space pTID decimal value as
listed by the procstack command for some of the threads that are shown in Example 2-17 on
page 60. In the last column, we added the matching thread pTID hexadecimal value.
Example 2-18 Some thread IDs

---------- tid# 15532537 (pthread ID: 1) ---------- T( 1)
---------- tid# 15466975 (pthread ID: 515) ---------- T(203)
---------- tid# 15270361 (pthread ID: 2057) ---------- T(809)
---------- tid# 23069115 (pthread ID: 37523) ---------- T(9293)
---------- tid# 35389853 (pthread ID: 918) ---------- T(396)
---------- tid# 29557123 (pthread ID: 2314) ---------- T(90a)
---------- tid# 34996731 (pthread ID: 38296) ---------- T(9598)
A thread can log records in any logical trace file. The main thread (pTID 1), for example,
writes at the daemon startup a time stamp message with the daemon process ID (PID) in
each trace file. The FDE thread (having the distinctive FDEthread() function invoked) is
writing with predilection in the trace.fde.*.sp files, as shown in Example 2-19, but it can also
log records in other trace files.
Example 2-19 A fde trace file excerpt

...
[01] 11/13/18 T(9598) _VMR 18:25:50.024794 DEBUG VMR_HG.C[11664]: FDE performing
doQuickQuery to 315CE56B-9BA0-46A1-B4BC-DA6108574E7E
[01] 11/13/18 T(9598) _VMR 18:25:50.024798 DEBUG VMR_retry.C[1151]: Doing
operation with opCode: 33(VMDR_QUICK_QUERY)
[01] 11/13/18 T(9598) _VMR 18:25:50.024820 DEBUG VMR_retry.C[178]: INFO: Trying
with HMC: rthmc3.
[01] 11/13/18 T(9598) _VMR 18:25:50.024828 DEBUG VMR_HMC.C[6905]: getQuickQuery:
Calling kriSubmitQuickQuery!. HMC:9.3.18.159, viosUuid:
315CE56B-9BA0-46A1-B4BC-DA6108574E7E
...
Again, the number in the third field, T(9598), repeated here in each line, is nothing else than
the hexadecimal value for the decimal 38296 pTID, which you saw in as the thread identifier in
the procstack output of Example 2-17 on page 60. The rest of the record is self-explanatory:
date, time, source file plus the line of the code with the instruction throwing the message, and
the descriptive message itself.

Resource and Class Registry
Example 2-20 shows the file system directory location and the file structure of the registry
that is used as local persistent store by the IBM.VMR resource manager for its resources and
classes.
Example 2-20 Persistent registry of IBM.VMR resource manager

# cd /var/ct/rbRMHA/registry/local_tree
# ls *VMR*
IBM,VMR_APP,Class IBM,VMR_HG,Class IBM,VMR_SITE,Class
IBM,VMR_APP,Resources IBM,VMR_HG,Resources IBM,VMR_SITE,Resources
IBM,VMR_CEC,Class IBM,VMR_HMC,Class IBM,VMR_SSP,Class
IBM,VMR_CEC,Resources IBM,VMR_HMC,Resources IBM,VMR_SSP,Resources
IBM,VMR_DG,Class IBM,VMR_LPAR,Class IBM,VMR_VIOS,Class
IBM,VMR_DG,Resources IBM,VMR_LPAR,Resources IBM,VMR_VIOS,Resources
IBM,VMR_DP,Class IBM,VMR_SA,Class
IBM,VMR_DP,Resources IBM,VMR_SA,Resources
#
Note: In a peer domain configuration with multiple nodes, a registry replica is kept
consistently on each node by the RMC framework. The KSYS subsystem supports only
one node peer domain, but the framework is prepared for multiple node extensions.
2.2.5 KSYS daemon restart exercise

We perform a daemon restart exercise to examine the content of the trace files in the
immediate interval after the new daemon is started and to get more insights about how it
operates. The ksysmgr sync ksyscluster command is used for restart, as shown in
Example 2-21.
Example 2-21 Restarting the KSYS cluster

# ksysmgr sync ksyscluster rbRMHA
Ksyscluster seems to already be online and running, restarting KSYS subsystem
Starting KSYS subsystem ...
Waiting for KSYS subsystem to start ...
KSYS subsystem has started, you can begin adding HMCs, Hosts, etc
#
We expect that a restart of the KSYS daemon ends with a process memory context and
thread layout similar to what was there before restart.
It is not that difficult to compare thread stack snapshots that are taken before and after restart
as a first confirmation. Example 2-22 presents the thread stack snapshot that is taken after a
restart.
Example 2-22 KSYS daemon thread stack after restart

# lssrc -a|grep VMR
# procstack 16646518|grep -c tid#
142
# procstack 16646518|grep -c "TaskQueue::getTaskFromQueue"
129
# procstack 16646518|grep -c "eventHandler"

1
# procstack 16646518|grep -c "threadHandler"
128
#
# procstack 16646518
16646518: /opt/rsct/bin/IBM.VMRd
---------- tid# 29557231 (pthread ID: 1) ----------
...
0xd28f9438 rsct_base::CDaemon::handleSRC(int)(0x3019cab0, 0x0) + 0x278
0xd27a7b8c rsct_rmf3v::RMDaemon::handleSRC(int)(0x3019cab0, 0x0) + 0x10c
0x1003e810 main() + 0x230
0x10000208 __start() + 0x68
---------- tid# 32047601 (pthread ID: 37780) ----------
...
---------- tid# 23396853 (pthread ID: 37266) ----------
...
0x10735d58 FDEthread(void*)(0x32777ac8) + 0x2af8
---------- tid# 21365075 (pthread ID: 37009) ----------
...
0x103f0a34 TaskQueue::getTaskFromQueue(Task**,int*)(0x3263eb40, 0x32774ac0,
0x32774ad0) + 0x134
0x103f1dcc eventHandler(void*)(0x3263eb40) + 0x18c
---------- tid# 32440809 (pthread ID: 36752) ----------
...
0x103f4d88 TaskQueue::getTaskFromQueue(Task**)(0x32624c30, 0x32748ad0) + 0xe8
0x103f4fbc threadHandler(void*)(0x3260f690) + 0xbc
---------- tid# 16843039 (pthread ID: 36495) ----------
...
...<<127x threadHandler/TaskQueue threads as pthread ID 36752 above>>
...
---------- tid# 20840721 (pthread ID: 515) ----------
...
0xd279a3c8 rsct_rmf3v::RMSchedule::run(void*)(0x301ff3d8, 0x0) + 0x648
0xd28fad78 rsct_base::CRunnable::threadMain()(0x301ff3d8) + 0x318
0xd28fcd94 rsct_base::stubCRunnable(void*)(0x301ff3d8) + 0x14
---------- tid# 12321057 (pthread ID: 258) ----------
...
#
We see that the same layout that is shown in Example 2-17 on page 60 is created after
restart with the number of threads as 142, among which are one event handler (starting
eventHandler()), one FDE thread (starting FDEthread()), one scheduler thread (starting
rsct_rmf3v::RMSchedule::run()), and the pool of 128 task handlers (starting
threadHandler() and TaskQueue::getTaskFromQueue()), as expected.

Example 2-23 shows the way we extracted the content that is logged to the fde and ksys
trace files into the post-restart_fde.log and post-restart_ksys.log log files after the
daemon restarts.
Example 2-23 Collecting the content of the ksys and fde trace files after the daemon restarts
# lssrc -a|grep VMR
# ksysmgr trace log=fde > fde.log
# cat fde.log|sed '/Trace Started - Pid = 16646518/,$!d' > post-restart_fde.log
# head -1 post-restart_fde.log
= 16646518 **********************
# tail -1 post-restart_fde.log
[06] 11/19/18 T(9192) _VMR 13:53:37.658738 DEBUG FDEthread.C[709]: Sleep for 20
sec. sleepCounter 13
# ksysmgr trace log=ksys > ksys.log
# cat ksys.log|sed '/Trace Started - Pid = 16646518/,$!d' > post-restart_ksys.log
# head -1 post-restart_ksys.log
= 16646518 **********************
# tail -1 post-restart_ksys.log
[15] 11/19/18 T(9192) _VMR 14:01:23.787096 DEBUG VMR_HMC.C[6938]: getQuickQuery
[315CE56B-9BA0-46A1-B4BC-DA6108574E7E] JobStatus: COMPLETED_OK, ReturnCode: 0
#
The ksysmgr trace log=fde and ksysmgr trace log=ksys commands in Example 2-23 are
the KSYS equivalents of the RSCT commands rpttr -odict
/var/ct/rbRMHA/log/mc/IBM.VMR/trace.fde* and rpttr -odict
/var/ct/rbRMHA/log/mc/IBM.VMR/trace.ksys*. Running the sed command deletes the rows
before the distinctive record signaling daemon restart with the new daemon PID (pattern
Trace Started - Pid = 16646518). From the time stamps in the first and last row of the
truncated log files, we see that the traces are from the first minutes after restart (12 minutes
for ksys).
Most active threads during the KSYS daemon start

We now identify the most active threads during the daemon start. We count how many times
each thread writes into the collected ksys trace file during the first minutes after the daemon
restart (Example 2-24).
Example 2-24 Thread writing frequency in the ksys trace file after restart
# cat post-restart_ksys.log|wc -l
1203
# grep "11/19/18 T(" post-restart_ksys.log|grep "T( 1)"|wc -l
5
# for t in `grep "11/19/18 T(" post-restart_ksys.log|grep -v "T( 1)"|while read
f1 f2 f3 others; do echo $f3; done|sort -u`; do echo "$t \c"; grep "11/19/18 T("
post-restart_ksys.log|grep -v "T( 1)"|while read f1 f2 f3 others; do echo $f3;
done|grep -c $t; done|sort -nrk2
T(304) 626
T(9192) 66
T(90a) 66
T(4041) 56
T(8788) 50
T(8687) 50

T(8889) 49
T(8586) 49
T(3d3e) 47
T(203) 30
T(9091) 12
T(405) 10
T(809) 6
T(9394) 4
T(4849) 1
T(3c3d) 1
T(393a) 1
T(2d2e) 1
#
Table 2-1 shows decimal pTID values in the second column. The last column shows the
distinctive call that is started in the thread’s stack.
Table 2-1 Writing frequency of threads showing up in the ksys trace file after daemon restart
Log pTID pTID Trace Distinctive thread stack call taken from procstack
(hex) writes output
T( 1) 1 5 main()
T(304) - 626 - (Terminated already at the moment we ran the

procstack command.)
T(9192) 37266 66 FDEthread()
T(90a) 2314 66 threadHandler()
T(4041) 16449 56 threadHandler()
T(3d3e) 15678 47 threadHandler()
T(203) 515 30 rmf3v::RMSchedule::run()
T(9091) 37009 12 eventHandler()
T(405) 1029 10 rsct_gscl_V1::GSController::dispatch()
T(809) 2057 6 rsct_gscl_V1::GSController::dispatch()
T(9394) 37780 4 rsct_rmf3v::RMRmcp::dispatchRequests
T(3c3d) 15421 1 threadHandler()
T(393a) 14650 1 threadHandler()
T(2d2e) 11566 1 threadHandler()

Here are more comments about the results that are shown in Table 2-1 on page 65:
򐂰 The thread T(304), which is not present in the procstack output, was active during restart.
򐂰 Eleven threads from the pool of 128 task handler threads had more or less activity.
򐂰 An FDE thread is created and writes in the ksys log, in addition to its dedicated fde and
fdelong logs.
򐂰 A scheduler thread and an event thread Handler were created as before restart.
򐂰 Other threads that are related to core RSCT activities write ksys traces, but discreetly
Resource and Resource Class Control Points instantiation

We now look for more structure in the startup sequence and for relevant patterns suggesting
the creation of the in-memory constructs that are associated with our expected resources and
objects: HMCs, hosts, VIOSs, host groups, VMs, and applications. As introduced in
“Resource Monitoring and Control concepts” on page 52, we search for RCP and
RCCP-related traces (Example 2-25).
Example 2-25 RCP and RCCP traces in the ksys trace file after the daemon restart
# grep -i rcp post-restart_ksys.log|wc -l
404
[# grep -i rccp post-restart_ksys.log|wc -l
112
# grep -i rccp post-restart_ksys.log|head -4
[15] 11/19/18 T(304) _VMR 13:48:38.197869 VMR_SITERccp::VMR_SITERccp Entered
[15] 11/19/18 T(304) _VMR 13:48:38.198199 VMR_SITERccp::VMR_SITERccp Leaving
[15] 11/19/18 T(304) _VMR 13:48:38.198395 VMR_HMCRccp::VMR_HMCRccp Entered
[15] 11/19/18 T(304) _VMR 13:48:38.198452 VMR_HMCRccp::createRcp Entered
#
The occurrences of the first matching string, VMR_SITERccp, suggest that a constructor of a
VMR_SITERccp class ran. Searching for other VMR_SITERccp occurrences produces the output
in Example 2-26.
Example 2-26 Searching for SITE RCCP

# sed '/VMR_SITERccp/!d;=' post-restart_ksys.log
13
[15] 11/19/18 T(304) _VMR 13:48:38.197869 VMR_SITERccp::VMR_SITERccp Entered
23
[15] 11/19/18 T(304) _VMR 13:48:38.198199 VMR_SITERccp::VMR_SITERccp Leaving
668
[15] 11/19/18 T(304) _VMR 13:49:26.232232 VMR_SITERccp::scheduleQuickCheck
Entered.
672
[15] 11/19/18 T(304) _VMR 13:49:26.232238 VMR_SITERccp::scheduleQuickCheck
Leaving.
674
[15] 11/19/18 T(304) _VMR 13:49:26.232239 VMR_SITERccp::scheduleDetailCollectOnce
Entered.
679
[15] 11/19/18 T(304) _VMR 13:49:26.232291 VMR_SITERccp::scheduleDetailCollectOnce
Leaving.
699
[15] 11/19/18 T(9394) _VMR 13:49:26.476082 VMR_SITERccp::validateSetClassParms
Entered, number of values=57.

700
[15] 11/19/18 T(9394) _VMR 13:49:26.476093 VMR_SITERccp::validateSetClassParms
Leaving.
1203
#
The VMR_SITERccp::VMR_SITERccp constructor leaves traces at the beginning of the start

sequence, in line range 13 - 23, and then some of the VMR_SITERccp class methods are
started almost a second later in the sequence, starting with line 668, somewhere in the
middle of the whole log file.
In Example 2-27, we then filter all output that is logged by the VMR_HMCRccp::VMR_HMCRccp
constructor, which corresponds to the next RCCP match in Example 2-25 on page 66.
Example 2-27 HMC RCCP constructor log

# sed '/VMR_HMCRccp::VMR_HMCRccp Entered/,/VMR_HMCRccp::VMR_HMCRccp Leaving/!d'
post-restart_ksys.log
[15] 11/19/18 T(304) _VMR 13:48:38.198395 VMR_HMCRccp::VMR_HMCRccp Entered
[15] 11/19/18 T(304) _VMR 13:48:38.198477 DEBUG VMR_HMC.C[407]: Setting site
ID as 0 as HA only cluster type for HMC rthmc3
[15] 11/19/18 T(304) _VMR 13:48:38.198583 VMR_HMCRcp::VMR_HMCRcp Entered
[15] 11/19/18 T(304) _VMR 13:48:38.200763 VMR_HMCRcp::VMR_HMCRcp Leaving
[15] 11/19/18 T(304) _VMR 13:48:38.200799 VMR_HMCRccp::createRcp Leaving,
pRcp=31667bc0
[15] 11/19/18 T(304) _VMR 13:48:38.200851 DEBUG VMR_HMC.C[407]: Setting site
ID as 0 as HA only cluster type for HMC rthmc6
[15] 11/19/18 T(304) _VMR 13:48:38.200907 VMR_HMCRcp::VMR_HMCRcp Entered
[15] 11/19/18 T(304) _VMR 13:48:38.202221 VMR_HMCRcp::VMR_HMCRcp Leaving
[15] 11/19/18 T(304) _VMR 13:48:38.202252 VMR_HMCRccp::createRcp Leaving,
pRcp=3168a150
[15] 11/19/18 T(304) _VMR 13:48:38.202262 VMR_HMCRccp::VMR_HMCRccp Leaving
#
Two HMC RCPs are created inside the HMC RCCP constructor, one for each HMC in the
infrastructure.
The Entered and Leaving strings appear frequently in our previous examples as start and end
markers for class constructors and methods. So, we use this technique to delimit the log
sections of each RCCP constructor. Example 2-28 shows the results of this technique for the
Rccp Entered and Rccp Leaving patterns.
Example 2-28 All RCCPs created at daemon startup

1203
# sed '/Rccp Entered/!d;=' post-restart_ksys.log |sed 'N;s/\n/:/'
13:[15] 11/19/18 T(304) _VMR 13:48:38.197869 VMR_SITERccp::VMR_SITERccp Entered
25:[15] 11/19/18 T(304) _VMR 13:48:38.198395 VMR_HMCRccp::VMR_HMCRccp Entered
38:[15] 11/19/18 T(304) _VMR 13:48:38.202592 VMR_CECRccp::VMR_CECRccp Entered
90:[15] 11/19/18 T(304) _VMR 13:49:10.223135 VMR_LPARRccp::VMR_LPARRccp Entered
551:[15] 11/19/18 T(304) _VMR 13:49:26.229629 VMR_SARccp::VMR_SARccp Entered
555:[15] 11/19/18 T(304) _VMR 13:49:26.230234 VMR_VIOSRccp::VMR_VIOSRccp Entered

588:[15] 11/19/18 T(304) _VMR 13:49:26.230852 VMR_DPRccp::VMR_DPRccp Entered
592:[15] 11/19/18 T(304) _VMR 13:49:26.231005 VMR_DGRccp::VMR_DGRccp Entered
597:[15] 11/19/18 T(304) _VMR 13:49:26.231318 VMR_HGRccp::VMR_HGRccp Entered
639:[15] 11/19/18 T(304) _VMR 13:49:26.231898 VMR_SSPRccp::VMR_SSPRccp Entered
646:[15] 11/19/18 T(304) _VMR 13:49:26.232153 VMR_APPRccp::VMR_APPRccp Entered
# sed '/Rccp Leaving/!d;=' post-restart_ksys.log |sed 'N;s/\n/:/'
23:[15] 11/19/18 T(304) _VMR 13:48:38.198199 VMR_SITERccp::VMR_SITERccp Leaving
36:[15] 11/19/18 T(304) _VMR 13:48:38.202262 VMR_HMCRccp::VMR_HMCRccp Leaving
88:[15] 11/19/18 T(304) _VMR 13:49:10.222904 VMR_CECRccp::VMR_CECRccp Leaving
549:[15] 11/19/18 T(304) _VMR 13:49:26.229397 VMR_LPARRccp::VMR_LPARRccp Leaving
553:[15] 11/19/18 T(304) _VMR 13:49:26.230084 VMR_SARccp::VMR_SARccp Leaving
585:[15] 11/19/18 T(304) _VMR 13:49:26.230695 VMR_VIOSRccp::VMR_VIOSRccp Leaving
590:[15] 11/19/18 T(304) _VMR 13:49:26.230864 VMR_DPRccp::VMR_DPRccp Leaving
595:[15] 11/19/18 T(304) _VMR 13:49:26.231168 VMR_DGRccp::VMR_DGRccp Leaving
637:[15] 11/19/18 T(304) _VMR 13:49:26.231759 VMR_HGRccp::VMR_HGRccp Leaving
644:[15] 11/19/18 T(304) _VMR 13:49:26.232014 VMR_SSPRccp::VMR_SSPRccp Leaving
660:[15] 11/19/18 T(304) _VMR 13:49:26.232224 VMR_APPRccp::VMR_APPRccp Leaving
# sed '13,660!d' post-restart_ksys.log|grep -c "T(304)"
602
# sed '13,660!d' post-restart_ksys.log|grep -v "T(304)"|wc -l
46
# sed '13,660!d' post-restart_ksys.log|grep -v "T(304)"|grep -v "^[[:space:]]*$"
Added VM's List =
# sed '13,660!d' post-restart_ksys.log|grep -v "T(304)"|grep -c "^[[:space:]]*$"
45
#
You can easily associate the RCCPs that are shown in Example 2-28 on page 67 with their
counterpart resource classes, which re identified in Example 2-14 on page 58.
With the last four sed commands, we check that there are no other threads than T(304) logs
in the line range 13 - 660. Thread T(304) logs 602 records as counted by the grep -c
"T(304)" command. Then, the remaining 46 lines prove to be 45 white space lines plus one
line consisting of only the “Added VM's List =” string. So, only T(304) is logging all these
entries in the sequence. By the row number of each start and end marker, we see that RCCP
constructors are run one after the other and log their messages in subsequent compact
sections.
The largest RCCP constructor section is the largest section of the LPAR RCCP,
VMR_LPARRccp::VMR_LPARRccp, from lines 90 - 549 because one LPAR RCP is created for
each of the 16 existing VMs, as shown in Example 2-29.
Example 2-29 Counting the LPAR RCPs

# sed '90,549!d;=' post-restart_ksys.log |sed 'N;s/\n/:/'|grep "createRcp Leaving"
139:[15] 11/19/18 T(304) _VMR 13:49:10.223764 VMR_LPARRccp::createRcp Leaving, pRcp=325e9810
165:[15] 11/19/18 T(304) _VMR 13:49:10.224098 VMR_LPARRccp::createRcp Leaving, pRcp=325e6380
...
517:[15] 11/19/18 T(304) _VMR 13:49:10.228205 VMR_LPARRccp::createRcp Leaving, pRcp=326024b0
543:[15] 11/19/18 T(304) _VMR 13:49:10.228476 VMR_LPARRccp::createRcp Leaving, pRcp=32600990
# sed '90,549!d;=' post-restart_ksys.log |sed 'N;s/\n/:/'|grep -c "createRcp Leaving"
16
#
The CEC RCCP constructor generated records in lines 38 - 88. Example 2-30 on page 69
lists only the first 10 lines in the range.

Example 2-30 CEC RCCP constructor log
# sed '/VMR_CECRccp::VMR_CECRccp Entered/,/VMR_CECRccp::VMR_CECRccp Leaving/!d'
post-restart_ksys.log|head
[15] 11/19/18 T(304) _VMR 13:48:38.202592 VMR_CECRccp::VMR_CECRccp Entered
[15] 11/19/18 T(304) _VMR 13:48:38.202596 DEBUG VMR_CEC.C[413]: creating thread
pool with 128 active threads
[15] 11/19/18 T(304) _VMR 13:48:38.219775 VMR_CECRccp::createRcp Entered

[15] 11/19/18 T(304) _VMR 13:48:38.219842 DEBUG VMR_CEC.C[813]: setting its
failover for CEC policy to best_fit
[15] 11/19/18 T(304) _VMR 13:48:38.219890 VMR_CECRcp::VMR_CECRcp Entered
[15] 11/19/18 T(304) _VMR 13:48:38.219904 DEBUG VMR_CEC.C[4398]:
Initializing the mutex
[15] 11/19/18 T(304) _VMR 13:48:38.219906 VMR_CECRcp::VMR_CECRcp Leaving
[15] 11/19/18 T(304) _VMR 13:48:38.219956 VMR_CECRcp::set_partnerUUIDs Entered
[15] 11/19/18 T(304) _VMR 13:48:38.219957 VMR_CECRcp::set_partnerUUIDs Leaving
# sed '/VMR_CECRccp::VMR_CECRccp Entered/,/VMR_CECRccp::VMR_CECRccp Leaving/!d'
post-restart_ksys.log|grep "VMR_CECRccp::createRcp"
[15] 11/19/18 T(304) _VMR 13:48:54.221478 VMR_CECRccp::createRcp Leaving,
rt13-8286-42A-21E0B2V, pRcp=325c4470
[15] 11/19/18 T(304) _VMR 13:49:10.222859 VMR_CECRccp::createRcp Leaving,
rt14-8286-42A-SN2100DEW, pRcp=325c688
#
Notice the message about the creation of the pool with 128 threads. An RCP is created for
each host that is registered as managed by KSYS.
The next log section in the sequence is about the host group RCCP constructor.
Example 2-31 shows the complete details of this section.
Example 2-31 Host group RCCP creation

# sed -n '/VMR_HGRccp::VMR_HGRccp Entered/=' post-restart_ksys.log
597
# sed '/VMR_HGRccp::VMR_HGRccp Entered/,/VMR_HGRccp::VMR_HGRccp Leaving/!d'
[15] 11/19/18 T(304) _VMR 13:49:26.231318 VMR_HGRccp::VMR_HGRccp Entered
[15] 11/19/18 T(304) _VMR 13:49:26.231400 VMR_HGRccp::createRcp Entered
[15] 11/19/18 T(304) _VMR 13:49:26.231409 DEBUG VMR_HG.C[828]: setting its
redundancy to 2
[15] 11/19/18 T(304) _VMR 13:49:26.231411 DEBUG VMR_HG.C[833]: create:
RestartPolicy is autorestart
HAmonitor to Enabled
hostFDT to 90
VMFDS to
failover policy to to best_fit
spawnFDE to 1

PreHAmonitor to Enabled
[15] 11/19/18 T(304) _VMR 13:49:26.231449 DEBUG VMR_HG.C[944]: Setting its
DisocveryState =1
[15] 11/19/18 T(304) _VMR 13:49:26.231464 VMR_HGRcp::VMR_HGRcp Entered

[15] 11/19/18 T(304) _VMR 13:49:26.231468 DEBUG VMR_HG.C[2825]: Initializing
the mutex in VMR_HGRcp::VMR_HGRcp
the mutex its_VMutex in VMR_HGRcp::VMR_HGRcp
the condition in VMR_HGRcp::VMR_HGRcp
Relocationmap Initialized. rc = 0
[15] 11/19/18 T(304) _VMR 13:49:26.231476 DEBUG VMR_threads.C[1201]:
EventProcessor rc from pthread_mutex_init is 0
[15] 11/19/18 T(304) _VMR 13:49:26.231543 DEBUG VMR_threads.C[1205]: rc from
pthread_create is 0
[15] 11/19/18 T(304) _VMR 13:49:26.231544 DEBUG VMR_threads.C[1206]: evtThID
is 37009
[15] 11/19/18 T(304) _VMR 13:49:26.231546 DEBUG VMR_HG.C[2864]: Created the
EventProcessor for HG: rbHG(id:1)
[15] 11/19/18 T(304) _VMR 13:49:26.231546 VMR_HGRcp::VMR_HGRcp Leaving
[15] 11/19/18 T(304) _VMR 13:49:26.231561 DEBUG VMR_HG.C[604]: Updated
its_activeSiteID = 0, its_SiteID1 = 0, its_PairSiteID = 0
its_Priority = 1
its_Redundancy = 2
[15] 11/19/18 T(304) _VMR 13:49:26.231586 DEBUG VMR_HG.C[2281]: Entered
printRCPList, size of list is 1
[15] 11/19/18 T(304) _VMR 13:49:26.231587 DEBUG VMR_HG.C[2292]: Entered
printRcpElements
[15] 11/19/18 T(304) _VMR 13:49:26.231589 DEBUG VMR_HG.C[2294]: name is rbHG,
HG id is 1 , activeSiteID = 0, number of hosts is 2, site ID is 0
[15] 11/19/18 T(304) _VMR 13:49:26.231590 DEBUG VMR_HG.C[2295]: number of PAIR
hosts is 0, pair site ID is 0, priority is 1
[15] 11/19/18 T(304) _VMR 13:49:26.231590 DEBUG VMR_HG.C[2298]: Leaving
printRcpElements
VmtoCECMapTable:
VM:4E395E99-FFFB-4345-8067-2D5296BA7D91, CEC:4462c37c-65c6-3614-b02a-aa09d752c2ee,
Misc:restart_cleanup, M, best_fit
UserVmtoCECMapTable:
[15] 11/19/18 T(304) _VMR 13:49:26.231628 DEBUG VMR_HG.C[14786]: updated
its_Flexmem_table
[15] 11/19/18 T(304) _VMR 13:49:26.231635 DEBUG VMR_HG.C[14798]: updated
its_Flexproc_table
[15] 11/19/18 T(304) _VMR 13:49:26.231637 DEBUG VMR_HG.C[1018]: Setting the
scheduler to resyncApps
[15] 11/19/18 T(304) _VMR 13:49:26.231641 DSchedCb::schedOnce Scheduling ONCE
delay = 10000
[15] 11/19/18 T(304) _VMR 13:49:26.231643 DSchedCb::schedOnce called

[15] 11/19/18 T(304) _VMR 13:49:26.231658 DEBUG VMR_HG.C[9385]: ResyncAPPs
Scheduler added.
[15] 11/19/18 T(304) _VMR 13:49:26.231659 VMR_HGRccp::createRcp Leaving,
pRcp=3263e420
[15] 11/19/18 T(304) _VMR 13:49:26.231759 VMR_HGRccp::VMR_HGRccp Leaving
# sed -n '/VMR_HGRccp::VMR_HGRccp Leaving/=' post-restart_ksys.log
637
#
The VMR_HGRccp::createRcp method is entered once, and inside this method the
VMR_HGRcp::VMR_HGRcp RCP constructor is also entered only one time. We defined only the
rbHG host group in our example, so these log entries apply to this unique host group.
Among the various configuration and state attributes being initialized, we notice that an FDE
thread is referred to is inside the VMR_HGRccp::createRcp call (setting its spawnFDE to 1). We
also notice that an EventProcessor thread is created (evtThID is 37009), which, by pTID, is
exactly our earlier assumed event thread Handler, that is, the one with the distinctive
eventHandler() function launched in its stack. We derive from these findings that dedicated
FDE and event thread Handlers are created for each existing host group.
The last relevant action that we notice in the HGRccp creation log is the scheduling of a
resyncApps action (ResyncAPPs Scheduler added), which is supposed to be run once and with
a delay parameter of 10000. The value appears to be in milliseconds because 10 seconds
later the scheduler thread, T(203), calls the HG_ResyncShed::callback function followed by
the VMR_HGRcp::resyncAppInfo method. These calls are logged at 13:49:36.234278 (see ksys
log lines 761 - 762 in Example 2-42 on page 80) and in Example 2-31 on page 69,
DSchedCb::schedOnce is called at 13:49:26.231643, which is a 10-second difference.
The next RCCP section in the sequence is for the SSP, and it is shown in Example 2-32.
Example 2-32 Shared Storage Pool RCCP creation

# sed '/VMR_SSPRccp::VMR_SSPRccp Entered/,/VMR_SSPRccp::VMR_SSPRccp Leaving/!d'
[15] 11/19/18 T(304) _VMR 13:49:26.231898 VMR_SSPRccp::VMR_SSPRccp Entered
[15] 11/19/18 T(304) _VMR 13:49:26.231937 VMR_SSPRccp::createRcp Entered
[15] 11/19/18 T(304) _VMR 13:49:26.231957 VMR_SSPRcp::VMR_SSPRcp Entered
[15] 11/19/18 T(304) _VMR 13:49:26.231958 VMR_SSPRcp::VMR_SSPRcp Leaving
[15] 11/19/18 T(304) _VMR 13:49:26.232004 VMR_SSPRccp::createRcp Leaving,
pRcp=3263f7d0
[15] 11/19/18 T(304) _VMR 13:49:26.232014 VMR_SSPRccp::VMR_SSPRccp Leaving
#
Again, only one SSP RCP is created because we have only one host group that is defined, so
there is only one SSP.
The last RCCP section in the sequence is for the managed applications (Example 2-33).
Example 2-33 Application RCCP creation

# sed '/VMR_APPRccp::VMR_APPRccp Entered/,/VMR_APPRccp::VMR_APPRccp Leaving/!d'
[15] 11/19/18 T(304) _VMR 13:49:26.232153 VMR_APPRccp::VMR_APPRccp Entered
[15] 11/19/18 T(304) _VMR 13:49:26.232155 DEBUG VMR_APP.C[257]: Inside
VMR_APPRccp constructor
[15] 11/19/18 T(304) _VMR 13:49:26.232155 DEBUG VMR_APP.C[262]: VMR_APPRccp
calling createRCPs.

[15] 11/19/18 T(304) _VMR 13:49:26.232189 VMR_APPRccp::createRcp Entered
VMR_APPRccp::CreateRCP
[15] 11/19/18 T(304) _VMR 13:49:26.232196 DEBUG VMR_APP.C[435]:
VMR_APPRccp::CreateRCP Calling VMR_APPRcp function.
[15] 11/19/18 T(304) _VMR 13:49:26.232201 VMR_APPRcp::VMR_APPRcp Entered
VMR_APPRcp function.
[15] 11/19/18 T(304) _VMR 13:49:26.232204 DEBUG VMR_APP.C[1688]: Leaving
VMR_APPRcp
[15] 11/19/18 T(304) _VMR 13:49:26.232205 VMR_APPRcp::VMR_APPRcp Leaving
set_AppLparDetail:Updating APP Lpar
details.Name:rt13001,UUID:4E395E99-FFFB-4345-8067-2D5296BA7D91
[15] 11/19/18 T(304) _VMR 13:49:26.232208 DEBUG VMR_APP.C[2086]: Updating app
status.
[15] 11/19/18 T(304) _VMR 13:49:26.232219 VMR_APPRccp::createRcp Leaving,
pRcp=32634170
VMR_APPRccp::CreateRCP leaving.
[15] 11/19/18 T(304) _VMR 13:49:26.232224 VMR_APPRccp::VMR_APPRccp Leaving
#
Only one application RCP is created because we have only one application that is defined
(Example 2-34).
Example 2-34 Defined applications

# ksysmgr q app
Name: tsApp
Status: GREEN
AppID: 1541473063965437000
VM: rt13001
Critical: no
MaxRestart: 3
Monitoring: 1
#
Quick Discovery and Deep Discovery scheduling records

Going further with the log entries, we notice a new group of scheduling-related records
immediately after the last RCCP creation section, as shown in Example 2-35, which lists the
next 24 recorded lines in the ksys log after the last RCCP constructor returns.
Example 2-35 Scheduler records in the ksys trace file after daemon restart
# sed '/VMR_APPRccp::VMR_APPRccp Leaving/,$!d;=' post-restart_ksys.log |sed
'N;s/\n/:/'|head -25
660:[15] 11/19/18 T(304) _VMR 13:49:26.232224 VMR_APPRccp::VMR_APPRccp Leaving
661:
662:[15] 11/19/18 T(304) _VMR 13:49:26.232225 DEBUG VMR_APP.C[274]: Leaving VMR_APPRccp
663:[15] 11/19/18 T(304) _VMR 13:49:26.232226 VMR_HMCRccp::schedule Entered.
664:[15] 11/19/18 T(304) _VMR 13:49:26.232228 DSchedCb::schedPeriodic starting HMC Ping
Interval=10
665:[15] 11/19/18 T(304) _VMR 13:49:26.232230 VMR_HMCRccp::schedule Leaving, pCycle=10.
666:
667:[15] 11/19/18 T(304) _VMR 13:49:26.232231 DEBUG VMRRmcp.C[241]: VMRRmcp Init, START

668:[15] 11/19/18 T(304) _VMR 13:49:26.232232 VMR_SITERccp::scheduleQuickCheck Entered.
669:[15] 11/19/18 T(304) _VMR 13:49:26.232233 DEBUG VMR_SITE.C[5261]: ***** SCHEDULE
QUICK *****
670:[15] 11/19/18 T(304) _VMR 13:49:26.232236 DEBUG VMR_SITE.C[5287]: qIntvl = 1 hrs
671:[15] 11/19/18 T(304) _VMR 13:49:26.232237 DSchedCb::schedPeriodic starting SITE
Quick Check Schedule Interval=3600
672:[15] 11/19/18 T(304) _VMR 13:49:26.232238 VMR_SITERccp::scheduleQuickCheck Leaving.
673:
674:[15] 11/19/18 T(304) _VMR 13:49:26.232239 VMR_SITERccp::scheduleDetailCollectOnce
Entered.
675:[15] 11/19/18 T(304) _VMR 13:49:26.232287 DEBUG VMR_SITE.C[5150]: ***** SCHEDULE
DETAILED COLLECT ONCE *****
676:[15] 11/19/18 T(304) _VMR 13:49:26.232288 DEBUG VMR_SITE.C[5153]:
.... number of seconds since epoch = 1542693600
677:[15] 11/19/18 T(304) _VMR 13:49:26.232289 DSchedCb::schedOnce Scheduling ONCE time in
sec = 1542693600
678:[15] 11/19/18 T(304) _VMR 13:49:26.232291 DSchedCb::schedOnce called
679:[15] 11/19/18 T(304) _VMR 13:49:26.232291 VMR_SITERccp::scheduleDetailCollectOnce
Leaving.
680:
681:[15] 11/19/18 T(304) _VMR 13:49:26.232293 DEBUG VMRRmcp.C[244]: VMRRmcp Init, FINISH
682:[15] 11/19/18 T(304) _VMR 13:49:26.232294 DEBUG VMRDaemon.C[275]: INFO: Calling
kri_global_init.
683:[15] 11/19/18 T(9091) _VMR 13:49:26.242487 DEBUG VMR_threads.C[1284]: Entered event
thread Handler. HG: 1
684:[15] 11/19/18 T(9091) _VMR 13:49:26.242490 DEBUG VMR_threads.C[1290]: Got the HGRcp
# sed '/VMR_APPRccp::VMR_APPRccp Leaving/,$!d;=' post-restart_ksys.log |sed
'N;s/\n/:/'|head -25|grep -c "T(304)"
19
# sed '683,$!d;=' post-restart_ksys.log|sed 'N;s/\n/:/'|grep "T(304)"
#
After finishing with the last RCCP, as part of a VMRRmcp Init piece of code, a periodic
QuickCheck action is scheduled at 1 hour and a DetailCollectOnce action is scheduled at an
epoch time. Converting the epoch time 1542693600 to a human-readable format, we see that it
is the next midnight moment (local CST time, Greenwich mean time: Tuesday 20 Nov 2018
6:00:00 AM) compared with the execution moment (11/19/18 T(304) _VMR
13:49:26.232288). The scheduling is done at the VMR_SITERccp method level, so we conclude
it applies for all host groups in the environment. This is the scheduling setup for the periodic
Quick Discovery and for the Deep Discovery tasks. For more information, see “Quick
Discovery and Deep Discovery scheduling records” on page 72.
We also see from the last command in Example 2-35 on page 72 that the last record that is
generated by the T(304) thread in our ksys log is about a global initialization of the krest
library module in line 682 (Calling kri_global_init.). So after line 682, we expect records
that are logged by the other threads: the FDE thread, event thread Handler, task threads in
the pool, scheduler thread, and so on.
Review the restart moment and the initial records that are logged in the ksys log up until the
first RCCP is created (Example 2-36).
Example 2-36 Detailed startup sequence of a KSYS daemon restart

# sed '1,13!d;=' post-restart_ksys.log|sed 'N;s/\n/:/'
1:[15] 11/19/18 T( 1) ____ 13:48:35.161478 ******************* Trace Started -
Pid = 16646518 **********************
2:[15] 11/19/18 T( 1) _VMR 13:48:35.161711VMRDaemon::VMRDaemon Leaving
3:[15] 11/19/18 T( 1) _VMR 13:48:35.162523VMRRmcp::VMRRmcp Entered

4:[15] 11/19/18 T( 1) _VMR 13:48:35.162729VMRRmcp::VMRRmcp Leaving
5:[15] 11/19/18 T( 1) _VMR 13:48:35.162828IBM.VMRd: Resource manager has been
successfully initialized.
6:[15] 11/19/18 T(304) _VMR 13:48:35.172692 VMRDaemon.C(177):[ INFO,VMRDaemon]
Start LPAR category logging
7:[15] 11/19/18 T(304) _VMR 13:48:35.172697 VMRDaemon.C(178):[ INFO,VMRDaemon]
Start Verify category logging
8:[15] 11/19/18 T(304) _VMR 13:48:38.196786 DEBUG VMRDaemon.C[227]: INFO: VMs not
provided. Selecting all the VMs ###############.
9:[15] 11/19/18 T(304) _VMR 13:48:38.196789 DEBUG VMRDaemon.C[235]: INFO:
KSYS_TST_CG: VMRDG_ KSYS_TST_VM:
10:[15] 11/19/18 T(304) _VMR 13:48:38.196791 DEBUG VMRDaemon.C[238]: vmr_restart
value set to 1
11:
12:[15] 11/19/18 T(304) _VMR 13:48:38.196792 DEBUG VMRDaemon.C[247]: INFO:
skip_policyMap = 0
13:[15] 11/19/18 T(304) _VMR 13:48:38.197869 VMR_SITERccp::VMR_SITERccp Entered
#
You can easily count the first five entries that are posted by the main thread (T( 1)), which
matches the number of traces that is left by the main thread in our whole ksys log excerpt, as
shown in Table 2-1 on page 65.
The main thread then hands off the execution control to the created thread T(304). This
created thread logs six initial records, and then handles the whole sequence of RCCPs and
RCPs initialization, as shown in Example 2-28 on page 67. We counted 602 records that are
signed by T(304). The number of records that are signed by T(304) in Example 2-35 on
page 72 after the return from the last RCCP constructor is 18 (19 - 1). Adding up all these
occurrences, we get exactly the number for T(304) that is shown in Table 2-1 on page 65
(6+602+18=626).
Event thread Handler start

We saw in Example 2-35 on page 72 that the last record that was left by the T(304) thread in
our ksys log is 682. Let us continue with the next 25 records (Example 2-37).
Example 2-37 Entering the event thread Handler

# sed '683,$!d;=' post-restart_ksys.log|sed 'N;s/\n/:/'|head -25
683:[15] 11/19/18 T(9091) _VMR 13:49:26.242487 DEBUG VMR_threads.C[1284]: Entered
event thread Handler. HG: 1
684:[15] 11/19/18 T(9091) _VMR 13:49:26.242490 DEBUG VMR_threads.C[1290]: Got the
HGRcp
685:[15] 11/19/18 T(9091) _VMR 13:49:26.242491 DEBUG VMR_threads.C[1298]:
recovery_LPAR function triggered
686:[15] 11/19/18 T(9091) _VMR 13:49:26.242732 DEBUG VMR_HG.C[15217]: Fetching the
HAstate for lpar 017D6CF0-4262-40FD-B83F-34EF0D58CFE5
HAstate for lpar 18554EB7-180D-41D5-988C-9742F1CB6F9F
HAstate for lpar 192575B8-E6A7-4E67-B77C-049C45ECD027
HAstate for lpar 4E395E99-FFFB-4345-8067-2D5296BA7D91
HAstate for lpar 192ABBED-306B-4D6B-B8A2-9F7C3BF9D2E6

HAstate for lpar 31603D8C-5A2F-437C-BA33-F25321415CC4
HAstate for lpar 57275F1E-3680-4EEF-945A-FE7670500310
HAstate for lpar 626FC239-7BF5-4976-B8C0-6D0C7C9ED9EB
694:[15] 11/19/18 T(9091) _VMR 13:49:26.243096 DEBUG VMR_HG.C[2324]: STATE: Phase
is being changed from: DISCOVER_ONLY to: READY
695:[15] 11/19/18 T(9394) _VMR 13:49:26.246393 VMRRmcp::connectionChanged entered,
conn_changed=1.
696:[15] 11/19/18 T(9394) _VMR 13:49:26.246394 VMRRmcp::connectionChanged leaving.
697:[15] 11/19/18 T(9192) _VMR 13:49:26.265532 DEBUG VMR_HMC.C[3578]: Session is
Empty !!!. Trying to get HMC Session.
698:[15] 11/19/18 T(9192) _VMR 13:49:26.265536 DEBUG VMR_HMC.C[3630]: Calling
krigetSessionId(). HMCIP=9.3.18.159, UserName=hscroot, Passwd=*****
699:[15] 11/19/18 T(9394) _VMR 13:49:26.476082 VMR_SITERccp::validateSetClassParms
Entered, number of values=57.
700:[15] 11/19/18 T(9394) _VMR 13:49:26.476093 VMR_SITERccp::validateSetClassParms
Leaving.
701:
702:[15] 11/19/18 T(809) _VMR 13:49:26.478390 VMR_HMCRccp::chgClassCommitted
Entered.
703:[15] 11/19/18 T(809) _VMR 13:49:26.478391 VMR_HMCRccp::chgClassCommitted
Leaving.
704:
705:[15] 11/19/18 T(9192) _VMR 13:49:27.865262 DEBUG VMR_HMC.C[6938]:
getQuickQuery [315CE56B-9BA0-46A1-B4BC-DA6108574E7E] JobStatus: COMPLETED_OK,
ReturnCode: 0
706:[15] 11/19/18 T(9192) _VMR 13:49:27.865475 DEBUG VMR_VIOS.C[4156]: set the
RelocInitiated value for VIOS 5AC3E0DC-13FC-4221-9617-8EF61C4CDD83 as 0
707:
# sed '683,$!d;=' post-restart_ksys.log|sed 'N;s/\n/:/'|head -25|grep -c
"T(9091)"
12
# sed '695,$!d;=' post-restart_ksys.log|grep -c "T(9091)"
0
#
In Example 2-37 on page 74, note the following facts:

򐂰 Entering the event thread Handler T(9091) triggers the VM recovery function for all eight
managed VMs in our configured host group, and the host group goes to the READY state.
򐂰 Some discreet RSCT activity that is done by the T(9394) and T(809) threads is detected.
This activity is state updates for resources in the RMC registry.
򐂰 The FDE thread T(9192) appears with its first ksys records, one of which is about opening
a REST API session with the HMC, and the other one about performing a first
getQuickQuery request.
򐂰 After logging this first group of 12 records about the VM recovery activation, our event
handler remains quiet until the end of the ksys log excerpt.

FDE thread start and the first QuickQuery
We try to correlate the records of the FDE thread in the ksys log excerpt with its beginning
records in the counterpart fde log excerpt (Example 2-38). We extracted these logs from
trace files, as shown in Example 2-23 on page 64.
Example 2-38 Initial records in the fde trace file after daemon restart
# sed '1,$!d;=' post-restart_fde.log|sed 'N;s/\n/:/'|head -25
1:[06] 11/19/18 T( 1) ____ 13:48:35.140684 ******************* Trace Started -
Pid = 16646518 **********************
2:[06] 11/19/18 T(9192) _VMR 13:49:26.243690 DEBUG FDEthread.C[69]: Entering
3:[06] 11/19/18 T(9192) _VMR 13:49:26.265358 DEBUG FDEthread.C[127]: Monitoring
Enabled for HG rbHG.
4:[06] 11/19/18 T(9192) _VMR 13:49:26.265375 DEBUG FDEthread.C[145]: CEC is
4462c37c-65c6-3614-b02a-aa09d752c2ee
5:[06] 11/19/18 T(9192) _VMR 13:49:26.265383 DEBUG FDEthread.C[158]: VIOS
315CE56B-9BA0-46A1-B4BC-DA6108574E7E in CEC
6:[06] 11/19/18 T(9192) _VMR 13:49:26.265391 DEBUG FDEthread.C[208]: Use VIOS
rt13v2: 315CE56B-9BA0-46A1-B4BC-DA6108574E7E for polling
7:[06] 11/19/18 T(9192) _VMR 13:49:26.265394 DEBUG FDEthread.C[285]: Current scan
[ 1 ]
8:[06] 11/19/18 T(9192) _VMR 13:49:26.265412 DEBUG VMR_VIOS.C[6134]:
setCAAtopology
setCAAtopology
setCAAtopology
setCAAtopology
12:[06] 11/19/18 T(9192) _VMR 13:49:26.265471 DEBUG VMR_HG.C[11664]: FDE
performing doQuickQuery to 315CE56B-9BA0-46A1-B4BC-DA6108574E7E
13:[06] 11/19/18 T(9192) _VMR 13:49:26.265475 DEBUG VMR_retry.C[1151]: Doing
14:[06] 11/19/18 T(9192) _VMR 13:49:26.265495 DEBUG VMR_retry.C[178]: INFO: Trying
with HMC: rthmc3.
getQuickQuery: Calling kriSubmitQuickQuery!. HMC:9.3.18.159, viosUuid:
315CE56B-9BA0-46A1-B4BC-DA6108574E7E
getQuickQuery: Job submitted. Now doing WaitTillJobCompletion() ..
krigetJobResult(). HMC: 9.3.18.159, jobid: 1540961034191, retCnt = 1
19:[06] 11/19/18 T(9192) _VMR 13:49:27.865264 DEBUG VMR_HMC.C[6938]: getQuickQuery
20:[06] 11/19/18 T(9192) _VMR 13:49:27.865286 DEBUG VMR_retry.C[345]: In doRetry
function, for opCode = 33(VMDR_QUICK_QUERY), rc = 0, retCode is 0, errstr is:
,retry flag is 22
21:[06] 11/19/18 T(9192) _VMR 13:49:27.865296 DEBUG VMR_HG.C[11671]: FDE
doQuickQuery success to 315CE56B-9BA0-46A1-B4BC-DA6108574E7E
setCAAtopology
23:[06] 11/19/18 T(9192) _VMR 13:49:27.865303 DEBUG needAttn.C[1566]: START
handleVIOResponse scan [ 1 ].

24:[06] 11/19/18 T(9192) _VMR 13:49:27.865442 DEBUG needAttn.C[167]: machine_type
= 8286
25:[06] 11/19/18 T(9192) _VMR 13:49:27.865444 DEBUG needAttn.C[176]: model = 42A
#
Comparing the moment when the FDE thread entered its run (13:49:26.243690) with the
moment the event thread Handler entered (13:49:26.242487 in Example 2-37 on page 74),
we see that the FDE thread started 1.2 ms later. The FDE thread logs at its start details about
the VIOS that it uses for polling and then initiates its first scan, which is a QuickQuery. The
steps of the QuickQuery are as follows:
1. This first FDE polling scan is started by starting a doQuickQuery call, which is logged in the
fde trace file at 13:49:26.265471, and also logged as tried by the rthmc3 HMC at
13:49:26.265495.
2. So, the FDE thread logs its first message in the ksys trace file, which is the record about
the empty HMC session found, at 13:49:26.265532 (Example 2-37 on page 74).
3. After the new HMC session is opened at 13:49:26.265536 (Example 2-37 on page 74), the
QuickQuery request is submitted to a VIOS at 13:49:26.517072 (Example 2-38 on
page 76).
4. The response is obtained by job polling, and a record about its completion is logged in the
ksys trace file at 13:49:27.865262.
5. The first FDE doQuickQuery call logs its return at 13:49:27.865296 1.6 seconds after its
invocation.
The whole sequence of HMC and VIOS low-level steps for this first QuickQuery request after
the daemon restart as they are logged in the krest and krestlog trace files is described in
“QuickQuery asynchronous job example” on page 105. Also, a complete analysis of a generic
QuickQuery request flow, including the way its response payload content is parsed and
further processed by the KSYS side, is available in “QuickQuery flow” on page 234.
After the first QuickQuery occurrence at line 705 of our ksys log file and the subsequent two
lines that are shown in Example 2-37 on page 74 (706 - 707), we continue with the next
section of 25 rows from the ksys log, as shown in Example 2-39. All these lines are ksys
records that are logged by the FDE thread about VIOS- and CEC-related initialization actions
performed while parsing and processing this first QuickQuery response payload.
Example 2-39 FDE thread traces in the ksys log after the first QuickQuery
709:
710: CEC IS UP, READY FOR RELOCATION
711:
712:
713:[15] 11/19/18 T(9192) _VMR 13:49:27.866167 DEBUG VMR_VIOS.C[4172]: Adding
cleanLpars task for the cec 4462c37c-65c6-3614-b02a-aa09d752c2ee
714:[15] 11/19/18 T(9192) _VMR 13:49:27.866172 DEBUG VMR_LPAR.C[11257]: adding
task for clean_lpars
715:[15] 11/19/18 T(9192) _VMR 13:49:27.866180 DEBUG VMR_LPAR.C[11260]: Add task
is done for CLEAN_LPARS
716:[15] 11/19/18 T(9192) _VMR 13:49:27.866198 DEBUG VMR_SITE.C[6261]: INFO:
eventNotify entering. event:VIOS_CAA_STATE_UP_DETECTED, event type:3, comp:VIOS,
notificationLevel:low, dupEventProcess:yes
717:[15] 11/19/18 T(9192) _VMR 13:49:27.866540 DEBUG VMR_SITE.C[7096]: ERROR:
catopen Failed

718:[15] 11/19/18 T(3c3d) _VMR 13:49:27.867594 DEBUG VMR_LPAR.C[14099]: Entered
into CLEAN_LPARS task
eventNotify entering. event:VIOS_HM_RESP_DETECTED, event type:4, comp:VIOS,
RelocInitiated value for VIOS 315CE56B-9BA0-46A1-B4BC-DA6108574E7E as 0
721:
723:
725:
726:
cleanLpars task for the cec 4462c37c-65c6-3614-b02a-aa09d752c2ee
731:[15] 11/19/18 T(393a) _VMR 13:49:27.881922 DEBUG VMR_LPAR.C[14099]: Entered
#
We can easily identify the following actions, which are performed for each VIOS separately
(starting from line 708 in Example 2-37 on page 74, then going through Example 2-39 on
page 77 and Example 2-40, and finishing at line 758 in Example 2-42 on page 80):
򐂰 The RelocInitiated flag is set to 0, so the underlying CEC enters the READY FOR
RELOCATION state.
򐂰 The FDE thread hands off a clean_lpars task to one thread in the pool of task threads.
򐂰 The eventNotify service is launched for the VIOS_CAA_STATE_UP_DETECTED and
VIOS_HM_RESP_DETECTED events.
Example 2-40 FDE thread traces in the ksys log after the first QuickQuery (continued)
RelocInitiated value for VIOS 166C89EF-57D8-4E75-815C-AA5B885560B1 as 0
734:
736:
738:
739:
cleanLpars task for the cec 6e2f37f5-1824-347a-99fd-488f818086dd

744:[15] 11/19/18 T(4849) _VMR 13:49:27.895744 DEBUG VMR_LPAR.C[14099]: Entered
RelocInitiated value for VIOS 4E0F6F60-214B-4D2C-A436-0EAC46F7F71F as 0
747:
749:
751:
752:
cleanLpars task for the cec 6e2f37f5-1824-347a-99fd-488f818086dd
757:[15] 11/19/18 T(2d2e) _VMR 13:49:27.909960 DEBUG VMR_LPAR.C[14099]: Entered
#
All threads handling the CLEAN_LPARS tasks are listed in Example 2-41. You can easily check
in Table 2-1 on page 65 that they are all from the task thread pool.
Example 2-41 CLEAN_LPARS tasks in thread pool

# sed '705,$!d;=' post-restart_ksys.log|sed 'N;s/\n/:/'|grep "Entered into
CLEAN_LPARS task"
718:[15] 11/19/18 T(3c3d) _VMR 13:49:27.867594 DEBUG VMR_LPAR.C[14099]: Entered
731:[15] 11/19/18 T(393a) _VMR 13:49:27.881922 DEBUG VMR_LPAR.C[14099]: Entered
757:[15] 11/19/18 T(2d2e) _VMR 13:49:27.909960 DEBUG VMR_LPAR.C[14099]: Entered
#
Because our test environment has two hosts, each with two VIOSs, we are done with these
side actions that were triggered by the first succeeded FDE QuickQuery. Checking after the
moment FDE logs its last line, 758, as shown in Example 2-42 on page 80, we see a long
interval of 8 seconds during which nothing happens. Then, a new thread, T(203), wakes up at
13:49:36.234247 and starts logging continuously resyncAppInfo records.

Application resynchronization
The new thread that appears after the quiet interval is the scheduler action that was set
before the return from the HG Rcp constructor that are shown in Example 2-31 on page 69. We
compared the moment when the ResyncAPPs action was added in the scheduler for later
execution with the moment when the scheduled action was started by the scheduler, which
correlated with the delay parameter of 10000 milliseconds. The T(203) scheduler thread
appears here after these 10 seconds elapsed, which explains these 8 seconds of complete
silence in our ksys log.
The T(203) scheduler thread initiates the VMR_HGRcp::resyncAppInfo method call under
HG_ResyncShed::callback, as shown in Example 2-42. Then, a complex ResyncAPPs action is
performed by multiple threads in parallel.
Example 2-42 Scheduler thread triggering application resync actions

759:[15] 11/19/18 T(203) _VMR 13:49:36.234247 DEBUG VMR_HMC.C[1704]: Checking the
HMC state of site 0
760:[15] 11/19/18 T(203) _VMR 13:49:36.234252 DEBUG VMR_HMC.C[1545]: Found one
reachable HMC rthmc3 for site 0
761:[15] 11/19/18 T(203) _VMR 13:49:36.234278 DEBUG VMR_HG.C[9305]:
HG_ResyncShed::callback Called.
VMR_HGRcp::resyncAppInfo Entered
VMR_LPARRccp::getHGLPARList
764:[15] 11/19/18 T(203) _VMR 13:49:36.235011 DEBUG VMR_LPAR.C[15130]: Leaving
VMR_LPARRccp::getHGLPARList
VMR_HGRcp::resyncAppInfo Name: rt13007
VMR_HGRcp::resyncAppInfo addTask is invoked for resync

#
T(203) is the thread that we assume to be a scheduler one because of its distinctive
rmf3v::RMSchedule::run() invoked function, which shows up in the procstack command
output in Example 2-22 on page 62. All 30 messages this thread logs in our ksys trace file
excerpt, as we counted for T(203) row in Table 2-1 on page 65, appear contiguously in
Example 2-42 on page 80 and Example 2-43. The sed command at the end of Example 2-43
confirms this fact. The range of 30 lines is 759 - 788, and there is no other occurrence of a
message from this T(203) scheduler thread in the ksys trace file excerpt.
Example 2-43 Scheduler thread triggering application resync actions (continued)

HG_ResyncShed::callback Done!!
789:[15] 11/19/18 T(4041) _VMR 13:49:36.235297 DEBUG VMR_LPAR.C[13924]:
RESYNC_APP:CECusage:1
790:[15] 11/19/18 T(4041) _VMR 13:49:36.235299 DEBUG VMR_LPAR.C[18542]: In
update_HAState for state_bit of VM rt13007 to 0x20000000.
791:[15] 11/19/18 T(3d3e) _VMR 13:49:36.235833 DEBUG VMR_LPAR.C[13924]:
792:[15] 11/19/18 T(3d3e) _VMR 13:49:36.235835 DEBUG VMR_LPAR.C[18542]: In
793:[15] 11/19/18 T(90a) _VMR 13:49:36.235890 DEBUG VMR_LPAR.C[13924]:
794:[15] 11/19/18 T(90a) _VMR 13:49:36.235892 DEBUG VMR_LPAR.C[18542]: In

chgResourceCommited for HASTATE of VM rt13007 to 0x20000004.
update_HAState for HASTATE of VM rt13007 from 0x4 to 0x20000004.
805:[15] 11/19/18 T(4041) _VMR 13:49:36.238470 DEBUG VMR_LPAR.C[13944]: ResyncAPP:
Acquired preempt lock of the VM: rt13007.
resyncAllAppsInfo entered
resyncAllAppsInfo ::LparUuid:[017D6CF0-4262-40FD-B83F-34EF0D58CFE5],dbell:0x2
# sed '789,$!d;=' post-restart_ksys.log|sed 'N;s/\n/:/'|grep -c "T(203)"
0
# sed '759,788!d;=' post-restart_ksys.log|sed 'N;s/\n/:/'|grep -c "T(203)"
30
#
HG_ResyncShed::callback calls the VMR_HGRcp::resyncAppInfo host group RCP method,

which first retrieves the list of all VMs (16 for our host group) and then launches an addTask
call for the resync of each managed VM among them. Example 2-44 lists the VMs that are
configured as managed.
Example 2-44 Managed VMs at the daemon startup moment

# ksysmgr q vm state=manage|grep "^Name:"
Name: rt14007
Name: rt14002
Name: rt14001
Name: rt14005
Name: rt13001
Name: rt13002
Name: rt13003
Name: rt13007
#
Table 2-2 lists the resyncAPP task threads in the first column as they show up in the log (see
Example 2-43 on page 81) and the assigned managed VM for each thread separately in the
second column.
Table 2-2 The resyncAPP task threads and assigned VMs

resyncAPP task thread Assigned VM
T(4041) rt13007
T(3d3e) rt13002
T(90a) rt13001

resyncAPP task thread Assigned VM
T(8889) rt14005
T(8788) rt14002
T(8687) rt14001
T(8586) rt14007
The first records of each thread already are presented in Example 2-43 on page 81. Similar
sequences of records are logged by each thread, and the records are interlaced as the
threads run in parallel. The last record of each sequence is shown in Example 2-52 on
page 88. We further examine in detail the sequence of the first resyncAPP task thread
T(4041), which handles the rt13007 VM.
In Example 2-45, we select the records of the T(4041) thread that is present in Example 2-43
on page 81 and also include the particular record showing up there as logged by the T(405)
thread because it is also related to the rt13007 VM.
Example 2-45 Start of the resyncAPP flow for the rt13007 VM

update_HAState for HASTATE of VM rt13007 from 0x4 to 0x20000004.
resyncAllAppsInfo entered
resyncAllAppsInfo ::LparUuid:[017D6CF0-4262-40FD-B83F-34EF0D58CFE5],dbell:0x2
These first records of the resyncAPP flow for the rt13007 VM show the required VM state
changes to start the actual flow. A highly significant state bit is set so that the VM HASTATE is
0x4 - 0x20000004. The state update appears to be even performed in the RMC persistent
registry by the T(405) thread, as suggested by the name of the function call
chgResourceCommited. We already see in Table 2-1 on page 65 that the distinctive stack call
for the T(405) thread is rsct_gscl_V1::GSController::dispatch(), so we rather associate
the T(405) thread with the core RSCT-related activity. The initial 0x4 state value matches the
HAState attribute value for the IBM.VMR_LPAR RMC resource of the rt13007 VM
(Example 2-46). We also check at that moment the HA_monitor_state VM attribute at the
ksysmgr level, which was set to the STARTED value.
Example 2-46 VM HA state attribute for the rt13007 VM

# ksysmgr -v q vm rt13007|grep -i stat
State: READY
VM_status: NO_OPERATION_IN_PROGRESS
HA_monitor_state: STARTED
RMC_state: active
LPM Validation Status
# lsrsrc -s 'Name="rt13007"' IBM.VMR_LPAR|grep -i stat

HMstate = "STARTED"
HAState = 4
VerifyState =
{["6e2f37f5-1824-347a-99fd-488f818086dd","Fail","Errors:
PartitionState = "running"
RmcState = "active"
MigrationState = ""
RRestartState = ""
HMstateHM1 = "STARTED"
HMstateHM2 = "STARTED"
# ksysmgr q vm rt13007|grep ÛUID
UUID: 017D6CF0-4262-40FD-B83F-34EF0D58CFE5
#
Continuing with the resyncAPP flow of rt13007 VM, we then see that a lock is acquired for the
VM. Then, the resyncAllAppsInfo function is entered and a dbell:0x2 code appears that is
associated with the rt13007 VM. The dbell identifier is a short name for Door Bell. Door Bell
is a short notification coming from the VM Agent toward the KSYS to notify you that some
status or configuration changes happened at the application level. KSYS reacts to the
notification by initiating a communication session where all relevant application details are
requested and obtained from the VM Agent. The dbell concept is covered together with the
Application Reporting (AR) Messaging protocol in 2.3.3, “Application Reporting flow” on
page 262.
Let us examine the rest of the records in the sequence that are logged by the T(4041) thread
(Example 2-47 and Example 2-48 on page 85).
Example 2-47 The resyncAPP flow for the rt13007 VM

# sed '808,$!d;=' post-restart_ksys.log|sed 'N;s/\n/:/'|grep "T(4041)"|wc -l
50
# sed '808,$!d;=' post-restart_ksys.log|sed 'N;s/\n/:/'|grep "T(4041)"|head -25
exchangeMsgWith_VM_HM::command:0x2
isVM_ProperState::State:0x20000004
810:[15] 11/19/18 T(4041) _VMR 13:49:36.241021 DEBUG VMR_LPAR.C[20647]: Entering
generateKSYSMesg
811:[15] 11/19/18 T(4041) _VMR 13:49:36.241046 DEBUG VMR_LPAR.C[20673]: Leaving
generateKSYSMesg : <Command code="0x2"/>
exchangeMsgWith_VM_HM::hsTag:1542894422 seqNub:1
doPutMsg::putMsgPayload::30250758:<Command code="0x2"/>
doPutMsg::CEC: 4462c37c-65c6-3614-b02a-aa09d752c2ee, vios:
315CE56B-9BA0-46A1-B4BC-DA6108574E7E
operation with opCode: 47(VMDR_PUT_MSG)
817:[15] 11/19/18 T(4041) _VMR 13:49:36.241211 DEBUG VMR_retry.C[178]: INFO:
Trying with HMC: rthmc3.

function, for opCode = 47(VMDR_PUT_MSG), rc = 0, retCode is 0, errstr is: ,retry
flag is 22
doMsg::putMsg successfully placed to HM.
operation with opCode: 46(VMDR_QUERY_MSG)
function, for opCode = 46(VMDR_QUERY_MSG), rc = 0, retCode is 0, errstr is: ,retry
flag is 22
flag is 22
#
A messaging flow is initiated by KSYS exactly as in the case where a dbell:0x2 notification
was received from the VM Agent. The retrieved result in this particular case, “No data from
XML”, means that no application is configured on the rt13007 VM.
Example 2-48 The resyncAPP flow for the rt13007 VM (continued)

# sed '949,$!d;=' post-restart_ksys.log|sed 'N;s/\n/:/'|grep "T(4041)"
flag is 22
operation with opCode: 48(VMDR_GET_MSG)

function, for opCode = 48(VMDR_GET_MSG), rc = 0, retCode is 0, errstr is: ,retry
flag is 22
doGetMsg::getMsg successfully got message.
operation with opCode: 49(VMDR_ACK_MSG)
function, for opCode = 49(VMDR_ACK_MSG), rc = 0, retCode is 0, errstr is: ,retry
flag is 22
doAckMsg::Ack completed successfully for tag:1542894422
970:[15] 11/19/18 T(4041) _VMR 13:49:42.055612 DEBUG VMR_LPAR.C[23205]: Parse
Message for complete info
parseAppsInfo::Application detail related doorbell received.:0
972:[15] 11/19/18 T(4041) _VMR 13:49:42.055736 DEBUG VMR_LPAR.C[20713]: Document
correctly cur
parseKSYSMsg Leaving
parseAppsInfo::existingAppUuidList has change.0
parseAppsInfo::No data from XML
Released preempt lock of the VM: rt13007.
clear_HAState for state_bit of VM rt13007 to 0x20000000.
clear_HAState for HASTATE of VM rt13007 from 0x20000004 to 0x4.
980:[15] 11/19/18 T(4041) _VMR 13:49:42.061668 DEBUG VMR_CEC.C[9776]: usage_count
2
#
Example 2-49 lists one record for each of the task threads that resulted from the previous
addTask invocations. Each thread handles the resyncapp action for its assigned VM. All
threads are from the task thread pool, which can be checked by TID in Table 2-1 on page 65.
Example 2-49 The resyncAPP task threads from thread pool

# sed '789,$!d;=' post-restart_ksys.log|sed 'N;s/\n/:/'|grep "ResyncAPP: Acquired
preempt lock of the VM"

888:[15] 11/19/18 T(90a) _VMR 13:49:36.429445 DEBUG VMR_LPAR.C[13944]: ResyncAPP:
904:[15] 11/19/18 T(3d3e) _VMR 13:49:36.462762 DEBUG VMR_LPAR.C[13944]: ResyncAPP:
#
We expect all threads to follow the same flow of execution, but we are more interested in the
result of the thread T(90a), which deals with the VM rt13001, as we know it is the only VM
that has a configured application (Example 2-50).
Example 2-50 Configured applications

# ksysmgr q app
Name: tsApp
Status: GREEN
AppID: 1541473063965437000
VM: rt13001
Critical: no
MaxRestart: 3
Monitoring: 1
Again, we do not get into many details of the resyncAPP flow of VM rt13001 because this topic
is covered in 2.3.3, “Application Reporting flow” on page 262. Example 2-51 shows the result
for the application running on the VM rt13001.
Example 2-51 Result of the resyncAPP flow for the rt13001 VM

# sed '789,$!d;=' post-restart_ksys.log|sed 'N;s/\n/:/'|grep "T(90a)"|tail -30
doAckMsg::Ack completed successfully for tag:1543092222
1055:[15] 11/19/18 T(90a) _VMR 13:49:43.338455 DEBUG VMR_LPAR.C[23205]: Parse
Message for complete info
parseAppsInfo::Application detail related doorbell received.:0
1057:[15] 11/19/18 T(90a) _VMR 13:49:43.338751 DEBUG VMR_LPAR.C[20713]: Document
correctly cur
parseKSYSMsg Leaving
1059:[15] 11/19/18 T(90a) _VMR 13:49:43.339384 DEBUG VMR_APP.C[1617]:
getLparAppUuidList:: LparUUID:[4E395E99-FFFB-4345-8067-2D5296BA7D91],
AppUuid:[1541473063965437000]
parseAppsInfo::existingAppUuidList has change.0
prepareAppAttrList::Entered.
prepareAppAttrList::AppUUID:1541473063965437000
prepareAppAttrList::App Rcp is available.
getAllAttrib_of_App::Entered.:1541473063965437000
getAllAttrib_of_App::Leaving.

prepareAppAttrList::All existing app details has been filled up.
extractAttrFromMap::Entered.
extractAttrFromMap::Leaving.
parseAppsInfo::prepareAppAttrList:1
parseAppsInfo::compare_And_DoUpdate calling
compare_And_DoUpdate::Entered.
compare_current_vs_new:: Entered.1541473063965437000
compare_current_vs_new:: Leaving.
checkRebootOrRelocation::Entered. AppUuid:1541473063965437000
APP:[1541473063965437000]:isMonitored:1:critical:no:color:GREEN
checkRebootOrRelocation::Leaving. AppUuid:1541473063965437000
APP:[1541473063965437000] not required any action on LPAR.
1078:[15] 11/19/18 T(90a) _VMR 13:49:43.339898 DEBUG VMR_APP.C[1452]: No Action
needed on LPAR[4E395E99-FFFB-4345-8067-2D5296BA7D91].
compare_And_DoUpdate::Leaving.
1080:[15] 11/19/18 T(90a) _VMR 13:49:43.343607 DEBUG VMR_LPAR.C[13947]: ResyncAPP:
Released preempt lock of the VM: rt13001.
clear_HAState for state_bit of VM rt13001 to 0x20000000.
1084:[15] 11/19/18 T(90a) _VMR 13:49:43.346949 DEBUG VMR_CEC.C[9776]: usage_count
1
#
As in the ksysmgr output of Example 2-50 on page 87, we see in the log records of
Example 2-51 on page 87 that the resyncAPP retrieved result contains details about only one
application that was discovered, which is AppUuid:1541473063965437000 with a GREEN status.
Now that we clarified the resyncAPP flow, we identify the latest logged line of all resyncAPP
tasks and continue from there. Example 2-52 shows the last line of each thread separately so
that it is easy to identify that the latest line is 1169.
Example 2-52 Last logged line of each resyncAPP task

# sed '789,$!d;=' postrestart_ksys.log|sed 'N;s/\n/:/'|sed -n '/T(4041)/h;$!b;g;p'
980:[15] 11/19/18 T(4041) _VMR 13:49:42.0616 DEBUG VMR_CEC.C[9776]: usage_count 2

# sed '789,$!d;=' postrestart_ksys.log|sed 'N;s/\n/:/'|sed -n '/T(90a)/h;$!b;g;p'
1084:[15] 11/19/18 T(90a) _VMR 13:49:43.3469 DEBUG VMR_CEC.C[9776]: usage_count 1
# sed '789,$!d;=' postrestart_ksys.log|sed 'N;s/\n/:/'|sed -n '/T(3d3e)/h;$!b;g;p'
1097:[15] 11/19/18 T(3d3e) _VMR 13:49:43.3750 DEBUG VMR_CEC.C[9776]: usage_count 0
#
We also must identify the lines that are logged in 789 - 1169 by other threads than those
handling the resyncAPP tasks (Example 2-53).
Example 2-53 Records in resyncAPP flows not logged by resyncAPP task threads
# sed '789,1169!d;=' post-restart_ksys.log|sed 'N;s/\n/:/'|egrep -v
"T$4041$|T$8586$|T$8687$|T$8788$|T$8889$|T$90a$|T$3d3e$"
#
So, only T(405) and T(809) inserted some records in this range (among the bulk resyncAPP
related records). All of the records are about updating the HASTATE persistent attribute, as
shown in Example 2-45 on page 83.
Steady state periodic actions

After resyncAPP tasks are complete, the ksys daemon goes to a steady state in which periodic
actions are run as follows:
򐂰 Two QuickQuery + 1x NeedAttention actions every 20s, which are performed by FDE.
򐂰 A QuickDiscovery every hour and a DeepDiscovery once a day.

For the QuickQuery and NeedAttention actions, we easily find records that are logged in the
ksys log from the excerpt that is shown in Example 2-54.
Example 2-54 Steady state records in ksys log

# sed '1168,$!d;=' post-restart_ksys.log|sed 'N;s/\n/:/'|head
1169:[15] 11/19/18 T(8788) _VMR 13:49:48.702969 DEBUG VMR_CEC.C[9776]: usage_count
0
ReturnCode: 0
1171:[15] 11/19/18 T(9192) _VMR 13:50:10.644768 DEBUG VMR_HMC.C[6827]: getNeedAttn
ReturnCode: 0
ReturnCode: 0
ReturnCode: 0
ReturnCode: 0
#
The hourly QuickDiscovery and the daily DeepDiscovery actions happen as described in
“Quick Discovery and Deep Discovery scheduling records” on page 72. The DeepDiscovery
start moment of the day can be manually changed by running ksysmgr, as detailed in
“Discovery and verification” on page 92.

In conclusion, Figure 2-7 shows a synthetic result of our findings in the daemon restart
exercise.
Figure 2-7 KSYS daemon threads
2.2.6 KSYS subsystem functional modules

Figure 2-8 presents the KSYS subsystem from a purely functional perspective. Functions are
implemented inside the code of the IBM.VMR resource manager and the ksysmgr utility. The
libkrest module is a helper library that the KSYS daemon uses to talk to HMCs and VIOSs.
Figure 2-8 KSYS functional modules
We further detail the role of each grouping module in the following sections.

Configuration Management
The Configuration Management module handles requests of the create, delete, or change
types, which are needed to manipulate KSYS objects (IBM.VMR resources). Such requests
come either directly from the ksysmgr commands that are entered by the user, or indirectly
from other modules that are running their specific jobs. All initial elementary steps, such as
when you first define the KSYS cluster, add HMCs and hosts, and then create a host group,
are typical configuration steps. Any reverse elementary steps, such as when you delete host
groups, hosts, and HMCs, are also typical configuration actions.
Discovery and verification

After host group creation, you must run a discovery operation for the host group, as shown in
Example 2-55.
Example 2-55 Host group discovery invocation

ksysmgr discover host_group <hg_name> [verify=<yes|no>]
During this initial discovery of a host group, the KSYS subsystem deploys a special
monitoring setup for the involved VMs and hosts. It creates an SSP cluster that is based on
the input that is specified in the preliminary configuration steps. The real SSP cluster
(Example 2-7 on page 50) of a host group is represented in the KSYS configuration by an
associated IBM.VMR_SSP resource (Example 2-56).
Example 2-56 SSP resource of a host group

# lsrsrc IBM.VMR_SSP
Resource Persistent Attributes for IBM.VMR_SSP
resource 1:
Name = "SSP_1"
SspName = "KSYS_rbRMHA_1"
SspUuid = "1509d1bd-3fbe-3b4b-bd52-7cb482efecca"
SspState = "UP"
PoolName = "default_pool_1"
PoolUuid = "000000000A2819F1000000005BD4C329"
ViosCount = 4
ViosUuidList =
{"315CE56B-9BA0-46A1-B4BC-DA6108574E7E","5AC3E0DC-13FC-4221-9617-8EF61C4CDD83","4E
0F6F60-214B-4D2C-A436-0EAC46F7F71F","166C89EF-57D8-4E75-815C-AA5B885560B1"}
VmCount = 16
RepositoryDisk =
"01M0lCTTIxNDUxMjQ2MDA1MDc2ODAyODEwNDI2ODgwMDAwMDAwMDAwMDExNA=="
SspDiskList =
{"01M0lCTTIxNDUxMjQ2MDA1MDc2ODAyODEwNDI2ODgwMDAwMDAwMDAwMDExMw=="}
HgID = 1
SiteID = 0
ActiveSiteID = 0
SspVersion = "VIOS 3.1.0.00 "
CommonXsdVersion = "4.00"
Phase = "INIT"
PhaseDetail = 0
ErrMsg = " "
DrTestPhase = ""
DrTestPhaseDetail = 0
WarnMsg = ""
InfoMsg = ""
Phasebit = 0

PhaseDetail1 = 0
DrTestPhaseDetail1 = 0
ActivePeerDomain = "rbRMHA"
#
Subsequent discoveries can be started by user or by the KSYS scheduler. You normally run a
discovery command each time a change is made in your environment. By default, the KSYS
scheduler starts a so called Deep Discovery once in every 24 hours at midnight, as described
in “Quick Discovery and Deep Discovery scheduling records” on page 72. You can change
this setting, as shown in Example 2-57.
Example 2-57 Changing the auto_discovery_time system attribute

# ksysmgr q system|grep auto_discovery_time
auto_discovery_time: 00:00 hours
# ksysmgr mod system auto_discovery_time -h
ksysmgr modify system

[auto_discovery_time=<hh:mm>]
hh - hour: 00 to 23
mm - minute: 00 to 59
...
#
Deep Discovery means that discovery is followed by a complete verification step. Every hour,
the system performs a Quick Discovery, which is discovery only with no verification. The
KSYS subsystem scans the environment for any changes that occurred since the last
discovery and adapts to the modified environment. Collected changes are saved persistently
such that they survive any daemon or KSYS node crash or restart event. Any changes in the
configuration or issues that are found during a discovery are reported to the user and related
change or error records are written in log and trace files.
A discovery must be completed successfully before running the first verification. The
verification process ensures that the backup host can accommodate the VM. Verification
consists of an LPM validate operation that is performed for every VM on all the available
backup hosts on the host group. Any issues that are found during verification trigger a
notification to registered users, and then are logged in to log files.
Such verification can be started either for a host group, as shown in Example 2-58, or only for
a host or VM, as shown in Example 2-59.
Example 2-58 Host group level verification

# ksysmgr discover host_group <hg_name> verify=true
# ksysmgr verify host_group <hg_name>
In the case of a host or VM, we rather use LPM validation instead of verification.
Example 2-59 Host or VM level validation

# ksysmgr lpm host <hostname|uuid> [to=<hostname|uuid>] action=validate
# ksysmgr lpm vm <vm1[,vm2,...]> [to=<hostname|uuid>] action=validate

Failure Detection Engine
The FDE thread of a host group starts only after a user-initiated first discovery action
completes successfully for that host group. If the KSYS cluster has multiple host groups that
are defined, then the set of all running FDE threads can be seen as a functional FDE module.
For a single defined host group, the FDE module consists of its unique FDE thread.
The responsibility of the FDE module is to check periodically the health status of the
managed VMs and hosts and to initiate VM relocation actions when needed. The FDE
module does not perform the actual VM relocation or cleanup. It just triggers the relocation by
informing the Relocation Engine module.
The health status of the managed VMs is monitored by heartbeats. VM Agents periodically
send heartbeats to the HMs running on the underlying VIOSs. HMs save received VM
heartbeat details in the HSDB. Each HM also updates periodically its own status to the same
HSDB. VIOSs monitor each other’s health among themselves through the SSP and
underlying CAA cluster mechanisms.
The FDE thread of a host group retrieves a consolidated health status for all hosts and VMs in
that host group by submitting periodic requests to a designated VIOS that is selected among
all of the ones that are on the host group. These FDE requests reach the designated VIOS by
using specific REST API requests that are addressed to the HMC, which intermediates
between the KSYS and the VIOS. For more information about how HMC intermediates the
communication with the VIOS, see “The libkrest library and HMC pass-through APIs” on
page 98.
Example 2-60 shows an excerpt with a typical sequence of QuickQuery and Need Attention
records from an FDE trace file.
Example 2-60 Excerpt from an FDE trace file

# ksysmgr trace log=fde|grep -E "doQuickQ|doNeedAtt"|head
[00] 11/14/18 T(9598) _VMR 17:04:34.868328 DEBUG VMR_HG.C[11671]: FDE doQuickQuery
success to 315CE56B-9BA0-46A1-B4BC-DA6108574E7E
doNeedAttn GLOBAL_DATA to 315CE56B-9BA0-46A1-B4BC-DA6108574E7E
[00] 11/14/18 T(9598) _VMR 17:04:56.208343 DEBUG VMR_HG.C[11592]: FDE doNeedAttn
success GLOBAL_DATA to 315CE56B-9BA0-46A1-B4BC-DA6108574E7E
...
#
We see that the FDE thread sends two QuickQuery requests followed by a Need Attention
request at 20-second intervals on average. FDE uses two types of periodic requests:
򐂰 QuickQuery (QQ)
򐂰 NeedsAttention (NA)

QuickQuery is issued more frequently. The reply of a QuickQuery request contains a
consolidated list of XML elements and subelements with one element for each host on the
host group inside each host element with a subelement for each VIOS on the host. The VIOS
XML subelement contains attributes about the VIOS health status:
state State of the VIOS as a node in the CAA cluster underlying the SSP
with possible values of UP or DOWN.
hmResponsive Indication that the HM daemon on the VIOS is sending heartbeats
(value 1) or not (value 0).
hmResponseSec The time that is elapsed (in seconds) since the HM sent a heartbeat to
HSDB.
hasData Indication that HSDB has information about a CAA event.
reason Code corresponding to the current CAA event reason, if any.
Example 2-102 on page 139 shows that when an issue is present at the VIOS level, it is also
reported in the NeedsAttention replies and the QuickQuery replies.
Although it can provide VIOS details, a NeedsAttention request obtains VM health and state
change information. We describe the specific XML elements, subelements, and attributes of a
NeedsAttention reply later and mention here only two relevant VM attributes for the current
context:
state State of the VMM monitoring agent running on the VM as reported to
HMs and registered in the HSDB at the VIOS level.
missedHb Time in seconds since the moment when the heartbeat was not
received as expected by the VIOS HMs from the VM Agent (when the
heartbeat is received, this attribute is not reported in the NA reply).
The QQ and NA requests, their XML replies and contained XML elements, subelements, and
attributes, together with what is happening at the KSYS and VIOS levels since the moment
FDE initiates doQuickQuery or doNeedAttn actions until the moment they return as logged in
Example 2-60 on page 94, are detailed in 2.3.2, “Failure Detection Engine: QuickQuery and
NeedsAttention flows” on page 234.
If both VIOSs of a host are unhealthy, then the FDE decides that all managed VMs of the
affected host must be relocated to other hosts on the host group. If VIOSs of a host are
healthy but some VMs are not, then the FDE initiates relocation of only those unhealthy VMs.
FDE first checks for host relocation and then for VM relocation. Let us see in more detail how
these decisions are made.
Host relocation decision

FDE initiates the relocation of the entire host only if both VIOSs of the host are found to be in
an unhealthy state. The primary factor that initiates a VIOS state health check is its CAA
state, which is retrieved from the HSDB as the VIOS state attribute value in QQ or NA
replies. If the VIOS CAA state is DOWN, then FDE performs the following steps:
1. Pings the VIOS. If that task succeeds, the VIOS is treated as healthy.
2. Get the LPAR runtime state of the VIOS from the HMC by running lssyscfg -r lpar.
3. Declare the down VIOS as healthy only when the following situations are true:
– The VIOS is in the Not Activated or in Open Firmware LPAR state.
– The reason attribute value (retrieved from the HSDB together with CAA state) is
SHUTDOWN.

For a host with both VIOSs declared unhealthy, an extra check is done before the host
relocation decision. The FDE sends a sanity health request directly to the involved VIOSs. If
at least one VIOS in the host is now healthy, then the relocation is canceled. Otherwise, the
host relocation decision occurs. Only VMs that are declared as managed at the ksysmgr level
are relocated; the unmanaged ones are ignored.
VM relocation decision
The primary factor for initiating a VM relocation is the lack of heartbeat from the VM. Only
VMs that are registered as managed and have the ha_monitor attribute set to enable at the
ksysmgr level are considered for VM relocation. The sensitivity of the FDE reaction to a
missed heartbeat event can be adjusted at the ksysmgr level by using two parameters:
host_failure_detection_time and vm_failure_detection_speed. You can set both
parameters either at the global KSYS system or at a particular host group level.
Example 2-61 shows the parameters default values and ranges. The
vm_failure_detection_speed can be also adjusted at the VM level.
Example 2-61 Missed heartbeat sensitivity parameters

# ksysmgr -v q system|grep -i detection
host_failure_detection_time: 90 seconds
vm_failure_detection_speed: normal
# ksysmgr -v q hg rbHG|grep -i detection
# ksysmgr -v q vm vmraix1|grep -i detection
#
# ksysmgr modify system -?|grep -i detection
[host_failure_detection_time=<90-600>]
[vm_failure_detection_speed=<fast | normal | slow>]
# ksysmgr modify hg rbHG -?|grep -i detection
...
[host_failure_detection_time=<90-600>]
# ksysmgr modify vm vmraix1 -?|grep -i detection
#
From the host_failure_detection_time and vm_failure_detection_speed parameters

(hostFDT and VMFDS) and from a third one called VMthreshold, which is hardcoded to a fix
value of 50, the VM Failure Detection Time (VMFDT) parameter is derived for each monitored
VM as follows:
򐂰 If VMFDS is set to fast, then the parameter is VMFDT=hostFDT+VMthreshold.
򐂰 If VMFDS is set to normal, then the parameter is VMFDT=hostFDT+2*VMthreshold.
򐂰 If VMFDS is set to slow, then the parameter is VMFDT=hostFDT+3*VMthreshold.
So, the default value of the VMFDT is 190, and its minimum and maximum values are 140 and
750.
Also, FDE has an HBoffset parameter for each managed and monitored VM. HBoffset is
used to ensure that the VM is not relocated again immediately after a successful relocation
and that it is left enough time to be activated on the new host. HBoffset is set to 0 by default
and increases each time FDE finds the VM is the Not Activated state.

While processing the NA reply, the FDE thread looks for the missedHb attribute in each VM
subelement. If the missedHb attribute is not present in a VM, then the VM is declared as
healthy, which means that the heartbeat was recently received at the HM level and the HSDB
was updated. The HBmissed variable is set to 0 and no other action is performed for the
considered VM. But if a missedHb attribute value is present, then the HBmissed variable for that
VM is set to the missedHb attribute value to be used later.
The HBmissed variable that is obtained is compared with the VMFDT value that is derived for the
specific VM. If (HBmissed < VMFDT), then no action is performed. Otherwise, FDE performs
the following process:
1. Checks whether the VM is pingable. If the VM is pingable, then the missed heartbeat event
is ignored and FDE does not relocate the VM.
2. Retrieves the LPAR runtime state (from the lssyscfg -r lpar -F state output), current
reference code (by viewing the refcode attribute in the lsrefcode -r lpar command
output), and current profile details (from the lssyscfg -r prof output) of the VM from the
HMC. The HMC GUI help page of the Activate Partitions task for a partition lists the
following possible LPAR state values:
– Running: The LPAR is powered on and the OS instance on the LPAR is running.
– Not Activated: The LPAR is powered off.
– Open Firmware: The LPAR is in the System Management Services (SMS) menu.
– Shutting Down: The OS instance on the LPAR is shutting down.
– Error: The LPAR is in an error state.
– Not Available: The HMC cannot retrieve the state of the LPAR from the host.
– Migrating - Running: Live LPM is in progress.
– Migrating - Not Activated: The LPAR is shut down and being migrated to another
host.
– Hardware Discovery: The HMC is performing hardware discovery.
3. Relocate the VM if the LPAR state is in the Running state ((HBmissed - HBoffset) > VMFDT)
and its current reference code indicates that the system is started. Here are the reference
codes:
– 0 or ‘’ (empty): AIX is fully started.
– Linux ppc64: Big Endian 64-bit Red Hat is started.
– Linux ppc64le: Little Endian 64-bit Red Hat is started.
– SUSE Linux: SUSE Linux is started.
4. If the LPAR state is Not Activated, then complete the following steps:
a. Update the offset so that it goes to Running state (HBoffset = HBmissed) in the future.
b. If HBmissed > 2*VMFDT, check the reference code history of the LPAR for the word3
attribute of the most recent entry with the refcode attribute having a value of D200A200.
If this word3 value is NULL, then relocate the VM. Otherwise, it contains the reason for
the shutdown (D200A200) and no action must be performed.
5. If the LPAR is in the Open Firmware state where Hbmissed > VMFDT and the bootMode in the
LPAR profile is not sms (SMS), then relocate the VM.

6. Generate a VM_LPM_HANG_DETECTED event if the VM is in the Migrating - Running state
and HBmissed is greater than 86400 (24 hours in seconds).
7. Generate a VM_NONRESP_DETECTED event if the VM state is Not Available, or Error or
Shutting Down or Hardware Discovery or Migrating - Not Activated) and HBmissed >
2*FDT.
The libkrest library and HMC pass-through APIs

The libkrest library is a wrapper library for the HMC REST APIs that intermediates the
KSYS daemon access to the REST resources that are exposed by the HMC REST server.
Any communication between the KSYS daemon and the HMCs is logged by libkrest in
dedicated krest and krestlong trace files. To understand the records in these trace files, we
need some familiarity with the HMC REST API framework and its basic concepts.
HMC REST API concepts

Client applications use the GET, PUT, POST, and DELETE HTTP methods to act on web resources
that are maintained by the HMC REST server. URL patterns, HTTP request or response
headers and payload formats, response status codes, and others are described in the HMC
REST API documentation. The request and response payloads are formatted as XML
structures that follow the Atom Publishing Protocol specifications.
Operations that are provided by the HMC REST API can be synchronous or asynchronous.
Some of the requests coming to the HMC REST server need a significant amount of elapsed
time to complete. To optimize the usage of HMC resources and give the client application the
opportunity to perform other processing, these long-running operations can run
asynchronously by using HMC REST API job resources, which are also called jobs.
In the synchronous case, the HMC REST server does not respond to the client request until
all the processing that is associated with the request completes. When this processing
completes either successfully or in error, the server provides the final result to the client side.
The client remains blocked during this time.
In the asynchronous case, the HMC REST server performs some minimal validation and
quickly returns an HTTP 200 (OK) result to the client, which indicates that the request was
accepted. Along with the HTTP 200 (OK) result, the REST server replies with an identifier that
represents the asynchronous job resource that is created to serve the request. The response
payload is formatted in a particular Atom Publishing Protocol XML structure. The
Content-Type header of the response is set to application/atom+xml, and the client is
provided with details about the newly created job resource, including the job URI, inside the
XML payload of the response.
After receiving the HTTP 200 (OK) response status code and the job details in the response
payload the client can now launch a Job Status GET request to determine whether the job
completed. When the is not completed and is still running, the returned job status code is set
to RUNNING. If the job completed successfully, the returned job status code is set to
COMPLETED_OK. Errors and other special cases are handled with extra status codes. These job
status codes and related details are provided inside the XML payload of the Job Status GET
method response.

An extensive set of use cases and URL patterns is presented in the HMC REST API
reference documentation in a so-called URL model. We compared the URL patterns in the
model with URL occurrences in more krestlong log excerpts, including the one taken after
the KSYS daemon restart. All URL patterns in the model and URL occurrences in the logs
have a common beginning path stub, /rest/api/. We obtained matches with only four URL
patterns in the model:
򐂰 /rest/api/web/Logon
򐂰 /rest/api/uom/{R}/{UUID}/do/{OP}
򐂰 /rest/api/uom/{R}/{UUID}
򐂰 /rest/api/uom/jobs/{JOBID}
The {R} identifier is a REST resource of a so-called root element type. The {UUID} identifier is
a UUID of a resource instance. The {OP} identifier is the name of a job operation. The {JOBID}
is the identifier of a submitted job. We selected the URL model details that are relevant to
these four matched URL patterns. They are cited exactly as they appear in the HMC REST
API reference documentation.
URL model
򐂰 Concepts:
– Anchor URL patterns provide services for a type of root/child element.
– Instance URL patterns provide services for a uniquely identified root/child element.
򐂰 URL Pattern Grammar building blocks:
{R} Root element type.
{UUID} A unique UUID value.
{OP} The name of a type of job (an operation).
{JOBID} The ID of a submitted job.
URL model use cases for root elements

򐂰 ROOT ANCHOR URL patterns:
/rest/api/uom/{R}/operations Get the defined job operations for {R}.
/rest/api/uom/{R}/jobs Get all known jobs for {R}.
/rest/api/uom/{R}get The feed of all known instances of {R}.
򐂰 ROOT INSTANCE URL patterns:
/rest/api/uom/{R}/{UUID}/do/{OP} Get a template for a job of type {OP}.
/rest/api/uom/{R}/{UUID}/jobs Get all known jobs for this element of {R}.
/rest/api/uom/{R}/{UUID}/jobs/{JOBID} Get the details for the one job.
/rest/api/uom/{R}/{UUID} Get the XML details of this instance of {R}.
Resource /rest/api/web/Logon
The API receives a user ID and password as a logon request and responds with
X-API-Session. This transaction establishes a valid user session.
Resource to obtain a job status by using a job ID

To obtain a job status by using a job ID, use the /rest/api/uom/jobs/{job_id} resource.
When a job is started, a status is returned. The status provides information about the result of
the job and other related details.

The current HMC REST API reference documentation is available in IBM Knowledge Center
as part of the product documentation for various Power Systems servers.
Example 2-62 shows the matches of URL occurrences in our krestlong log excerpt with the
URL patterns in the model.
Example 2-62 Matching model URL patterns with URL occurrences in the krestlong log excerpt
$ cat post-restart_krestlong.log|wc -l
38915
$ grep "/rest/api/" post-restart_krestlong.log|head -5
[10] 11/19/18 T(9192) _VMR 13:49:26.265679 DEBUG libkrest_query.c[1536]: PUT at
'/rest/api/web/Logon', with header 'Content-Type:
application/vnd.ibm.powervm.web+xml; type=LogonRequest', payload size 173 bytes
[10] 11/19/18 T(9192) _VMR 13:49:26.265681 DEBUG libkrest_query.c[1540]: URL:
[https://9.3.18.159:12443/rest/api/web/Logon]
'/rest/api/uom/VirtualIOServer/315CE56B-9BA0-46A1-B4BC-DA6108574E7E/do/PassThrough
Interface', with header 'Content-Type: application/vnd.ibm.powervm.web+xml;
type=JobRequest', payload size 1373 bytes
[https://9.3.18.159:12443/rest/api/uom/VirtualIOServer/315CE56B-9BA0-46A1-B4BC-DA6
108574E7E/do/PassThroughInterface]
<link rel="SELF"
href="https://9.3.18.159:12443/rest/api/uom/jobs/1540961034191"/>
$ grep "/rest/api/" post-restart_krestlong.log|tail -5
<link rel="MANAGEMENT_CONSOLE"
href="https://9.3.18.159:12443/rest/api/uom/ManagementConsole/ec3e2415-2e77-3a0d-b
107-20f0df07ec66"/>
[04] 11/19/18 T(9192) _VMR 13:57:28.863213 DEBUG libkrest_query.c[1536]: GET at
'/rest/api/uom/jobs/1540961034289', with header 'none', payload size 0 bytes
[https://9.3.18.159:12443/rest/api/uom/jobs/1540961034289]
<link rel="SELF"
href="https://9.3.18.159:12443/rest/api/uom/jobs/1540961034289/9095c232-1d0e-49ad-
bc1c-6817a9edc62a"/>
href="https://9.3.18.159:12443/rest/api/uom/ManagementConsole/ec3e2415-2e77-3a0d-b
107-20f0df07ec66"/>
$ grep -c "/rest/api/" post-restart_krestlong.log
480
$ grep -c "/rest/api/web/Logon" post-restart_krestlong.log
10
$ grep "/rest/api/uom/VirtualIOServer/" post-restart_krestlong.log|grep -c
"/do/PassThroughInterface"
108
$ grep -c "/rest/api/uom/jobs/" post-restart_krestlong.log
285
$ grep -c "/rest/api/uom/ManagementConsole/" post-restart_krestlong.log
77
$
We identified only two {R} root element types in our log: VirtualIOServer and
ManagementConsole.

The ManagementConsole root element type appears in occurrences of the
/rest/api/uom/ManagementConsole/{UUID} pattern, with two distinct UUID values, one for
each HMC.
The URL occurrences for the VirtualIOServer root element type match each time the
/rest/api/uom/{R}/{UUID}/do/{OP} pattern, with {OP} taking the PassThroughInterface
string value and {UUID} taking the UUID values of our managed VIOSs. Plenty of job status
requests that match the /rest/api/uom/jobs/{job_id} pattern are also present, so we have
good reasons to consider that the predominant requests are PassThroughInterface job
operations for instances of the VirtualIOServer root element type.
HMC pass-through APIs

The URL model shows that the /rest/api/uom/{R}/operations pattern is available to obtain
the defined job operations for the {R} root element type. To get more insights about the
PassThroughInterface operation of the VirtualIOServer root element type, we tried this
pattern with {R} set to VirtualIOServer (Example 2-64 on page 102). The obtained result is
presented in Example 2-63.
Example 2-63 Defined job operations for the VirtualIOServer root element
<entry xmlns="http://www.w3.org/2005/Atom"
xmlns:ns2="http://a9.com/-/spec/opensearch/1.1/"
xmlns:ns3="http://www.w3.org/1999/xhtml">
<id>b704f5cd-20b0-34c6-b33f-3c082aa34d51</id>
<title>OperationSet</title>
<published>2018-12-09T11:48:24.157-06:00</published>
<link rel="SELF"
href="https://rthmc3:12443/rest/api/uom/VirtualIOServer/operations/b704f5cd-20b0-3
4c6-b33f-3c082aa34d51"/>
href="https://rthmc3:12443/rest/api/uom/ManagementConsole/ec3e2415-2e77-3a0d-b107-
20f0df07ec66"/>
<author>...</author>
<content type="application/vnd.ibm.powervm.web+xml; type=OperationSet">
<OperationSet:OperationSet
xmlns:OperationSet="http://www.ibm.com/xmlns/systems/power/firmware/web/mc/2012_10
/" xmlns="http://www.ibm.com/xmlns/systems/power/firmware/web/mc/2012_10/"
xmlns:ns2="http://www.w3.org/XML/1998/namespace/k2" schemaVersion="V1_8_0">
<Metadata>...</Metadata>
<SetName kb="ROR" kxe="false">VirtualIOServer</SetName>
<DefinedOperations kxe="false" kb="ROO" schemaVersion="V1_8_0">
<Operation schemaVersion="V1_8_0">...</Operation>
...
<Operation schemaVersion="V1_8_0">
<OperationName kb="ROR" kxe="false">PassThroughInterface</OperationName>
<GroupName kb="ROR" kxe="false">VirtualIOServer</GroupName>
<ProgressType kb="ROR" kxe="false">DISCRETE</ProgressType>
<AllPossibleParameters kb="ROO" kxe="false" schemaVersion="V1_8_0">
<OperationParameter schemaVersion="V1_8_0">
<ParameterName kxe="false" kb="ROR">inputXML</ParameterName>
</OperationParameter>

</AllPossibleParameters>
<AllPossibleResults kb="ROO" kxe="false" schemaVersion="V1_8_0">
<ParameterName kxe="false" kb="ROR">StdError</ParameterName>
<ParameterName kxe="false" kb="ROR">StdOut</ParameterName>
<ParameterName kxe="false" kb="ROR">RMCReturnCode</ParameterName>
<ParameterName kxe="false" kb="ROR">CMDReturnCode</ParameterName>
</AllPossibleResults>
<AllDiscreteStates kb="ROO" kxe="false"
schemaVersion="V1_8_0">...</AllDiscreteStates>
</Operation>
</DefinedOperations>
</OperationSet:OperationSet>
</content>
</entry>
The result is an Atom entry XML element response payload and contains the details for all
defined operations. In Example 2-63 on page 101, we manually formatted the element
content for easier reading. We kept only the details about the input and result parameters for
the PassThroughInterface operation. The request itself,
/rest/api/uom/VirtualIOServer/operations, is shown in Example 2-64, together with the
required user session housekeeping.
Example 2-64 Obtaining the defined job operations for the VirtualIOServer root element
$ cat login.xml
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<LogonRequest
xmlns="http://www.ibm.com/xmlns/systems/power/firmware/web/mc/2012_10/"
schemaVersion="V1_1_0">
<Metadata> <Atom/> </Metadata>
<UserID kb="CUR" kxe="false">hscroot</UserID>
<Password kb="CUR" kxe="false">abc123</Password>
</LogonRequest>
$ curl -k -X PUT -H "Content-Type: application/vnd.ibm.powervm.web+xml;
type=LogonRequest" https://rthmc3:12443/rest/api/web/Logon -d @login.xml
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<LogonResponse
<X-API-Session kxe="false" kb="ROR">JF_rY...WDQI=</X-API-Session>
</LogonResponse>

$ curl -sk -H "X-API-Session: JF_rY...WDQI="
https://rthmc3:12443/rest/api/uom/VirtualIOServer/operations >vios.operations.xml
$ curl -k -X DELETE -H "X-API-Session: JF_rY...WDQI="
https://rthmc3:12443/rest/api/web/Logon
$
Running curl on a Linux client, we first established a user session with the HMC REST
server and obtained the session token. Then, we performed our request on the
VirtualIOServer root resource and then deleted the session. The session token is shortened
for easier reading. All requests that are used here can be considered as good examples of
synchronous operations.
Content that is exchanged by the PassThroughInterface operation between KSYS and VIOS,
like inputXML, stdOut, and the rest of parameters in Example 2-63 on page 101, might be in
XML format, so it must be encoded and encapsulated in the HTTP XML payload that is
exchanged between the KSYS and the HMC. The HMC REST server uses the RMC daemon
running on the HMC and the RMC connection that is maintained by the RMC daemon running
on the VIOS.
Figure 2-9shows this encapsulation mechanism of the XML data that is exchanged by KSYS
with the VIOS itself.
Figure 2-9 XML data encapsulation in the PassThroughInterface operation
The structure and meaning of the XML element messages that are exchanged by KSYS with
a VIOS are shown in the new vioHADR2.00 XML Schema Definition (XSD) that was
introduced with VIOS V3.1, as shown in Example 2-65.
Example 2-65 The vioHADR2.00 XML Schema Definition file

# /usr/ios/cli/ioscli ioslevel
3.1.0.00
# ls -l /usr/lib/vioHADR2.00.xsd
-r-------- 1 root system 44919 Sep 21 19:50 /usr/lib/vioHADR2.00.xsd
# head -10 /usr/lib/vioHADR2.00.xsd
<?xml version="1.0" encoding="UTF-8" standalone="no"?>


<xsd:schema
xmlns:xsd="http://www.w3.org/2001/XMLSchema"
targetNamespace="http://ausgsa.austin.ibm.com/projects/v/vios/schema/vioHADR2.00"
#

The RMC setup allows the invocation of the vioservice command on the VIOS side with
appropriate arguments and input that is passed from the HMC side. The vioservice
command syntax and usage is shown in Example 2-66.
Example 2-66 The vioservice command syntax

# /usr/ios/sbin/vioservice
vioservice [-E errorinfofile] [-O streamoutputfile] <resource> [options]
Known resources:
capabilities
cmd
ioscli
padmin
viosmgr
debug
debug_level (a number that indicates the debug level)
resource (a separate resource)
lib
libvio
cluster
lu
partition
query
sp
version
libviopass
passthru
pcm
resources
version
Options:
lib/*/* [-n ElementsPerChunk] [-t tracelevel] [-v version]
Example Commands:
vioservice cmd/ioscli/cluster -status
vioservice lib/libvio/lu -n 10
vioservice debug/8/lib/libvio/cluster
Note:
Any one of these fields can be terminated with "/resources" to display
available subcommands.
Examples:
lib/libvio/resources
cmd/resources
#

The command logs in viosvc.log and viosvc.log.err in the /home/ios/logs directory.
Example 2-67 shows a vioservice log excerpt with an entry that is determined by a
QuickQuery request (VIO_HS_QUICK_QUERY) that came from the KSYS side.
Example 2-67 QuickQuery request in viosvc.log

# alog -f /home/ios/logs/viosvc.log -o | more
...
[START 27066876 65733113 12/09/18-19:05:24.752 vioservice.c 1.18 320]
/usr/ios/sbin/vioservice lib/libviopass/passthru
[0 27066876 65733113 12/09/18-19:05:24.753 viosvc_res.c 1.26 456] stdin pipe input:[<?xml
version="1.0"?>
<VIO xmlns="http://ausgsa.austin.ibm.com/projects/v/vios/schema/vioHADR2.00" version="2.00"
author="LIBKREST" title="Req Quick Query">
<Request action_str="VIO_HS_QUICK_QUERY"/>
</VIO>
]
[0 27066876 65733113 12/09/18-19:05:24.806 viosvc_res.c 1.26 464]
vio_response.result=[<VIO><Response>
<quickQuery>
<viosStatList><viosStatus machine_type="8286" model="42A" serial="21E0B2V">
<viosStat uuid="315ce56b-9ba0-46a1-b4bc-da6108574e7e" state="UP" hmResponsive="1"
hmResponseSec='1' hasData='0' reason='0x00000000' syncMsg="0" />
<viosStat uuid="5ac3e0dc-13fc-4221-9617-8ef61c4cdd83" state="UP" hmResponsive="1"
</viosStatus></viosStatList>
</quickQuery>
<quickQuery>
<viosStatList><viosStatus machine_type="8286" model="42A" serial="2100DEW">
<viosStat uuid="166c89ef-57d8-4e75-815c-aa5b885560b1" state="UP" hmResponsive="1"
<viosStat uuid="4e0f6f60-214b-4d2c-a436-0eac46f7f71f" state="UP" hmResponsive="1"
</quickQuery>
</Response></VIO>
]
[END 27066876 65733113 12/09/18-19:05:24.806 vioservice.c 1.18 323] exited with rc=0
viosvc.log (99%)
We first see the inputXML content as extracted by the HMC REST server from the HTTP
request payload and passed through RMC as input to the vioservice command. Then, the
vioservice command runs, generates its result on standard output, and exits with a return
code. Standard output content and the return code value are passed back through RMC to
the HMC REST server to be encoded and sent back to libkrest and KSYS as parameter
values for the stdOut and CMDReturnCode parameters in the response XML payload.
QuickQuery asynchronous job example

Now, we look at our KSYS daemon and at the first 10 records that are logged in the krest
trace files after the KSYS daemon restart (Example 2-68).
Example 2-68 The krest log immediately after the KSYS daemon restart
# sed '1,10!d;=' post-restart_krest.log|sed 'N;s/\n/:/'
1:[00] 11/19/18 T( 1) ____ 13:48:35.136313 ******************* Trace Started -
Pid = 16646518 **********************

2:[00] 11/19/18 T(9192) _VMR 13:49:26.265561 DEBUG libkrest.c[1179]:
krigetSessionId:hmc 9.3.18.159...
krigetSessionId returning 0.
kriSubmitQuickQuery:hmc->(9.3.18.159),vios->(315CE56B-9BA0-46A1-B4BC-DA6108574E7E)
kriSubmitQuickQuery returning 0 with jobid->(1540961034191).
krigetJobResult:hmc->(9.3.18.159),jobid->(1540961034191)
krigetJobResult returning 0 with josbstatus RUNNING.
krigetJobResult:hmc->(9.3.18.159),jobid->(1540961034191)
krigetJobResult returning 0 with josbstatus COMPLETED_OK.
kriSubmit_Msg:hmc->(9.3.18.159),
#
Now that we introduced the HMC REST API and the mechanics of an asynchronous job, we
see in Example 2-68 on page 105 that the FDE thread, T(9192), opens a session and a
session token is returned. Then, the thread submits a QuickQuery job request to the HMC at
IP address 9.3.18.159 for the VIOS with UUID 315CE56B-9BA0-46A1-B4BC-DA6108574E7E. An
entire job progress pattern follows, and the job with the jobid 1540961034191 is created. The
first job status check shows the job in the RUNNING state, so it is still running. Finally, the
second check shows that the job is complete with the status COMPLETED_OK.
We now check the progress of the same job request in the corresponding krestlong trace file.
Example 2-69 shows an excerpt from the beginning of the krestlong log file, covering the
initial logon request and the immediate request that follows, which is the first QuickQuery
request submission after our KSYS daemon restart.
Example 2-69 The krestlong log immediately after the KSYS daemon restart
= 16646518 **********************
[10] 11/19/18 T(304) _VMR 13:48:35.172701 [ INFO,VMRDaemon,] Start KREST category
logging
[10] 11/19/18 T(304) _VMR 13:49:26.232328 DEBUG libkrest.c[4444]: libkrest global
initialization...
...
[10] 11/19/18 T(9192) _VMR 13:49:26.265637 DEBUG libkrest.c[521]: libkrest context
initialization successful
'/rest/api/web/Logon', with header 'Content-Type:
application/vnd.ibm.powervm.web+xml; type=LogonRequest', payload size 173 bytes
[https://9.3.18.159:12443/rest/api/web/Logon]
...
[10] 11/19/18 T(9192) _VMR 13:49:26.517114 DEBUG libkrest.c[9462]: Using 2.00 as
vio hadr version [ Default ]
[10] 11/19/18 T(9192) _VMR 13:49:26.517176 DEBUG <?xml version="1.0"?>

<VIO xmlns="http://ausgsa.austin.ibm.com/projects/v/vios/schema/vioHADR2.00"
version="2.00" author="LIBKREST" title="Req Quick Query">
</VIO>
...
[10] 11/19/18 T(9192) _VMR 13:49:26.517216 DEBUG <JobRequest:JobRequest
xmlns:JobRequest="http://www.ibm.com/xmlns/systems/power/firmware/web/mc/2012_10/"
<RequestedOperation kb="CUR" kxe="false" schemaVersion="V1_3_0">
<GroupName kxe="false" kb="ROR">VirtualIOServer</GroupName>
</RequestedOperation>
<JobParameters kb="CUR" kxe="false" schemaVersion="V1_3_0">
<JobParameter schemaVersion="V1_3_0">
<ParameterValue kxe="false" kb="CUR"><![CDATA[<?xml version="1.0"?> <VIO
xmlns="http://ausgsa.austin.ibm.com/projects/v/vios/schema/vioHADR2.00"
version="2.00" author="LIBKREST" title="Req Quick Query"> <Request
action_str="VIO_HS_QUICK_QUERY"/> </VIO>]]>
</ParameterValue>
</JobParameter>
</JobParameters>
</JobRequest:JobRequest>
'/rest/api/uom/VirtualIOServer/315CE56B-9BA0-46A1-B4BC-DA6108574E7E/do/PassThrough
Interface', with header 'Content-Type: application/vnd.ibm.powervm.web+xml;
type=JobRequest', payload size 1373 bytes
[https://9.3.18.159:12443/rest/api/uom/VirtualIOServer/315CE56B-9BA0-46A1-B4BC-DA6
108574E7E/do/PassThroughInterface]
...
Entries are logged by the FDE thread T(9192). XML input for the QuickQuery request that is
addressed to the VIOS is logged as generated, and then the XML payload for the HTTP PUT
request is addressed to the HMC. Inside the XML payload of the PUT request, we recognize
the CDATA section with the specific QuickQuery XML content for the ParameterValue element
that is associated with the inputXML ParameterName element, as shown in the definition of the
PassThroughInterface job operation that is detailed in Example 2-63 on page 101. This
content is supposed to be passed by the HMC REST server to the vioservice command that
is run by RMC on the VIOS that is identified with the UUID that is specified in the
/rest/api/uom/VirtualIOServer/315CE56B-9BA0-46A1-B4BC-DA6108574E7E/do/PassThroughI
nterface resource identifier.
The response for the PUT request, both header and payload, is shown in Example 2-70.
Example 2-70 QuickQuery PUT response

[10] 11/19/18 T(9192) _VMR 13:49:26.657014 DEBUG HTTP/1.1 200 OK
Date: Mon, 19 Nov 2018 18:49:27 GMT
Server: Apache
...
X-Powered-By: Servlet/3.0

X-Transaction-ID: XT10620389
Content-Type: application/atom+xml
Content-Language: en-US
Content-Length: 2830
...
<entry xmlns="http://www.w3.org/2005/Atom"
xmlns:ns2="http://a9.com/-/spec/opensearch/1.1/"
xmlns:ns3="http://www.w3.org/1999/xhtml">
<id>8bda9318-74d8-48a7-8ee4-054a2245a8d6</id>
<title>JobResponse</title>
<published>2018-11-19T18:49:27.879Z</published>
<link rel="SELF"
href="https://9.3.18.159:12443/rest/api/uom/jobs/1540961034191"/>
<author>
<name>IBM Power Systems Management Console</name>
</author>
<content type="application/vnd.ibm.powervm.web+xml; type=JobResponse">
<JobResponse:JobResponse
xmlns:JobResponse="http://www.ibm.com/xmlns/systems/power/firmware/web/mc/2012_10/
" xmlns="http://www.ibm.com/xmlns/systems/power/firmware/web/mc/2012_10/"
<RequestURL kb="ROR" kxe="false"
href="VirtualIOServer/315CE56B-9BA0-46A1-B4BC-DA6108574E7E/do/PassThroughInterface
" rel="via" title="The URL to which the JobRequest was submitted."/>
<TargetUuid kb="ROR" kxe="false">
315CE56B-9BA0-46A1-B4BC-DA6108574E7E
</TargetUuid>
<JobID kxe="false" kb="ROR">1540961034191</JobID>
<TimeStarted kb="ROR" kxe="false">0</TimeStarted>
<TimeCompleted kxe="false" kb="ROR">0</TimeCompleted>
<Status kb="ROR" kxe="false">NOT_STARTED</Status>
<JobRequestInstance kb="ROR" kxe="false" schemaVersion="V1_8_0">
<RequestedOperation kxe="false" kb="CUR" schemaVersion="V1_8_0">
<JobParameters kxe="false" kb="CUR" schemaVersion="V1_3_0">
<ParameterValue kxe="false" kb="CUR"><?xml version="1.0"?> <VIO
version="2.00" author="LIBKREST" title="Req Quick Query"> <Request
action_str="VIO_HS_QUICK_QUERY"/> </VIO>
</ParameterValue>
</JobParameter>
</JobParameters>
</JobRequestInstance>
<Progress kb="ROO" kxe="false" schemaVersion="V1_8_0">

</Progress>
<Results kb="ROR" kxe="false" schemaVersion="V1_8_0">
</Results>
</JobResponse:JobResponse>
</content>
</entry>
[10] 11/19/18 T(9192) _VMR 13:49:26.657421 DEBUG libkrest_xml.c[2100]: Start of
the Function:text
...
[10] 11/19/18 T(9192) _VMR 13:49:26.657430 DEBUG libkrest_xml.c[2115]: In process
node JobID:1540961034191
[10] 11/19/18 T(9192) _VMR 13:49:26.657433 DEBUG libkrest_xml.c[2115]: In process
node Status:NOT_STARTED
...
The relevant items that we emphasized are the Content-Type: application/atom+xml

header, and from the payload, the job URI; the jobid 1540961034191; and the job status
NOT_STARTED.
As already suggested by the krest log excerpt in Example 2-68 on page 105, we expect to
see in the krestlong file the detailed log entries about the way the KSYS daemon is polling
for the job status through Job Status GET requests. We obtain a first GET at
'/rest/api/uom/jobs/1540961034191' entry, as shown in Example 2-71.
Example 2-71 First job status GET request for jobid 1540961034191
...
The response for this first job status GET request is shown in Example 2-72.
Example 2-72 The response of the first job status GET request
Date: Mon, 19 Nov 2018 18:49:27 GMT
...
<entry xmlns="http://www.w3.org/2005/Atom" ... >

<id>91c71222-36a6-314c-8baa-dcd35e364b7a</id>
...
<JobResponse:JobResponse ... >
<TargetUuid kb="ROR" kxe="false">
315CE56B-9BA0-46A1-B4BC-DA6108574E7E

</TargetUuid>
<Status kb="ROR" kxe="false">RUNNING</Status>
...
</JobParameters>
<DiscreteProgress kb="ROR" kxe="false" schemaVersion="V1_8_0">
<CurrentState kb="ROR" kxe="false" schemaVersion="V1_8_0">
<UntranslatedMessage kb="ROR" kxe="false">
VIOS PassThroughInterface job is started
</UntranslatedMessage>
<MessageKey kxe="false" kb="ROR">PERFORMPASSTHROUGHAPI_STARTED</MessageKey>
</CurrentState>
<CompletedStates kb="ROR" kxe="false" schemaVersion="V1_8_0">
<NLSStaticMessage schemaVersion="V1_8_0">
<UntranslatedMessage kb="ROR" kxe="false">
VIOS PassThroughInterface job is not yet started
</UntranslatedMessage>
<MessageKey kxe="false" kb="ROR">NOT_STARTED</MessageKey>
</NLSStaticMessage>
</CompletedStates>
</DiscreteProgress>
</Progress>
</Results>
</content>
</entry>
[10] 11/19/18 T(9192) _VMR 13:49:27.776259 DEBUG libkrest.c[481]: Resetting
retinfo fields...
This time, we again emphasized the job status, which is now RUNNING. Then, we removed the
job parameter content, which is the same as in the response of the previous PUT request, and
emphasized the progress details for the PassThroughInterface operation itself. Its status is
PERFORMPASSTHROUGHAPI_STARTED.

The second job status GET request follows in the krestlong log 1 second later after the
response entry of the first request with an identical GET at the
'/rest/api/uom/jobs/1540961034191' entry (Example 2-73).
Example 2-73 Second job status GET request for jobid 1540961034191
The response for this second job status GET request is shown in Example 2-74.
Example 2-74 The response of the second job status GET request
Date: Mon, 19 Nov 2018 18:49:29 GMT
...
<entry xmlns="http://www.w3.org/2005/Atom" ...>

<id>91c71222-36a6-314c-8baa-dcd35e364b7a</id>
...
<JobResponse:JobResponse ...>
<TargetUuid kb="ROR"
kxe="false">315CE56B-9BA0-46A1-B4BC-DA6108574E7E</TargetUuid>
<Status kb="ROR" kxe="false">COMPLETED_OK</Status>
...
</JobParameters>
<DiscreteProgress kb="ROR" kxe="false" schemaVersion="V1_8_0">
<CompletedStates kb="ROR" kxe="false" schemaVersion="V1_8_0">

<UntranslatedMessage kb="ROR" kxe="false">VIOS PassThroughInterface job is
not yet started</UntranslatedMessage>
<MessageKey kxe="false" kb="ROR">NOT_STARTED</MessageKey>
</NLSStaticMessage>
started</UntranslatedMessage>
<MessageKey kxe="false" kb="ROR">PERFORMPASSTHROUGHAPI_STARTED</MessageKey>
</NLSStaticMessage>
in progress</UntranslatedMessage>
<MessageKey kxe="false"
kb="ROR">PERFORMPASSTHROUGHAPI_INPROGRESS</MessageKey>
</NLSStaticMessage>
completed</UntranslatedMessage>
<MessageKey kxe="false"
kb="ROR">PERFORMPASSTHROUGHAPI_COMPLETED</MessageKey>
</NLSStaticMessage>
</CompletedStates>
</DiscreteProgress>
</Progress>
<ParameterName kxe="false" kb="ROR">StdOut</ParameterName>
<ParameterValue kxe="false" kb="CUR"><VIO><Response>
<quickQuery>
<viosStatList><viosStatus machine_type="8286" model="42A"
serial="21E0B2V">
<viosStat uuid="5ac3e0dc-13fc-4221-9617-8ef61c4cdd83" state="UP"
hmResponsive="1" hmResponseSec='1' hasData='0' reason='0x00000000' syncMsg="0"
/>
<viosStat uuid="315ce56b-9ba0-46a1-b4bc-da6108574e7e" state="UP"
/>
</quickQuery>
<quickQuery>
<viosStatList><viosStatus machine_type="8286" model="42A"
serial="2100DEW">
<viosStat uuid="166c89ef-57d8-4e75-815c-aa5b885560b1" state="UP"
/>
<viosStat uuid="4e0f6f60-214b-4d2c-a436-0eac46f7f71f" state="UP"
/>
</quickQuery>

</Response></VIO>
</ParameterValue>
</JobParameter>
<ParameterName kxe="false" kb="ROR">RMCReturnCode</ParameterName>
<ParameterValue kxe="false" kb="CUR">0</ParameterValue>
</JobParameter>
<ParameterName kxe="false" kb="ROR">CMDReturnCode</ParameterName>
<ParameterValue kxe="false" kb="CUR">0</ParameterValue>
</JobParameter>
</Results>
</content>
</entry>
...
The response in Example 2-74 on page 111 is long because it contains all the progress
history of the PassThroughInterface operation and the final job results. We again
emphasized the job status, which is now COMPLETED_OK, so this is the last job status GET
request. The remaining of the payload must then contain the expected result for the job.
Examining the remaining payload, we see the expected result. The job input parameter
content was again removed for easier reading and the progress history of the
PassThroughInterface operation itself was again emphasized. This time, there is no current
state and the progress history contains all traversed states, with the final one being
PERFORMPASSTHROUGHAPI_COMPLETED.
The expected job result is in the last emphasized section, which is the Results XML element.
Inside it is the content for the StdOut parameter as returned by the vioservice command,
then the content of the CMDReturnCode parameter, which is the returned value of the
vioservice command execution, and finally the content of the RMCReturnCode parameter,
which is the returned value of the RMC command execution in corresponding ParameterValue
XML elements.
The last krestlong entries about this job operation, which immediately follow the response
payload with the job results, are listed in Example 2-75.
Example 2-75 Job results as extracted from the last job status response payload
[10] 11/19/18 T(9192) _VMR 13:49:27.864945 DEBUG libkrest.c[2473]: <StdOut>
[10] 11/19/18 T(9192) _VMR 13:49:27.864954 DEBUG libkrest.c[2555]:
job_result->result : <VIO><Response>
<quickQuery>
<viosStat uuid="5ac3e0dc-13fc-4221-9617-8ef61c4cdd83" state="UP"
hmResponsive="1" hmResponseSec='1' hasData='0' reason='0x00000000' syncMsg="0" />
<viosStat uuid="315ce56b-9ba0-46a1-b4bc-da6108574e7e" state="UP"
</quickQuery>
<quickQuery>
<viosStat uuid="166c89ef-57d8-4e75-815c-aa5b885560b1" state="UP"

<viosStat uuid="4e0f6f60-214b-4d2c-a436-0eac46f7f71f" state="UP"
</quickQuery>
</Response></VIO>
[10] 11/19/18 T(9192) _VMR 13:49:27.864956 DEBUG libkrest.c[2473]: <RMCReturnCode>
[10] 11/19/18 T(9192) _VMR 13:49:27.864957 DEBUG libkrest.c[2473]: <CMDReturnCode>
[10] 11/19/18 T(9192) _VMR 13:49:27.864959 DEBUG libkrest.c[2494]: job_result->rc
: 0
[10] 11/19/18 T(4041) _VMR 13:49:36.241274 DEBUG libkrest.c[481]: Resetting
retinfo fields...
This is the final job result that is obtained for the first QuickQuery job request that is submitted
by the FDE thread to its known servicing VIOS after the KSYS daemon restart. The way FDE
thread uses this XML response payload is described in “QuickQuery flow” on page 234.
We started the analysis of this QuickQuery request, which is also the first HMC REST API job
request after the KSYS daemon restart, in Example 2-69 on page 106. We presented in detail
a complete asynchronous job operation that is performed by KSYS to communicate with a
managed VIOS by the HMC pass-through APIs.
2.2.7 Host monitor, Health Status Database, and related VIOS internals
Section “HMC pass-through APIs” on page 101 showed how the HMC to VIOS RMC
connection and vioservice command support KSYS to communicate with the VIOS side by
HMC pass through APIs. On the VIOS side, the component endpoints that are involved in this
communication include the HSDB database on the SSP cluster and the HM daemon. The HM
daemon also intermediates KSYS communication with VM Agents running on VMs.
Figure 2-10 on page 115 shows a view of the internal VIOS subsystems supporting the
various communication flows that are involved. It also suggests possible flow initiators, either
external to VIOS, like KSYS; HMC; and VMM, or internal, like the vio_daemon and HM
(ksys_hsmond) daemons.

Figure 2-10 VIOS subsystems supporting KSYS communication with HSDB and HM
We further consider the VIOS components in Figure 2-10 and not addressed so far:
򐂰 Health library
򐂰 The SSP cluster and HSDB database
򐂰 Host monitor daemon
Health library
The libviohealth.a library has APIs to support KSYS to VIOS communication. The
vioservice lib/libviopass/passthru command is run for any request to the VIOS side that
originated on KSYS. The XML input of each request, which is generated on the KSYS side in
the vioHADR2.00 format, is extracted by the HMC REST server from the HTTP request
payload. Example 2-67 on page 105 shows how it occurs for a VIO_HS_QUICK_QUERY request.
We did not see how the message reaches its destination endpoint, but now we can describe
the next step. The vioservice command launches the libviopass.a library, which internally
launches the appropriate API from the libviohealth.a library, as shown in Example 2-76.
Example 2-76 The libviohealth.a APIs that were launched for KSYS requests
# dump -H /usr/ios/sbin/vioservice|grep -p "Import File Strings"
***Import File Strings***
INDEX PATH BASE MEMBER
0 /usr/lib:/lib:/usr/lpp/xlC/lib
1 libc.a shr.o
2 libviolog.a shr.o

3 libvio.a shr.o
4 libviopass.a shr.o
# dump -Tv /usr/ios/sbin/vioservice|grep IMP|grep "libviopass.a"

[54] 0x00000000 undef IMP DS EXTref libviopass.a(shr.o) vioPassThru
# dump -Tv /usr/lib/libviopass.a|grep IMP|grep "libviohealth.a"|awk '{print $1,
$4, $7, $8}'
[13] IMP libviohealth.a(shr.o) vioHsHandle
[14] IMP libviohealth.a(shr.o) vioHsFreeHandle
[15] IMP libviohealth.a(shr.o) vioHsRmHostGrp
[16] IMP libviohealth.a(shr.o) vioHsAddMs
[17] IMP libviohealth.a(shr.o) vioHsNeedsAttention
[18] IMP libviohealth.a(shr.o) vioHsQuickQuery
[19] IMP libviohealth.a(shr.o) vioHsQueryVM
[20] IMP libviohealth.a(shr.o) vioHsAddVm
[21] IMP libviohealth.a(shr.o) vioHsStartVm
[22] IMP libviohealth.a(shr.o) vioHsStopVm
[23] IMP libviohealth.a(shr.o) vioHsAcknowledge
[24] IMP libviohealth.a(shr.o) vioHsRemoveVm
[25] IMP libviohealth.a(shr.o) vioHsPutMsg
[26] IMP libviohealth.a(shr.o) vioHsGetMsg
[27] IMP libviohealth.a(shr.o) vioHsRemoveAllMsg
[28] IMP libviohealth.a(shr.o) vioHsAckMsg
[29] IMP libviohealth.a(shr.o) vioHsSyncMsg
[30] IMP libviohealth.a(shr.o) vioHsQueryMsg
[31] IMP libviohealth.a(shr.o) vioHsQueryNodeData
[32] IMP libviohealth.a(shr.o) vioHsConfigureHM
[33] IMP libviohealth.a(shr.o) vioHsSetComHmVersions
[34] IMP libviohealth.a(shr.o) vioHsRemoveMs
[35] IMP libviohealth.a(shr.o) vioHsRemoveVios
[36] IMP libviohealth.a(shr.o) vioHsAddVios
[37] IMP libviohealth.a(shr.o) vioHsQueryMSforVMs
[38] IMP libviohealth.a(shr.o) vioHsQueryConflict
#
You can easily see that most of the function names in the last column of the last output in
Example 2-76 on page 115 fit with a corresponding KSYS to VIOS request type in the set of
all these types of requests that is extracted in Example 2-77 from the
/usr/lib/vioHADR2.00.xsd XSD file.
Example 2-77 KSYS request types as defined in /usr/lib/vioHADR2.00.xsd

...
<vioHADR2.00xsd:simpleType name="reqActionType">
<xsd:restriction base="xsd:string">
<xsd:enumeration value="VIO_HS_ADD_MS"/>
<xsd:enumeration value="VIO_HS_REMOVE_MS"/>
<xsd:enumeration value="VIO_HS_STOP_VM"/>
<xsd:enumeration value="VIO_HS_START_VM"/>
<xsd:enumeration value="VIO_HS_ADD_VM"/>
<xsd:enumeration value="VIO_HS_REMOVE_VM"/>
<xsd:enumeration value="VIO_HS_ADD_VIOS"/>
<xsd:enumeration value="VIO_HS_REMOVE_VIOS"/>
<xsd:enumeration value="VIO_HS_REMOVE_HOST_GROUP"/>
<xsd:enumeration value="VIO_HS_CONFIGURE_HM"/>

<xsd:enumeration value="VIO_HS_PUT_MSG"/>
<xsd:enumeration value="VIO_HS_GET_MSG"/>
<xsd:enumeration value="VIO_HS_ACK_MSG"/>
<xsd:enumeration value="VIO_HS_SYNC_MSG"/>
<xsd:enumeration value="VIO_HS_REMOVE_ALL_MSG"/>
<xsd:enumeration value="VIO_HS_NEEDS_ATTENTION"/>
<xsd:enumeration value="VIO_HS_QUICK_QUERY"/>
<xsd:enumeration value="VIO_HS_ACKNOWLEDGE"/>
<xsd:enumeration value="VIO_HS_QUERY_NODE_DATA"/>
<xsd:enumeration value="VIO_HS_QUERY_VM"/>
<xsd:enumeration value="VIO_HS_QUERY_MSG"/>
<xsd:enumeration value="VIO_HS_QUERY_MS_FOR_VMS"/>
<xsd:enumeration value="VIO_HS_QUERY_CONFLICT"/>
</xsd:restriction>
</xsd:simpleType>
...
We regrouped these KSYS to VIOS request types into three categories:

򐂰 Actions.
򐂰 Queries to report the health status for VIOSs and VMs.
򐂰 Requests that are used for message passing from KSYS to VM and the HM daemon.
You can refer to the /usr/lib/vioHADR2.00.xsd XSD file for descriptions of these request
type definitions and for more information about the related input and response XML elements
format and attributes.
KSYS to VIOS communication traces are logged in to the viohs.log file in the
/home/ios/logs/health directory. In Example 2-78, we examine the occurrence of previously
identified APIs that starting with the vioHs prefix in the viohs.log file. We look first for the
vioHs pattern in a snapshot of viohs.log that was taken on one VIOS of our test environment.
Then, we count the occurrences of each identified KSYS to VIOS request to the libhealth.a
API.
Example 2-78 KSYS to VIOS communication traces in the viohs.log file

# pwd
/home/ios/logs/health
# alog -f viohs.log -o > viohs.log.libhealth.txt
# grep vioHs viohs.log.libhealth.txt|head -8
[3 15925534 64487765 01/19/19-09:27:33.160 hs_query_vm.c vioHsQueryVM 1.15 236]
Error handling msys rc = 129.
[3 17105352 63963515 01/19/19-09:27:33.173 hs_atten.c vioHsQuickQuery 1.107 4669]
End, rc=0
End, rc=0
[3 15597926 66519541 01/19/19-09:27:59.477 hs_remove_vm.c vioHsRemoveVm 1.18 103]
RemoveVM request tag is 1122B46D0FDEDF37
RemoveVM requested for (1) machines

skipping 24808803-eb32-4d0a-8aa5-a5b60216f15a: unknown machine
#
# grep vioHs viohs.log.libhealth.txt|tail -4
End, rc=0
End, rc=0
[3 20382202 66191779 01/20/19-01:57:25.594 hs_atten.c vioHsNeedsAttention 1.107
4587] End, rc=0
End, rc=0
# grep -c vioHs viohs.log.libhealth.txt
4045
# grep -c vioHsQuickQuery viohs.log.libhealth.txt
1747
# grep -c vioHsNeedsAttention viohs.log.libhealth.txt
873
# grep -c vioHsQueryVM viohs.log.libhealth,txt
276
# grep -c vioHsRemoveVm viohs.log.libhealth.txt
1104
# grep vioHs viohs.log.libhealth.txt|egrep -v
"vioHsQuickQuery|vioHsNeedsAttention|vioHsQueryVM|vioHsRemoveVm"|head -4
[3 16056632 65143055 01/19/19-10:32:36.797 hs_ms.c vioHsAddMs 1.83 2254] Begin.
[3 16056632 65143055 01/19/19-10:32:36.820 hs_ms.c vioHsAddMs 1.83 2274] Managed
system is already existing.
[3 16056632 65143055 01/19/19-10:32:36.820 hs_ms.c vioHsAddMs 1.83 2451] END.
# grep -c vioHsAddMs viohs.log.libhealth.txt
45
#
So, any string containing the vioHs substring in our log excerpt in Example 2-78 on page 117
is an occurrence of an API call from the libviohealth.a library:
4045=1747+873+276+1104+45.
Another legitimate question about our viohs.log file excerpt is which are the processes
writing there. From our presentation so far, the best candidate appears to be the vioservice
command. We choose the last entry of the tail output in Example 2-78 on page 117, which is
a vioHsQuickQuery trace that is left by the thread 66847175 of process 17170756. Checking
for this PID-TID combination in viosvc.log of the vioservice command, we obtain the
request and response details of a related VIO_HS_QUICK_QUERY request, as shown in
Example 2-79.
Example 2-79 VIO_HS_QUICK_QUERY request trace in viosvc.log and viohs.log

# alog -f /home/ios/logs/viosvc.log -o > viosvc.log.libhealth.txt
# more viosvc.log.libhealth.txt
...
[0 17170756 66847175 01/20/19-01:57:49.883 viosvc_res.c 1.26 456] stdin pipe
input:[<?xml version="1.0"?>

</VIO>
]
[0 17170756 66847175 01/20/19-01:57:49.945 viosvc_res.c 1.26 464]
<quickQuery>
</quickQuery>
<quickQuery>
</quickQuery>
</Response></VIO>
]
[END 17170756 66847175 01/20/19-01:57:49.945 vioservice.c 1.18 323] exited with
rc=0
viosvc.log.libhealth.txt (92%)
#
# cd health
# pwd
# more viohs.log.libhealth.txt
...
[START 17170756 66847175 01/20/19-01:57:49.884 ha_util.c 1.43 230]
[3 17170756 66847175 01/20/19-01:57:49.884 hs_util.c hs_libvio_traces 1.66 60]
HEALTH:66847175 -- violibExchange.c initTransaction 1.27 646 Got
cluster info from vioGetCluster
InfoFromODM! cluster name='KSYS_rbRMHA_1' id=43b050741b4111e9803e001018affe76
HEALTH:66847175 -- violibDB.c _allocHandleDB 1.132.12.3 589
HDBC = 200176e8
HDBC = 200176e8
End, rc=0
[END 17170756 66847175 01/20/19-01:57:49.944 ha_util.c 1.43 253] exited with rc=0
viohs.log.libhealth.txt: END
#
We also added in Example 2-79 on page 118 the traces that were written by our vioservice
thread (17170756 66847175) in viohs.log around the examined vioHsQuickQuery call such
that while looking at the time stamps of these traces in both logs, you can easily visualize the
whole sequence of steps that is performed.

HM also uses some health library APIs to report its health and the health of VMs. These APIs
are referred as callback APIs. Example 2-80 lists the libviohealth.a callback APIs that can
be started from the HM side.
Example 2-80 The libviohealth.a APIs started from HM

# dump -Tv /usr/sbin/ksys_hsmond|grep IMP|grep libviohealth.a|awk '{print $1, $4, $7, $8}'
[253] IMP /usr/lib/libviohealth.a(shr.o) vioCreateHmTrans
[254] IMP /usr/lib/libviohealth.a(shr.o) vioFreeHmTrans
[255] IMP /usr/lib/libviohealth.a(shr.o) vioHmClientMissed
[256] IMP /usr/lib/libviohealth.a(shr.o) vioHmHbDetected
[257] IMP /usr/lib/libviohealth.a(shr.o) vioHmHb
[258] IMP /usr/lib/libviohealth.a(shr.o) vioHmStartVMs
[259] IMP /usr/lib/libviohealth.a(shr.o) vioHmStopVMs
[260] IMP /usr/lib/libviohealth.a(shr.o) vioHmVmDiscover
[261] IMP /usr/lib/libviohealth.a(shr.o) vioHmRemoveVMs
[262] IMP /usr/lib/libviohealth.a(shr.o) vioHmVMremoved
[263] IMP /usr/lib/libviohealth.a(shr.o) vioHmVmSuspend
[264] IMP /usr/lib/libviohealth.a(shr.o) vioHmVmResume
[265] IMP /usr/lib/libviohealth.a(shr.o) vioHmReadMsg
[266] IMP /usr/lib/libviohealth.a(shr.o) vioHmShortMsg
[267] IMP /usr/lib/libviohealth.a(shr.o) vioHmWriteMsg
[268] IMP /usr/lib/libviohealth.a(shr.o) vioHmAckMsg
[269] IMP /usr/lib/libviohealth.a(shr.o) vioHmPrepErrorHandler
[270] IMP /usr/lib/libviohealth.a(shr.o) vioHmErrHandler
[271] IMP /usr/lib/libviohealth.a(shr.o) vioHmConfigureService
[272] IMP /usr/lib/libviohealth.a(shr.o) vioHmVmReport
[273] IMP /usr/lib/libviohealth.a(shr.o) vioHmRestartVMs
[274] IMP /usr/lib/libviohealth.a(shr.o) vioHmGetRecoveryLock
[275] IMP /usr/lib/libviohealth.a(shr.o) vioHmReleaseRecoveryLock
#
HM to VIOS communication traces are logged in to the viohm.log file in the same
/home/ios/logs/health directory. In Example 2-81, we look for the occurrences of each
identified HM to VIOS request to the libhealth.a API in a snapshot of viohm.log that was
taken on one VIOS of our test environment.
Example 2-81 HM to VIOS communication traces in the viohm.log file

# pwd
# alog -f viohm.log -o > viohm.log.libhealth.txt
# grep vioHm viohm.log.libhealth.txt|head -4
[3 16777576 72483203 10/29/18-16:46:44.935 hm_discover.c vioHmVmDiscover 1.41 946]
Begin, vm uuid '73a45810-6b63-4a1d-898e-bc6a7edcd043'
[3 16777576 72483203 10/29/18-16:46:44.964 hm_discover.c vioHmVmDiscover 1.41
1050] End rc = 0.
[3 16777576 75891147 10/29/18-16:47:10.993 ha_socket.c vioHmStartVMs 1.56.1.53
793] starting 1 vms
[3 16777576 72483203 10/29/18-16:47:45.66 hm_discover.c vioHmVmDiscover 1.41 946]
Begin, vm uuid '192575b8-e6a7-4e67-b77c-049c45ecd027'
# dump -Tv /usr/sbin/ksys_hsmond|grep IMP|grep libviohealth.a|awk '{print
$8}'|while read vioHmAPI; do echo "$vioHmAPI\t\c"; grep -c "$vioHmAPI\ "
viohm.log.libhealth.txt; done
vioCreateHmTrans 0
vioFreeHmTrans 0
vioHmClientMissed 9
vioHmHbDetected 83

vioHmHb 145
vioHmStartVMs 39
vioHmStopVMs 64
vioHmVmDiscover 66
vioHmRemoveVMs 0
vioHmVMremoved 0
vioHmVmSuspend 14
vioHmVmResume 0
vioHmReadMsg 254
vioHmShortMsg 472
vioHmWriteMsg 254
vioHmAckMsg 254
vioHmPrepErrorHandler 0
vioHmErrHandler 0
vioHmConfigureService 0
vioHmVmReport 103
vioHmRestartVMs 0
vioHmGetRecoveryLock 0
vioHmReleaseRecoveryLock 0
#
The libviohealth.a library is closely linked with libvio.a and uses APIs that are already
available there. We mention, as examples, APIs to parse XML input or APIs to access the
HSDB database in the SSP, as shown in Example 2-82.
Example 2-82 APIs in libvio.a that are used by libviohealth.a

# dump -H /usr/lib/libviohealth.a|grep -p "Import File Strings"
0 /usr/lib:/lib
1 libvio.a shr.o
2 libviolog.a shr.o
3 libc.a shr.o
4 libodm.a shr.o
5 libcfg.a shr.o
6 libpthreads.a shr_xpg5.o
7 / unix
# dump -H /usr/lib/libvio.a|grep -p "Import File Strings"

0 /usr/lib:/lib
1 libcfg.a shr.o
2 libc.a shr.o
3 libxml2.a libxml2.shr.o
4 libodm.a shr.o
5 liblvm.a shr.o
6 / unix
7 libbsd.a shr.o
8 libpthreads.a shr_xpg5.o
# dump -Tv /usr/lib/libviohealth.a|grep IMP|egrep -i "db|schema"|awk '{print $1,

$4, $7, $8}'
[0] IMP libvio.a(shr.o) vioFinishQueryDB
[1] IMP libvio.a(shr.o) vioFreeStmtDB

[2] IMP libvio.a(shr.o) vioDisconnectDB
[3] IMP libvio.a(shr.o) vioConnectDB
[4] IMP libvio.a(shr.o) vioAllocStmtDB
[9] IMP libvio.a(shr.o) vioRollbackDB
[10] IMP libvio.a(shr.o) vioCommitDB
[11] IMP libvio.a(shr.o) vioQueryDB
[12] IMP libvio.a(shr.o) vioGetCountFromDB
[28] IMP libvio.a(shr.o) vioCheckSchemaExists
[29] IMP libvio.a(shr.o) vioModifyWithRetryDB
[33] IMP libvio.a(shr.o) vioUpdateDB
[34] IMP libvio.a(shr.o) vioDeleteDB
[35] IMP libvio.a(shr.o) vioInsertDB
#
The SSP cluster and HSDB database

The vio_daemon daemon provides core SSP cluster functions for a VIOS node that is a
member of a cluster. Figure 2-10 on page 115 shows vio_daemon as being distinct from the
SSP cluster, but we describe it in this section because the cluster and database are closely
related. Other subsystems contribute to the operation of an SSP cluster, like cthags, caa, and
pool. In Example 2-83, we see the status of all these subsystems after a fresh VIOS V3.1
installation and before the creation of an SSP cluster.
Example 2-83 Initial SSP daemons status

3.1.0.10
# oslevel -s
7200-03-02-1846
# lssrc -a|head -1; lssrc -a|egrep "caa|cthags|^ pool|vio_daemon"|sort
clcomd caa 6816124 active
clconfd caa inoperative
pool inoperative
vio_daemon 7471600 active
#
Example 2-84 shows the same daemons on a VIOS in our test environment host group after
the SSP cluster creation with hdisk4 as repo_disk and hdisk3 as ha_disk.
Example 2-84 SSP daemons on a VIOS node in a host group SSP cluster
# lssrc -a|head -1; lssrc -a|egrep "caa|cthags|^ pool|vio_daemon"|sort
clcomd caa 8716570 active
clconfd caa 5833058 active
cthags cthags 13500738 active
pool 18809098 active
# /usr/ios/cli/ioscli cluster -status
Cluster Name State
KSYS_rbRMHA_1 OK

rt14v2 8286-42A022100DEW 2 OK OK
rt14v1 8286-42A022100DEW 1 OK OK

rt13v1 8286-42A0221E0B2V 1 OK OK
rt13v2 8286-42A0221E0B2V 2 OK OK
# /usr/lib/cluster/clras lsrepos
hdisk4 has a cluster repository signature.
Found 1 cluster repository disk.
# lspv|grep hdisk4
hdisk4 00f9e0b2b719cb37 caavg_private active
# /usr/ios/cli/ioscli pv -list
POOL_NAME: default_pool_1
TIER_NAME: SYSTEM
FG_NAME: Default
PV_NAME SIZE(MB) STATE UDID
hdisk3 20480 ONLINE 3321360050768028104268800000000000~
#
The PowerVM product documentation for SSP mentions that in VIOS V3.1 the SSP
Management database was migrated from SolidDB in previous VIOS versions to the
PostgreSQL database, as described in this excerpt from the “Getting started with shared
storage pools by using the VIOS command line” section of IBM Power Systems Virtual I/O
Server:
“In VIOS version 3.1 the SSP Management data is stored in a PostgreSQL database. All data
files of the database are stored in the file system of the SSP cluster pool. If the VIOS node
that manages the SSP database is unable to access the file system of the SSP cluster pool,
while the PostgreSQL database process is performing an I/O operation, the PostgreSQL
database aborts all operations and generates a core memory dump. The PostgreSQL
database also generates the pool file system errors and stores them in the system error log
file. The SSP database automatically recovers when the VIOS node that manages the SSP
database regains access to the file system of the SSP cluster pool.”
First, we identify the file system of the SSP cluster pool and the VIOS node that manages the
SSP database for our test environment host group SSP cluster. We expect the file system to
be on the disk that is provided by the ha_disk parameter at the host group creation and show
up under the same mount point on all VIOS nodes (Example 2-85).
Example 2-85 The shared file system of the host group SSP cluster pool
# pooladm pool list
Pool Path
------------------------------
/var/vio/SSP/KSYS_rbRMHA_1/D_E_F_A_U_L_T_061310
# pooladm pool lsdisk /var/vio/SSP/KSYS_rbRMHA_1/D_E_F_A_U_L_T_061310 -v
Tier: SYSTEM
Name: /dev/hdisk3
UID: 000000000A2819F3000000005C420426
Capacity: 20 GB
State: Up
Failure Group: Default
Partitions: 319
Spare Partitions: 1
Stale Partitions: 0
ECCR Partitions: 0
Sector Size: 512 bytes
Max Transfer: 512 KB
Data Start: 8388608 bytes
# mount|head -2;mount|grep "/var/vio/SSP/KSYS_rbRMHA_1/D_E_F_A_U_L_T_061310"

node mounted mounted over vfs date options
-------- --------------- --------------- ------ ------------ ---------------
/var/vio/SSP/KSYS_rbRMHA_1/D_E_F_A_U_L_T_061310
/var/vio/SSP/KSYS_rbRMHA_1/D_E_F_A_U_L_T_061310 poolfs Jan 18 11:20
# grep poolfs /etc/vfs
poolfs 20 /usr/sbin/pooladm none
# clcmd df -g /var/vio/SSP/KSYS_rbRMHA_1/D_E_F_A_U_L_T_061310
-------------------------------
NODE rt13v2.ausprv.stglabs.ibm.com
-------------------------------
Filesystem GB blocks Free %Used Iused %Iused Mounted on
/var/vio/SSP/KSYS_rbRMHA_1/D_E_F_A_U_L_T_061310 19.88 19.36 3% -
- /var/vio/SSP/KSYS_rbRMHA_1/D_E_F_A_U_L_T_061310
-------------------------------
-------------------------------
-------------------------------
-------------------------------
-------------------------------
-------------------------------
#
For more information about the pooladm options that are used in Example 2-85 on page 123,
run the pooladm help and pooladm pool help commands. The VIOS node that manages the
SSP database is known as a database node (DBN). Some details about the way that the
VIOS node that manages the SSP database regains access to the file system of the SSP
cluster pool are described in the “Problem summary” and “Problem conclusion” sections of
APAR IJ08692: VIOS SSP CANNOT ELECT DBN.
Though APAR IJ08692 applies to AIX 6.1 and was fixed in VIOS 2.2.6.32, we expect the
functions that are described to also be relevant for VIOS V3.1 and subsequent versions.
Going further with our SSP cluster examination, we see a PostgreSQL database server
instance that runs on the DBN node under the control of the vios_daemon subsystem
(Example 2-86).
Example 2-86 Database server instance on the DBN node

# lssrc -ls vio_daemon|grep DBN
DBN NODE ID: 43947aac1b4111e9803e001018affe76
DBN Role: Other

# /usr/ios/cli/ioscli cluster -status -verbose|grep -p DBN
Node Name: rt14v2.ausprv.stglabs.ibm.com
Node Id: 43947aac1b4111e9803e001018affe76
Node MTM: 8286-42A022100DEW
Node Partition Num: 2
Node State: OK
Node Repos State: OK
Node Upgrade Status: 3.1.0.00 ON_LEVEL
Node Roles: DBN
Pool Name: default_pool_1
Pool Id: 000000000A2819F3000000005C420425
Pool State: OK
# clcmd -m rt14v2.ausprv.stglabs.ibm.com lssrc -s vio_daemon
-------------------------------
-------------------------------
# clcmd -m rt14v2.ausprv.stglabs.ibm.com proctree 2032030
-------------------------------
-------------------------------
7864602 /usr/sbin/srcmstr
2032030 /usr/sbin/vio_daemon -d 4
17563970 /usr/ios/db/bin/postgres -D
/var/vio/SSP/KSYS_rbRMHA_1/D_E_F_A_U_L_T_061310/VIO
3735808 postgres: SSP: stats collector process
12976504 postgres: SSP: autovacuum launcher process
14418338 postgres: SSP: checkpointer process
14549484 postgres: SSP: logger process
18153846 postgres: SSP: wal writer process
20578610 postgres: SSP: writer process
23134664 postgres: SSP: bgworker: logical replication launcher
24183042 postgres: SSP: archiver process last was
000000010000000000000005.00000028.ba
10486100 vio_chgmgt
# clcmd -m rt14v2.ausprv.stglabs.ibm.com ps -fp 17563970
-------------------------------
-------------------------------
UID PID PPID C STIME TTY TIME CMD
vpgadmin 17563970 2032030 0 10:52:20 - 0:32 /usr/ios/db/bin/postgres -D
/var/vio/SSP/KSYS_rbRMHA_1/D_E_F_A_U_L_T_061310/VIOSCFG/DB/PG -p 6090
#
The PostgreSQL database server instance (PID 17563970) was started as a vpgadmin user,
and its listening port is specified in the command line as -p 6090. The database files are kept
in the shared file system of the SSP cluster pool (by using the -D data directory option).
Options in the command line have priority over the ones in the configuration files. The default
configuration files are in the same location (by using the -D option), so the remaining
database settings can be checked there.

As an example, we look in postgresql.conf on the DBN node itself for the log directory and
some local session settings (Example 2-87).
Example 2-87 Log location and trust authentication settings in the configuration files
DBN Role: Primary
# cd /var/vio/SSP/KSYS_rbRMHA_1/D_E_F_A_U_L_T_061310/VIOSCFG/DB/PG
# ls *.conf
pg_hba.conf pg_ident.conf postgresql.auto.conf postgresql.conf
# grep log_directory postgresql.conf
log_directory = '/home/ios/logs/pg_sspdb' # directory where log files are written,
# grep -i socket postgresql.conf
#unix_socket_directories = '/tmp' # comma-separated list of directories
#unix_socket_group = '' # (change requires restart)
#unix_socket_permissions = 0777 # begin with 0 to use octal notation
# grep ^local pg_hba.conf
local all all trust
# ls -l /tmp|grep vpgadmin|grep ^s
srwxrwxrwx 1 vpgadmin bin 0 Jan 19 03:11 .s.PGSQL.6080
srwxrwxrwx 1 vpgadmin bin 0 Jan 19 03:18 .s.PGSQL.6090
#
For more insights into the database layout, we use a psql client interactive session through a
local UNIX domain socket connection on the DBN node, as shown in Example 2-88.
Example 2-88 SSP cluster database layout

# /usr/ios/db/bin/psql -U vpgadmin -l -p 6090
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
-----------+-----------+----------+---------+-------+-----------------------
postgres | vpgadmin | LATIN1 | en_US | en_US |
template0 | vpgadmin | LATIN1 | en_US | en_US | =c/vpgadmin +
| | | | | vpgadmin=CTc/vpgadmin
template1 | vpgadmin | LATIN1 | en_US | en_US | =c/vpgadmin +
| | | | | vpgadmin=CTc/vpgadmin
vios | viosadmin | LATIN1 | en_US | en_US |
(4 rows)
# /usr/ios/db/bin/psql -U vpgadmin -d vios -p 6090 -P pager=off
psql (10.2)
Type "help" for help.
vios=# \dn
List of schemas
Name | Owner
--------+-----------
public | vpgadmin
vios | viosadmin
vioshs | viosadmin
(3 rows)
vios=# \dt vioshs.*
List of relations
Schema | Name | Type | Owner
--------+-------------------+-------+-----------
vioshs | cluster | table | viosadmin

vioshs | config_hm_daemon | table | viosadmin
vioshs | config_hm_service | table | viosadmin
vioshs | config_hs_service | table | viosadmin
vioshs | conflict | table | viosadmin
vioshs | locked | table | viosadmin
vioshs | map | table | viosadmin
vioshs | ms | table | viosadmin
vioshs | msg | table | viosadmin
vioshs | recovervm | table | viosadmin
vioshs | removed_vm | table | viosadmin
vioshs | trans | table | viosadmin
vioshs | vios | table | viosadmin
vioshs | vm | table | viosadmin
(14 rows)
vios=# \q
# /usr/ios/db/bin/postgres -V
postgres (PostgreSQL) 10.2#
The health status tables are grouped under the vioshs schema as part of the vios SSP
cluster database. Using the same database as the SSP cluster provides some benefits:
򐂰 Using existing database access code
򐂰 Rolling upgrade that is automatically supported
򐂰 Database resiliency as ensured by the DBN election mechanism
In Example 2-89, we continue with the same psql session, and suggest two methods that you
can use to list the column names of a table to help you select only the columns of interest in a
subsequent query.
Example 2-89 Some SQL queries

vios=# set search_path to vioshs;
SET
vios=# select * from cluster limit 0;
clusterid | dbstate | doexch | cur_major | cur_minor | ksysmsgsync | dbninstance
| recoverytag | recoveryts
-----------+---------+--------+-----------+-----------+-------------+-------------
+-------------+------------
(0 rows)
vios=# select clusterid, dbstate, cur_major, ksysmsgsync, dbninstance,
recoverytag from cluster;
clusterid | dbstate | cur_major | ksysmsgsync | dbninstance | recoverytag
-----------+---------+-----------+-------------+-------------+-------------
1 | 8 | 2 | 1 | 1 | 1
(1 row)
vios=# select column_name, data_type from information_schema.columns where
table_name='ms' and table_schema='vioshs';
column_name | data_type
-------------+-------------------
msid | bigint
clusterid | bigint
mtm | character varying
machinetype | character varying
model | character varying
serial | character varying
(6 rows)

vios=# select msid, clusterid, machinetype, model from ms;
msid | clusterid | machinetype | model
----------------------+-----------+-------------+-------
-3157136219997566938 | 1 | 8286 | 42A
-4860436880512962013 | 1 | 8286 | 42A
(2 rows)
vios=#
CAA event monitoring

Before the creation of any SSP cluster on a new VIOS V3.1 installation, you can find the
AHAFS AIX Event Infrastructure already set up, as shown in Example 2-90.
Example 2-90 Autonomic Health Advisor File System setup before SSP cluster creation
3.1.0.10
# oslevel -s
7200-03-02-1846
# df -vT ahafs
Filesystem 512-blocks Used Free %Used Iused Ifree %Iused
Mounted on
/ahafs - - - - 42 32726 1% /aha
# find /aha -name "*.mon"
/aha/fs/modFile.monFactory/var/sea/SEAevents.mon
# fuser -V /aha/fs/modFile.monFactory/var/sea/SEAevents.mon
/aha/fs/modFile.monFactory/var/sea/SEAevents.mon:
inode=37 size=1 fd=3 6881786
# ps -o pid,args -p 6881786
PID COMMAND
6881786 /usr/sbin/SEAmon /aha
# ls -d /aha/*/*.monFactory
/aha/cluster/cap.monFactory /aha/disk/repDiskState.monFactory
/aha/cluster/hostnameChange.monFactory /aha/disk/vgState.monFactory
/aha/cluster/networkAdapterState.monFactory /aha/fs/modDir.monFactory
/aha/cluster/nodeAddress.monFactory /aha/fs/modFile.monFactory
/aha/cluster/nodeContact.monFactory /aha/fs/modFileAttr.monFactory
/aha/cluster/nodeList.monFactory /aha/fs/utilFs.monFactory
/aha/cluster/nodeState.monFactory /aha/mem/vmo.monFactory
/aha/cluster/siteState.monFactory /aha/mem/waitTmPgInOut.monFactory
/aha/cpu/pidProcessMon.monFactory /aha/mem/waitersFreePg.monFactory
/aha/cpu/processMon.monFactory /aha/net/inetsock.monFactory
/aha/cpu/schedo.monFactory /aha/pool/pool.membership.monFactory
/aha/cpu/waitTmCPU.monFactory /aha/pool/tier.capacity.monFactory
/aha/disk/clDiskList.monFactory
/aha/pool/tier.ecLowWaterMark.monFactory
/aha/disk/clDiskState.monFactory /aha/pool/tier.freespace.monFactory
/aha/disk/diskState.monFactory /aha/pool/tier.overcommit.monFactory
#

The .monFactory directories for the expected predefined AIX and CAA event producers are
there. Additionally, we notice some VIOS-specific ones in the /aha/pool directory, which by
name appear to be related to the pool subsystem supporting the SSP cluster operation. Only
one monitor file shows up as being used to monitor the modifications to the
/var/sea/SEAevents file by a SEAmon process.
When an SSP cluster is deployed, a couple of other monitor files show up on its member
nodes, as shown in Example 2-91, where we picked the DBN node. The same monitor files,
except for the ones in the /aha/pool base directory, show up on all the other nodes.
Example 2-91 AHAFS monitor files on the DBN node after the SSP cluster creation
DBN Role: Primary
# find /aha -name "*.mon"
/aha/cpu/processMon.monFactory/usr/sbin/rsct/bin/hagsd.mon
/aha/fs/modFile.monFactory/var/sea/SEAevents.mon
/aha/fs/utilFs.monFactory/var.mon
/aha/cluster/hostnameChange.monFactory/nodeHostNameEvent.mon
/aha/cluster/cap.monFactory/capEvent.mon
/aha/cluster/siteState.monFactory/siteEvent.mon
/aha/cluster/nodeList.monFactory/nodeListEvent.mon
/aha/cluster/networkAdapterState.monFactory/networkAdapterStateEvent.mon
/aha/cluster/nodeAddress.monFactory/nodeAddressEvent.mon
/aha/cluster/nodeState.monFactory/nodeStateEvent.mon
/aha/disk/repDiskState.monFactory/repDiskStateEvent.mon
/aha/disk/clDiskState.monFactory/clDiskStateEvent.mon
/aha/pool/tier.freespace.monFactory/poolId:000000000A2819F3000000005C420425,tierNa
me:SYSTEM,unitName:percentage.mon
/aha/pool/tier.ecLowWaterMark.monFactory/poolId:000000000A2819F3000000005C420425,t
ierName:SYSTEM.mon
#
The vio_daemon (PID 2032030) is among the event consumers. It monitors CAA cluster
events and SSP pool events (Example 2-92).
Example 2-92 AHAFS monitor files and event consumers on the DBN node
# for f in `find /aha -name "*.mon"`; do fuser -V $f; done
/aha/cpu/processMon.monFactory/usr/sbin/rsct/bin/hagsd.mon:
inode=44 size=1 fd=11 11207164
/aha/fs/modFile.monFactory/var/sea/SEAevents.mon:
inode=37 size=1 fd=3 6291956
/aha/fs/utilFs.monFactory/var.mon:
inode=45 size=1 fd=4 24838578
/aha/cluster/hostnameChange.monFactory/nodeHostNameEvent.mon:
inode=60 size=1 fd=38 10617192
/aha/cluster/cap.monFactory/capEvent.mon:
inode=59 size=1 fd=37 10617192
/aha/cluster/siteState.monFactory/siteEvent.mon:
inode=54 size=1 fd=14 11207164
/aha/cluster/nodeList.monFactory/nodeListEvent.mon:
inode=53 size=2 fd=12 10617192
inode=53 size=2 fd=13 11207164
/aha/cluster/networkAdapterState.monFactory/networkAdapterStateEvent.mon:

inode=47 size=3 fd=34 2032030
inode=47 size=3 fd=36 10617192
inode=47 size=3 fd=10 11207164
/aha/cluster/nodeAddress.monFactory/nodeAddressEvent.mon:
inode=52 size=3 fd=13 10617192
inode=52 size=3 fd=12 11207164
inode=52 size=3 fd=5 24838578
/aha/cluster/nodeState.monFactory/nodeStateEvent.mon:
inode=46 size=3 fd=20 2032030
inode=46 size=3 fd=35 10617192
inode=46 size=3 fd=9 11207164
/aha/disk/repDiskState.monFactory/repDiskStateEvent.mon:
inode=57 size=1 fd=36 2032030
/aha/disk/clDiskState.monFactory/clDiskStateEvent.mon:
inode=56 size=1 fd=35 2032030
/aha/pool/tier.freespace.monFactory/poolId:000000000A2819F3000000005C420425,tierNa
me:SYSTEM,unitName:percentage.mon:
inode=58 size=1 fd=40 2032030
/aha/pool/tier.ecLowWaterMark.monFactory/poolId:000000000A2819F3000000005C420425,t
ierName:SYSTEM.mon:
inode=61 size=1 fd=41 2032030
# ps -o pid,args -p 11207164,24838578,10617192,2032030
PID COMMAND
2032030 /usr/sbin/vio_daemon -d 4
10617192 /opt/rsct/bin/IBM.ConfigRMd
11207164 /usr/sbin/rsct/bin/hagsd cthags
24838578 /usr/sbin/clconfd
#
Table 2-3 presents the documented CAA event types for each of the CAA event producers
showing up as configured for the vio_daemon event consumer in Example 2-92 on page 129.
Table 2-3 CAA event types

CAA event producer CAA event type Event description
networkAdapterState ADAPTER_UP, Generated when a network interface goes down or

ADAPTER_DOWN, comes up.
ADAPTER_ADD,
ADAPTER_DEL
nodeState NODE_UP, Generated, for example, when a node fails or is shut

NODE_DOWN down.
repDiskState REP_UP, REP_DOWN Generated when a repository disk goes down or

comes up.
clDiskState DISK_UP, Generated when a cluster disk goes down or comes

DISK_DOWN up.
This whole monitoring framework is extended for KSYS such that vio_daemon can update the
HSDB about received CAA events. When vio_daemon is notified about a CAA event
occurrence, it uses libvio.a and ultimately libviohealth.a to update tables in the
vios.vioshs schema. The VIOS state in the vios table is updated with the received node
state from CAA and the trans table is inserted with CAA event details. To see how some of
these event flows happen, we manually perform a shutdown of a non-DBN VIOS node and
then start it.

The VIOS shutdown generates a NODE_DOWN event on the rest of the CAA cluster nodes. We
are interested to see how this event is recorded in the HSDB database and in the logs of all
components that are involved. Example 2-93 shows the vios and trans database tables just
before the VIOS node shutdown. The VIOSs are in the normal state and the trans table is
clean.
Example 2-93 The vios and trans tables before the VIOS node shutdown
vios=# select viosuuid, state, reason from vios;
viosuuid | state | reason
--------------------------------------+-------+--------
315ce56b-9ba0-46a1-b4bc-da6108574e7e | 1 | 0
4e0f6f60-214b-4d2c-a436-0eac46f7f71f | 1 | 0
5ac3e0dc-13fc-4221-9617-8ef61c4cdd83 | 1 | 0
166c89ef-57d8-4e75-815c-aa5b885560b1 | 1 | 0
(4 rows)
vios=# select * from trans;
vmuuid | viosuuid | msid | tag | opcode | state | data | integer | txstarted
--------+----------+------+-----+--------+-------+------+---------+-----------
(0 rows)
vios=#
To reduce the rate of the updates in the database log and make sure that the CAA event
updates in the database are not affected by KSYS, we stop KSYS (by running stopsrc -s
IBM.VMR) during the test. We also stop HMs by running clcmd stopsrc -s ksys_hsmon on one
VIOS. At the database level, we modify some log parameters for easier reading, as shown in
Example 2-94.
Example 2-94 Database log parameters change

vios=# ALTER SYSTEM SET log_duration = false;
ALTER SYSTEM
vios=# ALTER SYSTEM SET log_connections = true;
ALTER SYSTEM
vios=# ALTER SYSTEM SET log_disconnections = true;
ALTER SYSTEM
vios=# ALTER SYSTEM SET log_line_prefix = '%m|%c|%r';
ALTER SYSTEM
vios=# SELECT pg_reload_conf();
pg_reload_conf
----------------
t
(1 row)
vios=#
In Example 2-95, we show the SSP cluster state after the node shutdown and an excerpt
from the CAA log on the DBN node with related data about the generated CAA NODE_DOWN
event.
Example 2-95 SSP cluster state and CAA log excerpt for the NODE_DOWN event
DBN Role: Primary
# /usr/ios/cli/ioscli cluster -status
Cluster Name State
KSYS_rbRMHA_1 DEGRADED

rt14v2 8286-42A022100DEW 2 OK OK
rt14v1 8286-42A022100DEW 1 OK OK
rt13v1 8286-42A0221E0B2V 1 OK OK
rt13v2 8286-42A0221E0B2V 2 DOWN
# cp /var/adm/ras/syslog.caa /var/adm/ras/syslog.caa.caaevttst.txt
# more /var/adm/ras/syslog.caa.caaevttst.txt
...
Jan 20 09:07:47 rt14v2 caa:info unix: kcluster_lock.c count_active_nodes
418 up_node_map 0xF db_node_map 0x0 local_up_node_map 0xF local_db_node_map
0x0
Jan 20 09:08:04 rt14v2 caa:info unix: kcluster_ip.c update_site_state
3748 update_site_state : site=LOCAL new_up_st=1 old_up_st=1
Jan 20 09:08:04 rt14v2 caa:info unix: kcluster_ip.c update_site_state
3808 Site Shid 1, State 10000, Flags 0
Jan 20 09:08:04 rt14v2 caa:info unix: kbase_kernext_services.c aha_thread_queue
646 The AHAFS event is EVENT_TYPE=NODE_DOWN REASON=NODE_SHUTDOWN NODE_N
UMBER=4 NODE_ID=0xD32BFA3C1B4111E9805D001018AFFE76
CLUSTER_ID=0x43B050741B4111E9803E001018AFFE76
Jan 20 09:08:04 rt14v2 caa:debug cluster[2032030]: [8500]
caa_event_monitor.c caa_aha_monitor 279 Have data on fd,slot [20,1]
Jan 20 09:08:04 rt14v2 caa:debug cluster[2032030]: [8500]
caa_event_monitor.c read_event_node 515 Found reason NODE_SHUTDOWN
Jan 20 09:08:04 rt14v2 caa:info cluster[2032030]: [8500]
caa_event_monitor.c caa_aha_process_event 881 Calling VIO with event
details:
caa_event_monitor.c caa_aha_process_event 882 Type 0x1
caa_event_monitor.c caa_aha_process_event 883 Reason 0x40
caa_event_monitor.c caa_aha_process_event 884 Node UUID
D32BFA3C1B4111E9805D001018AFFE76
Jan 20 09:08:04 rt14v2 caa:info unix: kcluster_event.c ahafs_cap_register
2635 Unregister monitoring the capabilities 0x1
Jan 20 09:08:04 rt14v2 caa:info unix: kcluster_event.c
ahafs_hostname_change_register 2754 Unregister monitoring the node's hostname
change 0x1
Jan 20 09:08:04 rt14v2 caa:info unix: kcluster_event.c ahafs_nodelist_register
2094 The nodeList /nodeListEvent is already registered.
Jan 20 09:08:04 rt14v2 caa:info unix: kcluster_event.c ahafs_nodelist_register
2108 nodeList set opqId = 0x1
Jan 20 09:08:04 rt14v2 caa:info unix: kcluster_event.c ahafs_address_register
1612 The node /nodeAddressEvent is already registered.
Jan 20 09:08:04 rt14v2 caa:info unix: kcluster_event.c ahafs_address_register
1625 nodeAddress set opqId = 0x1
Jan 20 09:08:04 rt14v2 caa:info unix: kcluster_event.c ahafs_nodeexit_register
1976 The nodeState /nodeStateEvent is already registered.
Jan 20 09:08:04 rt14v2 caa:info unix: kcluster_event.c ahafs_nodeexit_register
1990 nodeState set opqId = 0x1
Jan 20 09:08:04 rt14v2 caa:info unix: kcluster_event.c ahafs_interface_register
1735 The node /networkAdapterStateEvent is already registered.
Jan 20 09:08:04 rt14v2 caa:info unix: kcluster_event.c ahafs_interface_register
1750 networkAdapterState set opqId = 0x1

Jan 20 09:08:04 rt14v2 caa:info unix: kcluster_event.c ahafs_cap_register
2696 cap set opqId = 0x1
Jan 20 09:08:04 rt14v2 caa:info unix: kcluster_event.c
ahafs_hostname_change_register 2814 HostnameChange set opqId = 0x1
caa_event_monitor.c caa_aha_monitor 256 Starting wait on select().
Jan 20 09:08:31 rt14v2 caa:info cluster[20840742]: [1] clcmd.c main 72
START clcmd command starts...
syslog.caa.caaevttst.txt (98%)
We obtain the expected CAA NODE_DOWN event that is logged around 09:08:04, as marked in
bold in Example 2-95 on page 131. Now, we check the vio_demon log entries on the DBN
node around the same 09:08:04 moment and look for their TID. We also do a quick check of
the thread stack for the thread that logged the entries, which has the TID 0x2134
(Example 2-96).
Example 2-96 The vio_daemon log entries and involved thread details
# pwd
/home/ios/logs
# cp viod.log viod.log.caaevttst.txt
# more viod.log.caaevttst.txt
...
Jan 20 2019, 09:07:58.675 0x129 viod_dbhandler.c viod_start_db_timer 1.83 340
TRC -- Setting up the DB handler timer dbcl 110099400
Jan 20 2019, 09:08:04.201 0x2134 violibDB.c vioCheckSchemaExists
1.132.12.3 23524 DEB -- Parms schemaName=vioshs
Jan 20 2019, 09:08:04.219 0x2134 violibDB.c modifyDB
1.132.12.3 2040 DEB -- Statement = SET search_path TO vios
Jan 20 2019, 09:08:04.222 0x2134 violibDB.c vioGetEADB
1.132.12.3 16415 DEB -- eaName = VIO_DB_RWLEVEL has no value
Jan 20 2019, 09:08:04.222 0x2134 violibDB.c vioQueryDB
1.132.12.3 2692 DEB -- Statement= SELECT COUNT(schema_name) FROM
information_schema.schemata WHERE schema_name='vioshs'
Jan 20 2019, 09:08:04.225 0x2134 violibDB.c bindQueryResults
1.132.12.3 2497 DEB -- Query column count = 1
Jan 20 2019, 09:08:21.114 0x1 viod_misc_handler.c viod_cache_timercb 1.3 283
TRC -- Misc task handler condition set 0x1100994b0
viod.log.caaevttst.txt (91%)
# printf "%d\n" 0x2134
8500
# lssrc -s vio_daemon
# procstack 2032030|grep tid|grep 8500
---------- tid# 62521639 (pthread ID: 8500) ----------
# procstack 2032030|more
...
---------- tid# 62521639 (pthread ID: 8500) ----------
0x090000000016590c __fd_select(??, ??, ??, ??, ??) + 0xcc
0x090000000128b7a0 caa_aha_monitor(??) + 0x760
0x0900000000b40674 vio_caa_aha_monitor(??, ??) + 0x54
0x000000010006e8fc viod_cluster_caa_thread(??) + 0x21c
0x090000000059ffe8 _pthread_body(??) + 0xe8
---------- tid# 64029023 (pthread ID: 8243) ----------
/62521639
#

The health library log entries around the same moment, 09:08:04, are written by the same
vio_daemon thread, which is tid# 62521639 (Example 2-97).
Example 2-97 Health library entries around the CAA NODE_DOWN event occurrence moment
# pwd
# alog -f viohs.log -o > viohs.log.20jan.caaevttst.txt
# more viohs.log.20jan.caaevttst.txt
...
[START 2032030 62521639 01/20/19-09:08:04.226 ha_util.c 1.43 230]
cluster info from vioGetCluste
rInfoFromODM! cluster name='KSYS_rbRMHA_1' id=43b050741b4111e9803e001018affe76
HDBC = 1100987d0
HDBC = 1105341b0
viohs.log.20jan.caaevttst.txt: END
#
The entries in the database log around the same moment, 09:08:04, match perfectly in terms
of content and time stamps with the previous entries that were logged by the vio_daemon in its
own log and in the health library log (Example 2-98).
Example 2-98 Database log around the CAA NODE_DOWN event occurrence moment
# ifconfig en0|grep inet
# pwd
/home/ios/logs/pg_sspdb
# more pg_sspdb-20-08-27.log
...
2019-01-20 09:07:58.675 CST|5c448ece.bb01f0|10.40.25.243(49883)LOG: disconnection: session
time: 0:00:00.030 user=viosadmin database=vios host=10.40.25.243 port=49883
2019-01-20 09:08:04.206 CST|5c448ed4.18d01ce|10.40.25.243(49884)LOG: connection received:
host=10.40.25.243 port=49884
2019-01-20 09:08:04.210 CST|5c448ed4.18d01ce|10.40.25.243(49884)LOG: connection authorized:
user=viosadmin database=vios
2019-01-20 09:08:04.216 CST|5c448ed4.18d01ce|10.40.25.243(49884)LOG: statement: SET DateStyle =
'ISO';SET extra_float_digits = 2;show transaction_isolation
2019-01-20 09:08:04.217 CST|5c448ed4.18d01ce|10.40.25.243(49884)LOG: statement: select oid,
typbasetype from pg_type where typname = 'lo'
2019-01-20 09:08:04.218 CST|5c448ed4.18d01ce|10.40.25.243(49884)LOG: statement: set
client_encoding to 'UTF8'
2019-01-20 09:08:04.219 CST|5c448ed4.18d01ce|10.40.25.243(49884)LOG: statement: BEGIN;SET
search_path TO vios
2019-01-20 09:08:04.220 CST|5c448ed4.18d01ce|10.40.25.243(49884)LOG: statement: COMMIT
2019-01-20 09:08:04.222 CST|5c448ed4.18d01ce|10.40.25.243(49884)LOG: statement: BEGIN;SELECT
COUNT(schema_name) FROM information_schema.schemata WHERE schema_name='vioshs'
2019-01-20 09:08:04.231 CST|5c448ed4.18d01ce|10.40.25.243(49884)LOG: disconnection: session

2019-01-20 09:08:04.237 CST|5c448ed4.18d01d0|10.40.25.243(49885)LOG: connection received:
host=10.40.25.243 port=49885
2019-01-20 09:08:04.258 CST|5c448ed4.18d01d0|10.40.25.243(49885)LOG: connection authorized:
2019-01-20 09:08:04.263 CST|5c448ed4.18d01d0|10.40.25.243(49885)LOG: statement: SET DateStyle =
2019-01-20 09:08:04.264 CST|5c448ed4.18d01d0|10.40.25.243(49885)LOG: statement: select oid,
2019-01-20 09:08:04.268 CST|5c448ed4.18d01d0|10.40.25.243(49885)LOG: statement: set
2019-01-20 09:08:04.274 CST|5c448ed4.18d01d0|10.40.25.243(49885)LOG: statement: BEGIN;SET
search_path TO vios
2019-01-20 09:08:04.274 CST|5c448ed4.18d01d0|10.40.25.243(49885)LOG: statement: COMMIT
2019-01-20 09:08:04.277 CST|5c448ed4.18d01d0|10.40.25.243(49885)LOG: statement: BEGIN;UPDATE
vioshs.vios SET state=2, reason=64, stateRecovery=0, recoveryDbnInstance=0, recoveryTag=0 WHERE
nodeid_1=-3230213171645967895 AND nodeid_2=-9197194794887020938
2019-01-20 09:08:04.279 CST|5c448ed4.18d01d0|10.40.25.243(49885)LOG: statement: COMMIT
2019-01-20 09:08:04.283 CST|5c448ed4.18d01d0|10.40.25.243(49885)LOG: disconnection: session
2019-01-20 09:08:04.290 CST|5c448ed4.13c0198|10.40.25.243(49886)LOG: connection received:
host=10.40.25.243 port=49886
2019-01-20 09:08:04.293 CST|5c448ed4.13c0198|10.40.25.243(49886)LOG: connection authorized:
2019-01-20 09:08:04.295 CST|5c448ed4.13c0198|10.40.25.243(49886)LOG: statement: SET DateStyle =
2019-01-20 09:08:04.296 CST|5c448ed4.13c0198|10.40.25.243(49886)LOG: statement: select oid,
2019-01-20 09:08:04.297 CST|5c448ed4.13c0198|10.40.25.243(49886)LOG: statement: set
2019-01-20 09:08:04.298 CST|5c448ed4.13c0198|10.40.25.243(49886)LOG: statement: BEGIN;SET
search_path TO vios
2019-01-20 09:08:04.300 CST|5c448ed4.13c0198|10.40.25.243(49886)LOG: statement: COMMIT
2019-01-20 09:08:04.301 CST|5c448ed4.13c0198|10.40.25.243(49886)LOG: statement: BEGIN;SELECT
viosuuid,msid FROM vioshs.vios WHERE vioshs.vios.nodeid_1='-3230213171645967895' AND
vioshs.vios.nodeid_2='-9197194794887020938'
2019-01-20 09:08:04.303 CST|5c448ed4.13c0198|10.40.25.243(49886)LOG: statement: select
n.nspname, c.relname, a.attname, a.atttypid, t.typname, a.attnum, a.attlen, a.atttypmod,
a.attnotnull, c.relhasrules, c.relkind, c.oid, pg_get_expr(d.adbin, d.adrelid), case t.typtype
when 'd' then t.typbasetype else 0 end, t.typtypmod, c.relhasoids, attidentity from
(((pg_catalog.pg_class c inner join pg_catalog.pg_namespace n on n.oid = c.relnamespace and
c.oid = 16916) inner join pg_catalog.pg_attribute a on (not a.attisdropped) and a.attnum > 0 and
a.attrelid = c.oid) inner join pg_catalog.pg_type t on t.oid = a.atttypid) left outer join
pg_attrdef d on a.atthasdef and d.adrelid = a.attrelid and d.adnum = a.attnum order by
n.nspname, c.relname, attnum
2019-01-20 09:08:04.312 CST|5c448ed4.13c0198|10.40.25.243(49886)LOG: statement: SAVEPOINT
_EXEC_SVP_110868230;INSERT INTO vioshs.trans(vmuuid, viosuuid, msid, tag, opcode, state, data,
integer, txstarted) VALUES('none','315ce56b-9ba0-46a1-b4bc-da6108574e7e', -3157136219997566938,
144264181326017407, 9, 1, 'BEGIN_EVENT_INFO
TIME_tvsec=1547996884
TIME_tvnsec=200198833
SEQUENCE_NUM=4
RC_FROM_EVPROD=0
BEGIN_EVPROD_INFO
EVENT_TYPE=NODE_DOWN
REASON=NODE_SHUTDOWN

NODE_NUMBER=4
NODE_ID=0xD32BFA3C1B4111E9805D001018AFFE76
END_EVPROD_INFO
END_EVENT_INFO
', 64, NOW())
2019-01-20 09:08:04.312 CST|5c448ed4.13c0198|10.40.25.243(49886)LOG: statement: COMMIT
2019-01-20 09:08:04.313 CST|5c448ed4.13c0198|10.40.25.243(49886)LOG: disconnection: session
2019-01-20 09:08:18.743 CST|5c448ee2.19b0118|10.40.25.242(34310)LOG: connection received:
host=10.40.25.242 port=34310
pg_sspdb-20-08-27.log (99%)
#
We see in Example 2-98 on page 134 that three subsequent database sessions opened from
local IP 10.40.25.243 on ports 49884, 49886, and 49886. The session starting entry
(containing the connection received string) for each of them is marked as bolded for easier
orientation in this long excerpt. The %c escape in the log_line_prefix parameter prints a
quasi-unique session identifier that consists of two 4-byte hexadecimal numbers (without
leading zeros) that are separated by a dot. The numbers are the process start time and the
process ID (PID), so %c can also be used for easier orientation.
Comparing the time stamps of the actions that are performed on the database in the first
session (5c448ed4.18d01ce on port 49884) with the time stamps and actions that are written
by the vio_demon thread 0x2134 in viod.log from Example 2-96 on page 133, we see both log
the same query, SELECT COUNT(schema_name) FROM information_schema.schemata WHERE
schema_name='vioshs', performed at the same moment, 09:08:04.222.
We now examine the actions of the second database session, 5c448ed4.18d01d0, initiated
from the same DBN node, IP address 10.40.25.243 and port 49885. The same vio_daemon
thread logs an _allocHandleDB entry at 09:08:04.226 in the health library (Example 2-97 on
page 134) and then we see how this second database session starts at 09:08:04.237 and
performs an update in the vioshs.vios table for the row that is identified by the WHERE
nodeid_1=-3230213171645967895 AND nodeid_2=-9197194794887020938 clause.
The nodeid_1 and nodeid_2 values in the WHERE clause are derived from the 128-bit CAA
cluster node UUID that is delivered within the CAA event message itself. Here it shows up first
in Example 2-95 on page 131 inside the syslog.caa excerpt, as
NODE_ID=0xD32BFA3C1B4111E9805D001018AFFE76. The most 64 significant bits and the least 64
significant bits of the NODE_ID convert to the nodeid_1 and nodeid_2 field values of type
bigint in the vios table (Example 2-99).
Example 2-99 Database nodeid_1, nodeid_2, SSP cluster node_id, and CAA cluster node UUID
vios=# select viosuuid, nodeid_1, nodeid_2 from vios;
viosuuid | nodeid_1 | nodeid_2
--------------------------------------+----------------------+----------------------
166c89ef-57d8-4e75-815c-aa5b885560b1 | 4869651976704561641 | -9205920519165051274
4e0f6f60-214b-4d2c-a436-0eac46f7f71f | 9220310481544221161 | -9201135444560970122
5ac3e0dc-13fc-4221-9617-8ef61c4cdd83 | -6297861994804342295 | -9199165119723995530
315ce56b-9ba0-46a1-b4bc-da6108574e7e | -3230213171645967895 | -9197194794887020938
(4 rows)
vios=# select to_hex(-3230213171645967895);
to_hex
------------------

d32bfa3c1b4111e9
(1 row)
to_hex
------------------
805d001018affe76
(1 row)
vios=#
The reason=64 value in the UPDATE command appears to be taken from the CAA event
message, which is Reason 0x40. state=2, and all the other values appear to be hardcoded
for this type of event.
Similarly, the third session (5c448ed4.13c0198) starts at 09:08:04.290, just after the
vio_daemon thread logs the _allocHandleDB entry at 09:08:04.280 in the health library
(Example 2-97 on page 134). The code inside this third session extracts the viosuuid and
msid values from the vios table row that is identified by the same WHERE clause as in the
previous session. Then, the code inserts a row with appropriate field values in the trans table.
The vmmuuid is not relevant here (it is a constant string ‘none’ value), the viosuuid and msid
values are the ones that were extracted previously from the vios table, and tag appears to be
generated by code. Both opcode and state appear to be hardcoded, data and integer are
taken from the event message itself, and txtstarted is computed by the NOW() function.
Example 2-100 shows the resulting vios and trans tables in the database.
Example 2-100 Updated vios and trans tables

--------------------------------------+-------+--------
166c89ef-57d8-4e75-815c-aa5b885560b1 | 1 | 0
4e0f6f60-214b-4d2c-a436-0eac46f7f71f | 1 | 0
5ac3e0dc-13fc-4221-9617-8ef61c4cdd83 | 1 | 0
315ce56b-9ba0-46a1-b4bc-da6108574e7e | 2 | 64
(4 rows)
vios=# select viosuuid, tag, opcode, state from trans;
viosuuid | tag | opcode | state
--------------------------------------+--------------------+--------+-------
315ce56b-9ba0-46a1-b4bc-da6108574e7e | 144264181326017407 | 9 | 1
(1 row)
vios=# select data, integer, txstarted from trans;
data | integer | txstarted
-----------------------------------------------+---------+------------------------
BEGIN_EVENT_INFO +| 64 | 2019-01-20 09:08:04.301
TIME_tvsec=1547996884 +| |
TIME_tvnsec=200198833 +| |
SEQUENCE_NUM=4 +| |
RC_FROM_EVPROD=0 +| |
BEGIN_EVPROD_INFO +| |
EVENT_TYPE=NODE_DOWN +| |
REASON=NODE_SHUTDOWN +| |
NODE_NUMBER=4 +| |
NODE_ID=0xD32BFA3C1B4111E9805D001018AFFE76 +| |
CLUSTER_ID=0x43B050741B4111E9803E001018AFFE76+| |
+| |
END_EVPROD_INFO +| |

END_EVENT_INFO +| |
| |
(1 row)
vios=#
Subsequent VIO_HS_QUICK_QUERY or VIO_HS_NEEDS_ATTENTION requests make KSYS aware of

the event so that it can act. To get an idea of what KSYS might receive as response, we run
manually these requests at the VIOS level as the next step of our exercise while KSYS is still
down. Example 2-101 shows the returned response for our manual VIO_HS_QUICK_QUERY.
Example 2-101 Manual VIO_HS_QUICK_QUERY request after the CAA event occurrence
# cat qq.xml
</VIO>
# /usr/ios/sbin/vioservice lib/libviopass/passthru <qq.xml
<VIO><Response>
<quickQuery>
<viosStat uuid="315ce56b-9ba0-46a1-b4bc-da6108574e7e" state="DOWN"
hmResponsive="0" hmResponseSec='206498' hasData='1' reason='0x00000040'
syncMsg="0" />
</quickQuery>
<quickQuery>
</quickQuery>
</Response></VIO>
#
Let us compare the response in Example 2-101 with a typical VIO_HS_QUICK_QUERY response
of a normal steady state where all VIOSs and their HMs are in the OK state. As reference, we
use the VIO_HS_QUICK_QUERY response in Example 2-210 on page 235, where we also detail
the meaning of each viosStat element attribute of a VIOS. We see the normal expected
values of the attributes as being state="UP" hmResponsive="1" hmResponseSec='1'
hasData='0' reason='0x00000000'. Here we have a series of abnormal values, state="DOWN"
hmResponsive="0" hmResponseSec='206498' hasData='1' reason='0x00000040', for the
VIOS we stopped. Each value is expected. The state, hasData, and reason attributes provide
CAA status data. The node state is reported as DOWN and there are extra details in HSDB for
this NODE_DOWN CAA event with reason code 0x40. Conversely, when we stopped HM daemons
on all four VIOSs, they are reported as unresponsive (hmResponsive="0") with the last
heartbeat received from them 206498 seconds before.

Similarly, Example 2-102 shows a returned response of a manual VIO_HS_NEEDS_ATTENTION
request. The same VIOS status data is returned as in the previous VIO_HS_QUICK_QUERY
response, so we note that when an issue is present at the VIOS level that it is also reported in
the VIO_HS_NEEDS_ATTENTION and VIO_HS_QUICK_QUERY replies. Also, we see no details about
any VM probably because the HMs are stopped and the state of the VMs was OK at the
moment when they stopped.
Example 2-102 Manual VIO_HS_NEEDS_ATTENTION request after a CAA event occurrence

# cat na.xml
version="2.00" author="LIBKREST" title="Req Needs Attention">
<Request action_str="VIO_HS_NEEDS_ATTENTION" dataType="GLOBAL_DATA"/>
</VIO>
# /usr/ios/sbin/vioservice lib/libviopass/passthru <na.xml
<VIO><Response>
<needsAtt>
<viosStat uuid="315ce56b-9ba0-46a1-b4bc-da6108574e7e" state="DOWN"
hmResponsive="0" hmResponseSec='207055' hasData='1' reason='0x00000040'
syncMsg="0" />
</needsAtt>
<needsAtt>
</needsAtt>
</Response></VIO>
#
The vioHADR2.00.xsd file mentions how KSYS retrieves this special CAA event data from
HSDB by using the VIO_HS_QUERY_NODE_DATA query. Related data about the node and
repository disk CAA events is sent to KSYS with a tag value. Then, KSYS must acknowledge
the tag back to the VIOS side, which results in clearing that data from the HSDB. The
question is when and how KSYS decides it is the correct moment to use the
VIO_HS_QUERY_NODE_DATA request. We look for this course of action in the KSYS and VIOS
logs after we restart the KSYS daemon, VIOS node, and all HMs that we stopped at the
beginning of this exercise. We also check the database for the expected data clearing.
Example 2-103 shows the trans table shortly after the VIOS node restart.
Example 2-103 HSDB trans table shortly after the VIOS restart
vios=# select data, integer, txstarted from trans order by txstarted;
-----------------------------------------------+---------+------------------------
BEGIN_EVENT_INFO +| 64 | 2019-01-20 09:08:04.301
TIME_tvsec=1547996884 +| |
TIME_tvnsec=200198833 +| |
SEQUENCE_NUM=4 +| |

EVENT_TYPE=NODE_DOWN +| |
REASON=NODE_SHUTDOWN +| |
NODE_NUMBER=4 +| |
+| |
END_EVENT_INFO +| |
| |
BEGIN_EVENT_INFO +| 0 | 2019-01-23 20:24:27.060
TIME_tvsec=1548296666 +| |
TIME_tvnsec=963038500 +| |
SEQUENCE_NUM=9 +| |
EVENT_TYPE=ADAPTER_UP +| |
INTERFACE_NAME=en0 +| |
NODE_NUMBER=4 +| |
+| |
END_EVENT_INFO +| |
| |
BEGIN_EVENT_INFO +| 0 | 2019-01-23 20:24:57.069
TIME_tvsec=1548296696 +| |
TIME_tvnsec=962989923 +| |
SEQUENCE_NUM=5 +| |
EVENT_TYPE=NODE_UP +| |
NODE_NUMBER=4 +| |
+| |
END_EVENT_INFO +| |
| |
(3 rows)
vios=#
We notice that two extra CAA events were inserted into the trans table, one ADAPTER_UP event
type and one NODE_UP event type, for the VIOS node we restarted. The occurrence of these
events is expected, but we also expect that the VIO_HS_QUERY_NODE_DATA request will appear
followed by VIO_HS_ACKNOWLEDGE, which clears the database. The correct places to look for
their direct traces are the krestlong log on the KSYS side and viosvc.log on the VIOS side.

We find the VIO_HS_QUERY_NODE_DATA request easily in krestlong, as shown in
Example 2-104, at about 18 minutes after the registered KSYS start moment. We generated
logs from KSYS trace files by running the rpttr command.
Example 2-104 VIO_HS_QUERY_NODE_DATA request trace in the krestlong log

# pwd
/var/ct/rbRMHA/log/mc/IBM.VMR
# ls 24jan*log*txt
24jan_fde.log.nodeupcaaevttst.txt 24jan_krestlong.log.nodeupcaaevttst.txt
24jan_fdelong.log.nodeupcaaevttst.txt 24jan_ksys.log.nodeupcaaevttst.txt
24jan_krest.log.nodeupcaaevttst.txt 24jan_user.log.nodeupcaaevttst.txt
# grep "Trace Started - Pid" 24jan_ksys.log.nodeupcaaevttst.txt
= 10813744 **********************
# grep "Trace Started - Pid" 24jan_krest.log.nodeupcaaevttst.txt
= 10813744
# more 24jan_krestlong.log.nodeupcaaevttst.txt
...
[02] 01/23/19 T(397) _VMR 22:04:35.571021 DEBUG libkrest.c[11502]: Using 2.00 as
vio hadr version [ Default ]
[02] 01/23/19 T(397) _VMR 22:04:35.571050 DEBUG <?xml version="1.0"?>
version="2.00" author="LIBKREST" title="VIO_HS_QUERY_NODE_DATA">
<Request action_str="VIO_HS_QUERY_NODE_DATA"/>
</VIO>
...
#
Checking the krest log file around the same moment, 22:04:35.57, we see the sequence of a
few requests that are submitted by KSYS before and after the kriSubmitQueryNodeData
specific call, which corresponds to the VIO_HS_QUERY_NODE_DATA request (Example 2-105).
Example 2-105 Submitted requests in krest log around the VIO_HS_QUERY_NODE_DATA moment
# grep kriSubmit 24jan_krest.log.nodeupcaaevttst.txt|more
...
kriSubmitQuickQuery:hmc->(9.3.207.78),vios->(166C89EF-57D8-4E75-815C-AA5B885560B1)
kriSubmitQuickQuery:hmc->(9.3.18.159),vios->(5AC3E0DC-13FC-4221-9617-8EF61C4CDD83)
kriSubmitNeedAttn:hmc->(9.3.18.159),vios->(5AC3E0DC-13FC-4221-9617-8EF61C4CDD83)
kriSubmitNeedAttn returning 0 with jobid->(1547106660325).
kriSubmitAck:hmc->(9.3.18.159),vios->(5AC3E0DC-13FC-4221-9617-8EF61C4CDD83)
[04] 01/23/19 T(9192) _VMR 22:04:35.483532 DEBUG libkrest.c[9664]: kriSubmitAck
returning 0 with jobid->(1547106660326).

kriSubmitQueryNodeData:hmc->(9.3.18.159),vios->(5AC3E0DC-13FC-4221-9617-8EF61C4CDD
83)
kriSubmitQueryNodeData returning 0 with jobid->(1547106660327).
kriSubmitAck:hmc->(9.3.18.159),vios->(5AC3E0DC-13FC-4221-9617-8EF61C4CDD83)
[04] 01/23/19 T(397) _VMR 22:04:35.884611 DEBUG libkrest.c[9664]: kriSubmitAck
returning 0 with jobid->(1547106660328).
...
Subsequent to the identified kriSubmitQueryNodeData request, we see the expected

acknowledgment kriSubmitAck message. Also, just before the kriSubmitQueryNodeData
request, we see kriSubmitNeedAttn followed by a similar kriSubmitAck acknowledgment
message.
Up the stack at the ksys log level, we see in Example 2-106 the same sequence of actions in
the time interval around 22:04:35.57: getNeedAttn, getAcknowledge, getCAAevent, and
getAcknowledge. The condition for the sequence to happen appears to be the fact that all the
HMs get responses back a bit earlier, which is suggested by the VIOS_HM_RESP_DETECTED
events logged before the getNeedAttn call as an effect of processing the preceding
getQuickQuery.
Example 2-106 Sequencing of events and actions at the ksys log level
# more 24jan_ksys.log.nodeupcaaevttst.txt
...
[166C89EF-57D8-4E75-815C-AA5B885560B1] JobStatus: COMPLETED_OK, ReturnCode: 0
[5AC3E0DC-13FC-4221-9617-8EF61C4CDD83] JobStatus: COMPLETED_OK, ReturnCode: 0
[10] 01/23/19 T(9192) _VMR 22:04:14.019399 DEBUG VMR_SITE.C[6261]: INFO:
[10] 01/23/19 T(9192) _VMR 22:04:35.364432 DEBUG VMR_HMC.C[6827]: getNeedAttn
operation with opCode: 34(VMDR_ACK)
with HMC: rthmc3.
[10] 01/23/19 T(9192) _VMR 22:04:35.570806 DEBUG VMR_HMC.C[7050]: getAcknowledge

[10] 01/23/19 T(9192) _VMR 22:04:35.570813 DEBUG VMR_retry.C[345]: In doRetry
function, for opCode = 34(VMDR_ACK), rc = 0, retCode is 0, errstr is: ,retry flag
is 22
[10] 01/23/19 T(397) _VMR 22:04:35.773250 DEBUG VMR_HMC.C[7157]: getCAAevent
eventNotify entering. event:VIOS_NODE_FAILURE, event type:1, comp:VIOS,
eventNotify entering. event:NETWORK_INTERFACE_ACTIVE, event type:3, comp:VIOS,
eventNotify entering. event:VIOS_NODE_ACTIVE, event type:3, comp:VIOS,
[10] 01/23/19 T(397) _VMR 22:04:35.775796 DEBUG VMR_retry.C[1151]: Doing operation
with opCode: 34(VMDR_ACK)
with HMC: rthmc3.
function, for opCode = 34(VMDR_ACK), rc = 0, retCode is 0, errstr is: ,retry flag
is 22
24jan_ksys.log.nodeupcaaevttst.txt (99%)
Example 2-106 on page 142 shows how internal events are conveyed toward the event
notification subsystem of the KSYS itself. We see this mechanism acting in two distinct
instances:
򐂰 The first instance occurs after the getQuickQuery processing when the
VIOS_HM_RESP_DETECTED events, corresponding to start of the HM daemons we performed
manually, are passed by the eventNotify call to the KSYS event notification subsystem.
򐂰 In the second instance, three other events, VIOS_NODE_FAILURE,
NETWORK_INTERFACE_ACTIVE, and VIOS_NODE_ACTIVE, corresponding to the CAA events that
are generated by our manual VIOS node shutdown and restart, are passed to the KSYS
event notification subsystem by an eventNotify call, this time after a getCAAevent call.
Coming back to the VIOS level, we first correlate the HM daemon start moment with the
immediate preceding and subsequent VIO_HS_QUICK_QUERY requests entries in the
vioservice logs on the involved VIOS (Example 2-107).
Example 2-107 HM restart moment versus the preceding and subsequent VIO_HS_QUICK_QUERY
requests
# lsattr -El vios0|grep uuid
vios_uuid 5ac3e0dc-13fc-4221-9617-8ef61c4cdd83 VIOS Unique Identifier
False
# more /home/ios/logs/health/host_monitor.log.24jan.nodeupcaaevttst.txt
...
[2 13894106 64225577 01/20/19-09:12:28.187 main.C:main(int, char **)
: 190] HostMonitor has been shutdown.

[END 13894106 64225577 01/20/19-09:12:28.187
ViologWriter.C:ViologWriter::~ViologWriter() : 64] exited with rc=0
[START 24379748 72876383 01/23/19-20:58:44.569
ViologWriter.C:ViologWriter::ViologWriter(s...: 50]
[2 24379748 72876383 01/23/19-20:58:44.569 main.C:main(int, char **)
: 165] Log level set to: 3
host_monitor.log.24jan.nodeupcaaevttst.txt (99%)
# more /home/ios/logs/viosvc.log.24jan.nodeupcaaevttst.txt
...
[0 22610190 26214731 01/23/19-20:58:31.47 viosvc_res.c 1.26 464]
<quickQuery>
</quickQuery>
<quickQuery>
</quickQuery>
</Response></VIO>
]
rc=0
</VIO>
]
[0 22610194 58130901 01/23/19-20:58:52.790 viosvc_res.c 1.26 464]
<quickQuery>
</quickQuery>
<quickQuery>

</quickQuery>
</Response></VIO>
]
rc=0
viosvc.log.24jan.nodeupcaaevttst.txt (99%)
#
We notice in Example 2-107 on page 143 that all three HMs except the one on the VIOS node
started. Earlier, they were reported in the reply that was returned before the HM restart as
unresponsive since the same moment (hmResponsive="0" hmResponseSec='301561'), and as
responsive immediately after (hmResponsive="1" hmResponseSec='0'). We stopped and
started the HMs simultaneously by running the clcmd stopsrc -s ksys_hsmon (stop) and
clcmd startsrc -s ksys_hsmon -a "-l2" (start) commands, so the observed behavior is
consistent.
In the last VIO_HS_QUICK_QUERY of Example 2-107 on page 143, it is the first time since the
KSYS restart that the health library does not log any errors after retrieving the VIOS status
data from the HSDB, as shown in Example 2-108, for the same instances of the vioservice
command, 22610190 26214731 and 22610194 58130901, which were run before and after the
HM restart.
Example 2-108 Health library log around HM restart moment

# more /home/ios/logs/health/viohs.log.24jan.nodeupcaaevttst.txt
...
[START 22610190 26214731 01/23/19-20:58:30.984 ha_util.c 1.43 230]
cluster info from vioGetClusterInfoFromODM! cluster name='KSYS_rbRMHA_1'
id=43b050741b4111e9803e001018affe76
HDBC = 200176e8
HDBC = 200176e8
[3 22610190 26214731 01/23/19-20:58:31.39 hs_atten.c hs_get_all_vioses_for_MS
1.107 875] Error: ALERT:hmState=0 missCnt=301561
End, rc=0
[START 22610194 58130901 01/23/19-20:58:52.727 ha_util.c 1.43 230]
id=43b050741b4111e9803e001018affe76

HDBC = 200176e8
HDBC = 200176e8
End, rc=0
[START 22610200 60424487 01/23/19-20:59:14.96 ha_util.c 1.43 230]
viohs.log.24jan.nodeupcaaevttst.txt (99%)
#
The next VIO_HS_NEEDS_ATTENTION request is replied to by the health library with a tag
attribute value and is followed by an acknowledgment request, VIO_HS_ACKNOWLEDGE, from the
KSYS side with the same tag value, hsTag="-6630805255682309212" (Example 2-109). Then,
the sequence continues, as shown in the KSYS side logs, by the acknowledged
VIO_HS_QUERY_NODE_DATA request, with the hsTag="378617663756949861" (Example 2-110 on
page 147).
Example 2-109 Acknowledged VIO_HS_NEEDS_ATTENTION request

...
</VIO>
]
[0 22610200 60424487 01/23/19-20:59:14.173 viosvc_res.c 1.26 464]
<needsAtt hsTag="-6630805255682309212">
</needsAtt>
<needsAtt hsTag="-6630805255682309212">
<vmStatList>
<vmList machine_type="8286" model="42A" serial="21E0B2V">
</vmList>
</vmStatList>
</needsAtt>
<needsAtt hsTag="-6630805255682309212">
<vmStatList>
<vmList machine_type="8286" model="42A" serial="2100DEW">
<vmStat uuid="57275f1e-3680-4eef-945a-fe7670500310"><VM state="UNKNOWN">
<discoveryInfo type='UNMANAGED'/>
</VM></vmStat>
<vmStat uuid="57275f1e-3680-4eef-945a-fe7670500310"><VM state="UNKNOWN">
</VM></vmStat>

<vmStat uuid="31603d8c-5a2f-437c-ba33-f25321415cc4"><VM state="UNKNOWN">
</VM></vmStat>
<vmStat uuid="31603d8c-5a2f-437c-ba33-f25321415cc4"><VM state="UNKNOWN">
</VM></vmStat>
</vmList>
</vmStatList>
</needsAtt>
</Response></VIO>
]
rc=0
version="2.00" author="LIBKREST" title="acknowledgment to VIOS">
<Request action_str="VIO_HS_ACKNOWLEDGE" hsTag="-6630805255682309212"/>
</VIO>
]
[0 22610202 60424489 01/23/19-20:59:15.441 viosvc_res.c 1.26 464]
vio_response.result=[]
rc=0
The VIO_HS_NEEDS_ATTENTION response in Example 2-109 on page 146 contains VM status

elements and the element with the status for the VIOS node that underwent the CAA events
(hasData='1'). But, the status of the other three VIOSs is normal, so this response does not
contain any more status elements for them, as was the case in the previous
VIO_HS_QUICK_QUERY responses (Example 2-107 on page 143) or even in the previous
VIO_HS_NEEDS_ATTENTION response (Example 2-102 on page 139).
We reached the moment when the details about the CAA events are retrieved from the HSDB
and sent up the stack toward KSYS by the VIO_HS_QUERY_NODE_DATA request. The request is
acknowledged (Example 2-110).
Example 2-110 Acknowledged VIO_HS_QUERY_NODE_DATA request

...
version="2.00" author="LIBKREST" title="VIO_HS_QUERY_NODE_DATA">
<Request action_str="VIO_HS_QUERY_NODE_DATA"/>
</VIO>
]
[0 22610204 60424491 01/23/19-20:59:15.643 viosvc_res.c 1.26 464]
<nodeData hsTag="378617663756949861">

<nodeList machine_type="8286" model="42A" serial="21E0B2V">
<node uuid="315ce56b-9ba0-46a1-b4bc-da6108574e7e" event="BEGIN_EVENT_INFO
SEQUENCE_NUM=4
RC_FROM_EVPROD=0
BEGIN_EVPROD_INFO
EVENT_TYPE=NODE_DOWN
REASON=NODE_SHUTDOWN
NODE_NUMBER=4
END_EVPROD_INFO
END_EVENT_INFO
"/>
SEQUENCE_NUM=9
RC_FROM_EVPROD=0
BEGIN_EVPROD_INFO
EVENT_TYPE=ADAPTER_UP
INTERFACE_NAME=en0
NODE_NUMBER=4
END_EVPROD_INFO
END_EVENT_INFO
"/>
SEQUENCE_NUM=5
RC_FROM_EVPROD=0
BEGIN_EVPROD_INFO
EVENT_TYPE=NODE_UP
NODE_NUMBER=4
END_EVPROD_INFO
END_EVENT_INFO
"/>
</nodeList>
</nodeData>
</Response></VIO>
]
rc=0
version="2.00" author="LIBKREST" title="acknowledgment to VIOS">
<Request action_str="VIO_HS_ACKNOWLEDGE" hsTag="378617663756949861"/>

</VIO>
]
[0 22610206 60424493 01/23/19-20:59:15.854 viosvc_res.c 1.26 464]
vio_response.result=[]
rc=0
Following the VIO_HS_QUERY_NODE_DATA acknowledgment, the vios and trans tables look
“clean” like they did before this exercise. The last exercise output in Example 2-111 is
identical to the output in Example 2-93 on page 131.
Example 2-111 The vios and trans tables after VIO_HS_QUERY_NODE_DATA acknowledgment
--------------------------------------+-------+--------
5ac3e0dc-13fc-4221-9617-8ef61c4cdd83 | 1 | 0
4e0f6f60-214b-4d2c-a436-0eac46f7f71f | 1 | 0
166c89ef-57d8-4e75-815c-aa5b885560b1 | 1 | 0
315ce56b-9ba0-46a1-b4bc-da6108574e7e | 1 | 0
(4 rows)
vios=# select data, integer, txstarted from trans order by txstarted;
------+---------+-----------
(0 rows)
vios=#
This whole CAA event monitoring exercise presented the case where the event is happening
on a non-DBN node and the HSDB remains operational. Things become even more
complicated if the event affects the DBN node and the running SSP database crashes.
We used throughout this section various CAA, SSP, and PostgreSQL concepts and
commands. For extensive coverage of these topics, see the following resources:
򐂰 AIX Event Infrastructure for AIX and AIX clusters-AHAFS
򐂰 Cluster management
򐂰 IBM PowerVM Enhancements What is New in 2013, SG24-8198
򐂰 VIOS Shared Storage Pool 2.2.5 Enhancements
򐂰 VIOS Shared Storage Pool phase 6 - New Features
򐂰 PostgreSQL 10
Host monitor daemon

The last component in Figure 2-10 on page 115 we cover is the HM daemon. We mentioned it
in multiple places in previous sections, but here we focus on its inner structure and functions:
򐂰 HM activation
򐂰 HM threads
򐂰 HM startup sequence

HM activation
HM is configured as an SRC subsystem. Example 2-112 shows an already activated HM
instance, its ksys_hsmon SRC subsystem configuration details, and the fileset of the daemon
binary file.
Example 2-112 The ksys_hsmon SRC subsystem

# lssrc -a|grep ksys
ksys_hsmon 15401234 active
ksys_vmmon inoperative
# lssrc -S -s ksys_hsmon
#subsysname:synonym:cmdargs:path:uid:auditid:standin:standout:standerr:action:mult
i:contact:svrkey:svrmtype:priority:signorm:sigforce:display:waittime:grpname:
ksys_hsmon:::/usr/sbin/ksys_hsmond:0:0:/dev/console:/dev/console:/dev/console:-R:-
Q:-S:0:0:20:15:9:-d:10::
# lslpp -w /usr/sbin/ksys_hsmond
File Fileset Type
----------------------------------------------------------------------------
/usr/sbin/ksys_hsmond ksys.hsmon.rte File
#
The -R action attribute value restarts automatically the ksys_hsmon subsystem if it stops
abnormally. The ksys_hsmon subsystem activates when the first discovery for the involved
host group occurs. Example 2-113 shows some related excerpts from KSYS trace files that
were collected shortly after discovery.
Example 2-113 Host monitor in the host group discovery trace files
# ksysmgr discover hg rbHG
# ksysmgr trace log=ksys > ksys.log.HGdiscov; ksysmgr trace log=krest >
krest.log.HGdiscov; ksysmgr trace log=krestlong > krestlong.log.HGdiscov
# ls
krest.log.HGdiscov krestlong.log.HGdiscov ksys.log.HGdiscov
# grep hsmon *
krestlong.log.HGdiscov: <ParameterValue kb="CUR" kxe="false">0513-059 The
ksys_hsmon Subsystem has been started. Subsystem PID is 22086114.
krestlong.log.HGdiscov:[04] 04/26/19 T(91b5) _VMR 18:32:33.015960 DEBUG
libkrest.c[2558]: job_result->result : 0513-059 The ksys_hsmon Subsystem has been
started. Subsystem PID is 22086114.
krestlong.log.HGdiscov: <ParameterValue kb="CUR" kxe="false">0513-059 The
ksys_hsmon Subsystem has been started. Subsystem PID is 16122210.
krestlong.log.HGdiscov:[04] 04/26/19 T(91b5) _VMR 18:32:38.325017 DEBUG
libkrest.c[2558]: job_result->result : 0513-059 The ksys_hsmon Subsystem has been
started. Subsystem PID is 16122210.
ksys.log.HGdiscov:[02] 04/26/19 T(91b5) _VMR 18:32:33.016320DEBUG VMR_HMC.C[7355]:
jresultP->result: 0513-059 The ksys_hsmon Subsystem has been started. Subsystem
PID is 22086114.
ksys.log.HGdiscov:[02] 04/26/19 T(91b5) _VMR 18:32:38.325368DEBUG VMR_HMC.C[7355]:
jresultP->result: 0513-059 The ksys_hsmon Subsystem has been started. Subsystem
PID is 16122210.
# grep -p 18:32:33.015960 krestlong.log.HGdiscov
[04] 04/26/19 T(91b5) _VMR 18:32:33.015958 DEBUG libkrest.c[2476]: <StdOut>
[04] 04/26/19 T(91b5) _VMR 18:32:33.015960 DEBUG libkrest.c[2558]:
job_result->result : 0513-059 The ksys_hsmon Subsystem has been started. Subsystem
PID is 22086114.
<?xml version="1.0"?>

<VIO
version="2.00"
title="VIOS ADD_MS"
published="2019-04-26T01:17:00Z"
author="IBM Power Systems VIOS"
>
<Response>
</Response>
</VIO>
#
The host group that is used for this analysis consists of two frames, each of which has two
VIOS LPARs. The trace files contain entries about the activation of only two ksys_hsmon
subsystems even though there are four VIOSs. In the filtered krestlong trace file, the output
of the last grep command (Example 2-113 on page 150) shows the activation as part of a
VIOS ADD_MS pass-through API request. By checking the logged PIDs at the VIOS level, we
see that the selected VIOSs that are used by KSYS for the ADD_MS requests are
usaxvib053ccpx1 and usaxvib063ccpx1 (one is the designated VIOS for each host)
(Example 2-114).
Example 2-114 Activated ksys_hsmon subsystems

# uname -uL
IBM,0221282FW 2 usaxvia053ccpx1
# /usr/ios/cli/ioscli cluster -status -fmt ',' -field node_name node_mtm
usaxvib063ccpx1,8408-E8E02212424W
usaxvia063ccpx1,8408-E8E02212424W
usaxvib053ccpx1,8408-E8E0221282FW
usaxvia053ccpx1,8408-E8E0221282FW
# clcmd lssrc -s ksys_hsmon
-------------------------------
NODE usaxvia053ccpx1
-------------------------------
-------------------------------
NODE usaxvib053ccpx1
-------------------------------
-------------------------------
NODE usaxvia063ccpx1
-------------------------------
-------------------------------
NODE usaxvib063ccpx1
-------------------------------
#

Checking for the VIOS_ADD_MS request in the vioservice command log on usaxvib053ccpx1,
we identify the involved PID to TID vioservice thread instance, as shown in Example 2-115.
Example 2-115 VIOS_ADD_MS call in the vioservice log

# uname -uL
IBM,0221282FW 3 usaxvib053ccpx1
# alog -f /home/ios/logs/viosvc.log -o > viosvc.log.postHGdiscovery
# more viosvc.log.postHGdiscovery
...
version="2.00" author="LIBKREST" title="Add Managed System">
<Request action_str="VIO_HS_ADD_MS" hsTag="0">
<addMS>
<viosList machine_type="8408" model="E8E" serial="21282FW">
<vios uuid="72dee902-1210-4bd7-a35f-3a6c771c6453" isHM="true" partNum="3"
macAddr="16DC6A23650F"/>
<vios uuid="10875b47-d737-44f9-a745-554f4df4adf8" isHM="true" partNum="2"
macAddr="16DC662CB90E"/>
</viosList>
</addMS>
</Request>
</VIO>
]
[0 30474542 72089921 04/26/19-18:32:28.193 viosvc_res.c 1.26 464]
vio_response.result=[<?xml version="1.0"?>
version="2.00"
title="VIOS ADD_MS"
published="2019-04-26T01:17:00Z"
>
<Response>
</Response>
</VIO>
]
rc=0
viosvc.log.postHGdiscovery (99%)
Then, we filter the trace of this thread instance in the health library log, which contains the
expected SRC subsystem activation command. Example 2-116 shows this trace with the
vioHsAddMs health library API call.
Example 2-116 The vioHsAddMs health library API call

# alog -f /home/ios/logs/health/viohs.log -o > viohs.log.postHGdiscovery
# grep "30474542 72089921" viohs.log.postHGdiscovery
[START 30474542 72089921 04/26/19-18:32:27.830 ha_util.c 1.43 230]
id=e4c3993266fd11e9801efa242424f21c

[3 30474542 72089921 04/26/19-18:32:27.831 hs_ms.c hsParseAddMsInputs 1.83 1758]
Found '1' elements for '/VIO/Request/addMS'.
Found '1' elements for '/VIO/Request/addMS/viosList'.
[3 30474542 72089921 04/26/19-18:32:27.831 hs_ms.c hsParseSysProperties 1.83 1850]
Input managed system machine_type is '8408'
Input managed system model is 'E8E'
Input managed system serial is '21282FW'
Found '2' elements for '/VIO/Request/addMS/viosList/vios'.
[3 30474542 72089921 04/26/19-18:32:27.832 hs_ms.c hsParseViosList 1.83 1650]
Found '2' elements for '/VIO/Request/addMS/viosList/vios'.
HDBC = 2001a758
[3 30474542 72089921 04/26/19-18:32:27.859 hs_ms.c hsCurMsPendop 1.83 1429] MS
exists=0, pendop=0, type=8408, model=E8E, serial=21282FW
HDBC = 2001a758
[3 30474542 72089921 04/26/19-18:32:27.876 hs_ms.c doHsPreConfigurations 1.83
2094] sspMTM='8408-E8E0221282FW', ksysMTM='8408-E8E21282FW'
[3 30474542 72089921 04/26/19-18:32:27.879 hs_ms.c handleHsSchema 1.83 2022]
Health schema exists already.
HEALTH:72089921 -- violibGetHmVer.c hmParseVerFile 1.3 264 Content:
2,0,2,0,5
[3 30474542 72089921 04/26/19-18:32:28.84 hs_ms.c startHsmon 1.83 318]
cmd=/usr/bin/startsrc -s ksys_hsmon -a '-l3', rc=0, errno=0
[3 30474542 72089921 04/26/19-18:32:28.84 hs_ms.c startHsmon 1.83 374] ksys_hsmon
daemon started.
#
On the partner VIOS of the same host, usaxvib053ccpx1, we obtain a different health library
sequence of records. As shown in Example 2-117, this time the action is performed by a
vio_daemon thread.
Example 2-117 The ksys_hsmon call started by vio_daemon on the partner VIOS
# uname -uL
# alog -f /home/ios/logs/health/viohs.log -o > viohs.log.postHGdiscovery
# grep ksys_hsmon viohs.log.postHGdiscovery
daemon started.
# grep "8323398 20971909" viohs.log.postHGdiscovery
[START 8323398 20971909 04/26/19-18:32:27.909 ha_util.c 1.43 230]

id=e4c3993266fd11e9801efa242424f21c
[3 8323398 20971909 04/26/19-18:32:27.910 hs_msg.c vioHsMsg 1.46 84] Received
ADD_HM message
HDBC = 110538fd0
HEALTH:20971909 -- violibGetHmVer.c hmParseVerFile 1.3 264 Content:
2,0,2,0,5
daemon started.
# ps 8323398
PID TTY STAT TIME COMMAND
8323398 - A 2:34 /usr/sbin/vio_daemon -d 4
# procstack 8323398
...
---------- tid# 20971909 (pthread ID: 4884) ----------
0x09000000005b4854 _event_sleep(??, ??, ??, ??, ??, ??) + 0x514
0x09000000005b54f0 _event_wait(??, ??) + 0x350
0x09000000005c485c _cond_wait_local(??, ??, ??) + 0x4fc
0x09000000005c4df4 _cond_wait(??, ??, ??) + 0x34
0x09000000005c5828 pthread_cond_wait(??, ??) + 0x1a8
0x000000010004d274 viod_ke_prcs_thread(??) + 0x254
0x090000000059ffe8 _pthread_body(??) + 0xe8
...
#
Looking at the vio_daemon log around 18:32:27.909 - 18:32:28.130 interval, we see specific
messages that are logged by the thread with tid# 20971909 (Example 2-118).
Example 2-118 ADD_HM message processing at the vSCSI kernel extension layer on the partner
VIOS
# cp /home/ios/logs/viod.log.bkp viod.log.postHGdiscovery.bkp
# more viod.log.postHGdiscovery.bkp
...
Apr 26 2019, 18:32:26.844 0x2124 viod_misc_handler.c
viod_start_cache_timer 1.3 72 TRC -- Setting up the miscellaneous
timers 11008abd0
Apr 26 2019, 18:32:27.909 0xf10 viod_ke.c viod_ke_recv_thread 1.81
1085 TRC -- VIOKE Recvd pkt
1111 TRC -- VIOKE Recvd pkt pyld 64
1153 TRC -- VIODKE enqueue msg 110089e90
Apr 26 2019, 18:32:27.909 0x1314 viod_ke.c viod_ke_prcs_thread
1.81 1270 TRC -- VIODKE condition is true
1.81 1284 TRC -- VIODKE dequeue msg 0x110089e90
Apr 26 2019, 18:32:27.909 0x1314 viod_protocol.c viodDoReq
1.28 52 TRC -- viodDoReq: Enter

1.28 53 INF -- opcode: 0x1002d length: 64 Req type =2
1.28 57 INF -- Start: cluster_id: e4c3993266fd11e9801efa242424f21c
collater: 48534d540000000
1 trans_id: 0
1.28 62 TRC -- 65581 Operation in VIOD_KE_HS_MSG event
Apr 26 2019, 18:32:27.909 0x1314 viod_hs.c viod_dispatch_hs_msg
1.19 106 TRC -- BEGIN
Apr 26 2019, 18:32:28.131 0x1314 viod_hs.c viod_dispatch_hs_msg
1.19 127 TRC -- END
1.28 161 INF -- End: cluster_id: e4c3993266fd11e9801efa242424f21c
collater: 48534d5400000001
trans_id: 0
Apr 26 2019, 18:32:28.131 0x1314 viod_ke.c viod_kefree_msg
1.81 224 TRC -- free msg ptr : 110089e90
1.81 1295 TRC -- Completed the VIOKE req
viod.log.postHGdiscovery.bkp (82%)
The user space pthread ID 4884 in decimal is 0x1314 in hexadecimal. So, we deduce that the
partner VIOS receives the ADD_HM message at the health library layer from the designated
VIOS on the same frame through the virtual Small Computer System Interface (vSCSI) kernel
extension (VKE) cluster communication layer and local viod_daemon. Matching the time
stamps, apparently the received message is processed inside the viod_dispatch_hs_msg call
of the 0x1314 viod_daemon thread. For more information about the VIOS VKE and its related
cluster communication protocol, see Communication protocol for virtual input/output server
(VIOS) cluster communication.
Now that we clarified the way that HM daemon activates on each VIOS, let us examine its
thread structure after its activation.
HM threads
Example 2-119 shows the threads of the /usr/sbin/ksys_hsmond monitor daemon (PID
24445232). Each one is identified by a kernel TID value and a user space pTID value. We are
interested in the pTID value because it shows up in various logs that we check for problem
determination and other purposes.
Example 2-119 Kernel threads of the ksys_hsmond daemon

# proctree -t -p 24445232
TID : 14418391 (pTID : 1)
24445232 /usr/sbin/ksys_hsmond -l3
TID : 70189439 (pTID : 1)
TID : 47186265 (pTID : 2057)
TID : 67437005 (pTID : 1800)
TID : 69140847 (pTID : 1543)
TID : 70123875 (pTID : 1286)
TID : 15204793 (pTID : 1029)
TID : 65863959 (pTID : 772)
TID : 57934171 (pTID : 515)
TID : 20578667 (pTID : 258)
#

Examining the thread stacks might give us hints about the functions that are intended for each
separate thread. In the process stack snapshot that is shown in Example 2-120, we removed
some calls at the top of each thread stack to make the output a bit shorter and more readable.
Example 2-120 Host monitor threads and associated stacks

24445232: /usr/sbin/ksys_hsmond -l3
---------- tid# 70189439 (pthread ID: 1) ----------
...
0x100013c8 main(0x2, 0x2ff22d60) + 0xc88
0x100001b8 __start() + 0x68
---------- tid# 47186265 (pthread ID: 2057) ----------
...
0x1012df3c UnixSocket::threadMain()(0x30029640) + 0x31c
0x10129a60 Runnable::runThread(void*)(0x30029640) + 0x40
---------- tid# 67437005 (pthread ID: 1800) ----------
...
0x10111734 Processing::threadMain()(0x30025710) + 0x314
0x10129a60 Runnable::runThread(void*)(0x30025710) + 0x40
---------- tid# 69140847 (pthread ID: 1543) ----------
...
0x100eb3d0 ProcessErrHandling::threadMain()(0x30025ab4) + 0x30
0x10129a60 Runnable::runThread(void*)(0x30025ab4) + 0x40
---------- tid# 70123875 (pthread ID: 1286) ----------
...
0x100ec798 ProcessingDbHb::threadMain()(0x30025a3c) + 0x338
0x10129a60 Runnable::runThread(void*)(0x30025a3c) + 0x40
---------- tid# 15204793 (pthread ID: 1029) ----------
...
0x10074f14 HvncpSender::threadMain()(0x30020d4c) + 0x2a54
0x10129a60 Runnable::runThread(void*)(0x30020d4c) + 0x40
---------- tid# 65863959 (pthread ID: 772) ----------
...
0x10080a6c HvncpReceiver::threadMain()(0x30020b84) + 0x312c
0x10129a60 Runnable::runThread(void*)(0x30020b84) + 0x40
---------- tid# 57934171 (pthread ID: 515) ----------
...
0x100f37bc VlanCommunication::threadMain()(0x30020b50) + 0x3c
0x10129a60 Runnable::runThread(void*)(0x30020b50) + 0x40
---------- tid# 20578667 (pthread ID: 258) ----------
...
0x100b3928 ProcessingMsg::threadMain()(0x30025cec) + 0x1e8
0x10129a60 Runnable::runThread(void*)(0x30025cec) + 0x40
#

In Example 2-120 on page 156, you can easily identify the expected initial thread of the
process (pthread ID 1) running the main() function. Then, you see that a specific class
method, <class>::threadMain(), was launched for each of the remaining threads. The class
names in the filtered list that is shown in Example 2-121 are suggestive of the purpose of
each thread.
Example 2-121 Main thread calls of ksys_hsmond daemon

# procstack 24445232|grep -E "main|threadMain"
0x100013c8 main(0x2, 0x2ff22d60) + 0xc88
#
Table 2-4 describes each thread type that is observed. In the first column, there are some
abbreviations for further use.
Table 2-4 Threads of ksys_hsmond daemon

Thread type Class name Description
Initial process thread N/A Initial thread of the process, pTID=1, running
main() function call.
UnixSocket UnixSocket Gets the commands from the VIOS side and does
the related processing.
VM_HB Processing Processes the VM heartbeat time stamp and

does the HSDB update in case there is a miss.
ErrorHandler ProcessErrHandling Does recovery from errors.
HM_HB ProcessingDbHb Handles heartbeats from HM to HSDB.
HvncpSender HvncpSender Sends messages over the HM VMM Network

Communication Protocol (HVNCP) layer and over
the trunk adapter to VM Agents.
HvncpReceiver HvncpReceiver Extracts HVNCP packets from the receiving

queue and assembles messages from them
(including VM heartbeat messages). Messages
are passed to other threads for further
processing.
VlanCommunication VlanCommunication Receives frames coming on the trunk adapter

from VM Agents and queues them to the receiving
queue.
ProcessingMsg ProcessingMsg Dedicated thread to handle AR messages.

Figure 2-11 shows the HM threads mentioned so far with the components around the HM with
which is interacts. Some communication flows are also suggested by the lines and arrows in
the figure.
Figure 2-11 Host monitor interaction with components
The main job of the HM is to monitor the health of the VMs and to update their status in the
HSDB. The Processing thread of the HM receives heartbeat messages that originated at the
VM level, and updates the VM status in the HSDB. The VM heartbeat path is shown by the
red arrows with “HB” labels coming from VM, through the HM, to the SSP HSDB. Another
heartbeating mechanism is used to provide status updates about the HM itself, as shown by
the red arrow from the ProcessingDbHb thread to the SSP HSDB.
The HM also acts as a gateway between the VM and KSYS in the AR Messaging protocol.
The protocol supports the transfer of the application status and configuration updates from
VM to the KSYS side. The communication flow for this data transfer is represented by a green
line with arrows at its ends in Figure 2-11. The flow is a sequence of subflows that are listed
here in the order they occur:
Door Bell (dbell) Short notification message from the VMM to the KSYS that signals
that application changes happened.
Put Message Request & Response (PMSG_REQ, PMSG_RSP)
When DBELL is received at the KSYS side, a request message is sent
by KSYS to the VMM asking for details. VMM replies with a response
message containing the requested details.
Get Message (GMSG)
A confirmation is expected by VMM that the Put Message response
reached the KSYS. After the confirmation is received, the sent payload
information is discarded from the local AR cache on the VM side.
Clear DB A subflow of messages between the VMM on one side and HM and
HSDB on the other side that ensures that any entry that is related to
the pending AR Messaging cycle is cleared out from the HSDB. It
closes the whole cycle that is initiated by dbell.
For more information about the AR Messaging protocol, see 2.3.3, “Application Reporting
flow” on page 262.

On the VMM side, the HM exchanges the messages over the HVNCP layer. HVNCP packets
are encapsulated in Ethernet frames that cross the hypervisor switch. The HVNCP packets
are handled on the HM side by the communication threads - VlanCommunication,
HvncpSender, and HvncpReceiver. For more information about the HVNCP protocol, see 2.2.9,
“HVNCP protocol” on page 199.
On the VIOS side, the HM updates the heartbeat and messaging details that are received
from the VMs in the HSDB through the callback APIs that are described in “Health library” on
page 115. HM handles its own heartbeat updates in a similar fashion. KSYS retrieves these
updates from HSDB through periodic polling that is performed by the HMC and pass-through
REST APIs. Communication from KSYS toward a target VM goes through the HMC and
pass-through APIs to the VIOS RMC level. The VIOS hands commands from KSYS to the HM
by using the UNIX socket of the UnixSocket thread. Based on the type of the received
command, the UnixSocket thread cooperates with other HM threads, which perform related
processing and forward specific messages to the target VM by using HVNCP.
HM startup sequence
Example 2-122 shows the initial 10 records that are logged by the HM daemon instance with
PID 24445232 when it is activated on VIOS usaxvia053ccpx1.
Example 2-122 HM daemon startup sequence: First 10 log entries

# uname -uL
# lssrc -s ksys_hsmon
# alog -f /home/ios/logs/health/host_monitor.log -o > hm.log.postHGdiscovery
# cat hm.log.postHGdiscovery|sed '/24445232/,$!d' > hm.24445232.log.postHGdiscovery
# cat hm.24445232.log.postHGdiscovery|wc -l
145
# head hm.24445232.log.postHGdiscovery
[START 24445232 70189439 04/26/19-18:32:28.106 ViologWriter.C:ViologWriter::ViologWriter(s...:
50]
[2 24445232 70189439 04/26/19-18:32:28.106 main.C:main(int, char **) :
165] Log level set to: 3
[2 24445232 70189439 04/26/19-18:32:28.106 DatabaseAccess.C:DatabaseAccess::DatabaseAccess...:
68] INFO: getting a transaction handle...
74] Host Monitor MAC address=16DC662CB90E
79] After vioCreateHmTrans, Upper version: 2.0 Lower version: 2.0
84] Inside DBAccess: After vioCreateHmTrans(): Set Daemon log level to 3
[3 24445232 70189439 04/26/19-18:32:28.318 HmUtils.C:hmutils::getNetIntf(string) :
29] BEGIN macAddressSearched=16DC662CB90E
44] Found 14 Network Device Drivers.
64] Found interface 'ent22' that matches mac address '16DC662CB90E'
106] +HostMonitor information
#

The HM daemon log file is host_monitor.log in the /home/ios/logs/health directory. We see
that the typical log entry row has in its first three fields, from left to right, the current log level,
the PID, and the kernel TID of the thread generating the entry. Then, the time stamp of the
entry, the source code file name, and the function being run follow. Then, there might be a “:”
separator, the source code line number of the executed instruction, and a detailed message
describing the performed action and resulting output.
Now that we have these syntax details, we can parse the log entries that are shown in
Example 2-122 on page 159. The main thread, with TID 70189439, generates most of the log
entries. After the main function call, the thread sets the log level to 3 and obtains a transaction
handle for HSDB access. The MAC address 16DC662CB90E value of the VEPA trunk adapter is
retrieved from the HSDB where it was stored during the previous host group discovery stage.
This MAC address value is used to identify the trunk adapter logical device, which is ent22
here.
Transaction handles are further obtained for the rest of HM daemon threads that are going to
access the HSDB, which are Receiver thread calls, Processing thread calls, HM
heartbeating, Error Handler, and messaging, as shown in Example 2-123.
Example 2-123 HM daemon startup sequence: HM identifier generation

# sed '11,$!d;=' hm.24445232.log.postHGdiscovery |sed 'N;s/\n/:/'|head -20
11:[2 24445232 70189439 04/26/19-18:32:28.319
DatabaseAccess.C:DatabaseAccess::DatabaseAccess...: 107] | UUID :
10875b47-d737-44f9-a745-554f4df4adf8
12:[2 24445232 70189439 04/26/19-18:32:28.319
DatabaseAccess.C:DatabaseAccess::DatabaseAccess...: 108] | Network Adapter : ent22
13:[2 24445232 70189439 04/26/19-18:32:28.319
DatabaseAccess.C:DatabaseAccess::DatabaseAccess...: 109] | MacAddr : 16DC662CB90E
14:[2 24445232 70189439 04/26/19-18:32:28.319
DatabaseAccess.C:DatabaseAccess::DatabaseAccess...: 110] +
15:[2 24445232 70189439 04/26/19-18:32:28.319
DatabaseAccess.C:DatabaseAccess::DatabaseAccess...: 112] INFO: getting a transaction handle for
Receiver thread calls...
16:[2 24445232 70189439 04/26/19-18:32:28.400
DatabaseAccess.C:DatabaseAccess::DatabaseAccess...: 118] INFO: getting a transaction handle for
Processing thread calls...
17:[2 24445232 70189439 04/26/19-18:32:28.480 ProcessingDbHb.C:ProcessingDbHb::ProcessingDbHb()
: 43] INFO: getting a transaction handle for HM hearbeating ...
18:[3 24445232 70189439 04/26/19-18:32:28.560 ProcessingDbHb.C:ProcessingDbHb::ProcessingDbHb()
: 53] After vioCreateHmTrans, Upper version: 2.0 Lower version: 2.0
19:[2 24445232 70189439 04/26/19-18:32:28.560
ProcessingDbHb.C:ProcessErrHandling::ProcessErr...: 150] INFO: getting a transaction handle for
Error Handler ...
20:[3 24445232 70189439 04/26/19-18:32:28.638
ProcessingDbHb.C:ProcessErrHandling::ProcessErr...: 162] After vioCreateHmTrans, Upper version:
2.0 Lower version: 2.0
21:[2 24445232 70189439 04/26/19-18:32:28.638 DatabaseAccess.C:DatabaseAccess::DatabaseAccess()
: 146] INFO: getting a transaction handle for messaging ...
22:[2 24445232 70189439 04/26/19-18:32:28.719 Processing.C:ProcessingMsg::ProcessingMsg()
: 1293] INFO: In ProcessingMsg class constructor.
23:[2 24445232 70189439 04/26/19-18:32:28.719 Processing.C:Processing::Processing(string &)
: 104] HM version range: 2.0 - 2.0
24:[3 24445232 70189439 04/26/19-18:32:28.719
DatabaseAccess.C:DatabaseAccess::getNegotiatedR...: 196] Upper version: 2.0 Lower version: 2.0

25:[2 24445232 70189439 04/26/19-18:32:28.719
VmHealthStatusMgr.C:VmHealthStatusMgr::setSuppo...: 690] Setting VM supported versions. 2.0 -
2.0
26:[2 24445232 70189439 04/26/19-18:32:28.719 Processing.C:Processing::updateHmRange(bool)
: 187] Operating HM version: 2.0
27:[2 24445232 70189439 04/26/19-18:32:28.719 Processing.C:Processing::updateHmRange(bool)
: 227] HostMonitor identifier updated to '1556303548.719328000.38', a 'Hello' message will be
broadcasted every 30.00 seconds.
: 109] updateHmRange(true) errorDetected=0
: 133] INFO: In processing class constructor.
30:[2 24445232 20578667 04/26/19-18:32:28.719 Processing.C:ProcessingMsg::threadMain()
: 1368] INFO: Running thread ProcessingMsg.
#
Among the various initializations that are performed, note the update of the HM identifier in
row 27 and the related message, HostMonitor identifier updated to
'1556303548.719328000.38', a 'Hello' message will be broadcast every 30.00 seconds.
A unique identifier for each HM daemon instance is generated based on the VIOS system
clock. It changes at any subsequent HM restart so that the components around it become
aware of the new HM instance incarnation and act.
The last entry in Example 2-123 on page 160 is logged by the ProcessingMsg thread, TID
20578667, which announces that it created.
A socket is further configured and bound to the trunk VEPA interface that was identified
earlier by the MAC address, as shown in Example 2-124.
Example 2-124 HM daemon startup sequence: HVNCP socket bind

31:[3 24445232 70189439 04/26/19-18:32:28.719
ethernet.C:configureEthernetSocketAddr(socketHa...: 255] +Socket '3' parameters
(SNDD_8022)
32:[3 24445232 70189439 04/26/19-18:32:28.719
ethernet.C:configureEthernetSocketAddr(socketHa...: 256] | family : 23
33:[3 24445232 70189439 04/26/19-18:32:28.719
ethernet.C:configureEthernetSocketAddr(socketHa...: 257] | len : 36
34:[3 24445232 70189439 04/26/19-18:32:28.719
ethernet.C:configureEthernetSocketAddr(socketHa...: 258] | filtertype : 4
35:[3 24445232 70189439 04/26/19-18:32:28.719
ethernet.C:configureEthernetSocketAddr(socketHa...: 259] | ethertype : 1536
36:[3 24445232 70189439 04/26/19-18:32:28.719
ethernet.C:configureEthernetSocketAddr(socketHa...: 260] | filterlen : 12
37:[3 24445232 70189439 04/26/19-18:32:28.720
ethernet.C:configureEthernetSocketAddr(socketHa...: 261] | nddname : ent22
38:[3 24445232 70189439 04/26/19-18:32:28.720
ethernet.C:configureEthernetSocketAddr(socketHa...: 262] +
39:[3 24445232 70189439 04/26/19-18:32:28.720
ethernet.C:configureEthernetSocketAddr(socketHa...: 304] Found interface 'ent22' :
macAddr '16DC662CB90E'.
40:[3 24445232 70189439 04/26/19-18:32:28.825 HostMonitor.C:HostMonitor::HostMonitor()
: 68] Connected to the network interface 'ent22'.
: 74] Network controller started.

: 80] HostMonitor thread started.
43:[2 24445232 67437005 04/26/19-18:32:28.856
DatabaseAccess.C:DatabaseAccess::restartVms(vec...: 753] vioHmRestartVMs: VMs to
Restart=0, rc=0
44:[3 24445232 67437005 04/26/19-18:32:28.856 Processing.C:Processing::sendMessage(bool,
stri...: 682] Broadcasting 'Hello' message with identifier '1556303548.719328000.38'
45:[3 24445232 67437005 04/26/19-18:32:28.856
HvncpSender.C:HvncpSender::addCommTableEntry(co...: 300] Sender opened communication
table with macAddr 'FFFFFFFFFFFF'.
46:[3 24445232 67437005 04/26/19-18:32:28.856
HvncpReceiver.C:HvncpReceiver::linkCommTables(c...: 271] Receiver opened communication
table with macAddr 'FFFFFFFFFFFF'.
47:[3 24445232 15204793 04/26/19-18:32:28.856 HvncpSender.C:HvncpSender::threadMain()
: 814] Sending Packet to addr: FFFFFFFFFFFF
48:[3 24445232 15204793 04/26/19-18:32:28.857 HvncpPacket.C:HvncpPacket::display() const
: 292] +Packet Frame (23 bytes) 1556303548.719328000.38
49:[3 24445232 67437005 04/26/19-18:32:29.334 Processing.C:Processing::cleanMsgBuffer(const
s...: 833] Removing message to VmMon at macAddr 'FFFFFFFFFFFF' from send queue. Reason:
Fully sent
#
The Processing thread (TID 67437005) that is handling VM heartbeats, and the HvncpSender
thread (TID15204793) that is dedicated to sending HVNCP packets over the trunk interface,
get ready to communicate with the VM Agents around them. The first Hello broadcast
message is sent at this stage, as logged in rows 44-49 of the excerpt in Example 2-124 on
page 161. Row 50 and the subsequent entries in the HM log file are mostly recording
repeated Hello broadcast messages once every 30 seconds, as shown in Example 2-125.
Example 2-125 HM daemon startup sequence: Repeated Hello broadcast messages

53:[3 24445232 70123875 04/26/19-18:32:59.77 ProcessingDbHb.C:ProcessingDbHb::threadMain() :
86] HM: Sending HM heartbeats to health library
Fully sent
# sed '51,$!d;=' hm.24445232.log.postHGdiscovery |sed 'N;s/\n/:/'|tail
136:[3 24445232 70123875 04/26/19-18:41:18.386 ProcessingDbHb.C:ProcessingDbHb::threadMain()
: 86] HM: Sending HM heartbeats to health library

Fully sent
141:[3 24445232 70123875 04/26/19-18:41:49.600 ProcessingDbHb.C:ProcessingDbHb::threadMain()
: 86] HM: Sending HM heartbeats to health library
Fully sent
#
For HM daemon startup sequence scenario, we did not install the VM Agent on any of the
managed VMs, which explains why the HM does not receive any messages at from the VM
side to its repeated broadcast. A complete HM to VMM communication establishment flow,
where an active agent reacts on the VM side to a received Hello message, is described in
“HM-VMM communication establishment” on page 215.
Example 2-125 on page 162 also contains traces of the ProcessingDbHb thread (TID
70123875), which informs us about the HM heartbeat that was sent to HSDB. Checking for
their occurrence, we see that they show up every 30 seconds, as shown in Example 2-126.
Example 2-126 Host monitor heartbeat records

# grep 70123875 hm.24445232.log.postHGdiscovery|head -3
[3 24445232 70123875 04/26/19-18:32:59.77 ProcessingDbHb.C:ProcessingDbHb::threadMain() :
# grep 70123875 hm.24445232.log.postHGdiscovery|tail -3
#

2.2.8 VM Agent architecture
The VM Agent plays a key role in two of the use cases that are described in 2.1.2,
“Capabilities” on page 41. Here is a review of them:
VM monitoring The health of the VM is tracked by a heartbeating function that
consists of sending a periodic short status update from the VM
side to the HM on the VIOS side. HM passes this update to the
VIOS HSDB, and KSYS puts it through periodic polling and
processes it for a possible VM relocation decision.
Application monitoring The application monitoring framework provides basic application
monitoring functions for a registered application. The application
restarts locally if monitoring fails. After a limited number of failed
restart cycles, the application remains in permanent failure state.
If the application is registered as critical to the framework, then
the recovery action continues.
After the application reaches a permanent failure state, the KSYS

controller is notified and takes control. It tries to restart locally the
VM running the application, and then lets application monitoring
act on its own. If a permanent failure state is reached again, the
local VM restart cycle is repeated for a limited number of times.
When the VM local restart limit is reached, the KSYS controller
considers the relocation of the VM on a neighbor host on the host
group according to enabled policies.
We describe the structure and functions of the VM Agent in two parts:

򐂰 VM Agent activation
򐂰 Application Management Engine
Section “VM Agent activation” describes basic common aspects like VMM daemon
initialization, threads and other local components, and VMM to HM communication
establishment. It also details the activation of the VMM subsystem that is responsible for the
heartbeat with the HMs.
Section “Application Management Engine” on page 176 describes the specific aspects of the
Application Management Framework subsystem.
VM Agent activation
The initial installation of the VM Agent fileset creates the ksys_vmm SRC subsystem so that a
VMM daemon starts automatically at any subsequent AIX restart. Example 2-127 shows
some details about the VMM configuration as an SRC subsystem. We also mention the
location of the related binary, configuration, and log files that were the result of the fileset
installation.
Example 2-127 The ksys_vmm SRC subsystem

# oslevel -s
7200-03-02-1845
# ls /var/ksys
ls: 0653-341 The file /var/ksys does not exist.
# installp -acFXYd . ksys.vmmon.rte
...
Installation Summary
--------------------

Name Level Part Event Result
-------------------------------------------------------------------------------
ksys.vmmon.rte 1.3.0.1 USR APPLY SUCCESS
ksys.vmmon.rte 1.3.0.1 ROOT APPLY SUCCESS
ksys_vmm 14549402 active
# lssrc -S -s ksys_vmm
#subsysname:synonym:cmdargs:path:uid:auditid:standin:standout:standerr:action:mult
i:contact:svrkey:svrmtype:priority:signorm:sigforce:display:waittime:grpname:
ksys_vmm:::/usr/sbin/ksys_vmmd:0:0:/dev/console:/dev/console:/dev/console:-R:-Q:-S
:0:0:20:15:9:-d:10::
# lslpp -w /usr/sbin/ksys_vmmd
File Fileset Type
----------------------------------------------------------------------------
/usr/sbin/ksys_vmmd ksys.vmmon.rte File
# ls /var/ksys
config log tmp
# ls /var/ksys/config
HAMonitoring.xml
# ls /var/ksys/log
aix_vmmke.log ksys_vmm.debug ksys_vmm_vmH.debug
errpt.log ksys_vmm.log ksys_vmm_vmH.log
interfaceList.txt ksys_vmm_ame.debug ksysvmmgr.critlog
ksys_vmm.critlog ksys_vmm_ame.log ksysvmmgr.log
#
To get insights about the daemon’s thread structure, we examine a simpler case where the
AIX VM has monitoring disabled at the KSYS level, as shown in Example 2-128.
Example 2-128 VM with monitoring disabled at the KSYS level

# ksysmgr q vm vmraix2|grep -E "^Host|HA_monitor"
Host: Server-8408-E8E-SN21282FW
HA_monitor: disable
Example 2-129 shows that the VMM daemon starts with six threads.
Example 2-129 Kernel threads of the ksys_vmm daemon: VM monitoring not enabled yet
# uname -uL
IBM,0221282FW 4 vmraix2
TID : 8257809 (pTID : 1)
14549402 /usr/sbin/ksys_vmmd
TID : 24641969 (pTID : 1)
TID : 21234107 (pTID : 1286)
TID : 28967291 (pTID : 1029)
TID : 26870267 (pTID : 772)
TID : 22872321 (pTID : 515)
TID : 15860149 (pTID : 258)
#

Each thread has both a kernel TID value and a user space pTID value. We are interested in
the pTID value because it is used to identify the thread at the origin of each log record in VMM
log files. The thread stacks at this incipient stage, as shown in Example 2-130.
Example 2-130 VMM thread stacks at startup

14549402: /usr/sbin/ksys_vmmd
---------- tid# 24641969 (pthread ID: 1) ----------
...
0x10007e9c main(??, ??) + 0xf1c
0x100001e0 __start() + 0x68
---------- tid# 21234107 (pthread ID: 1286) ----------
...
0x100e6110 UnixSocket::threadMain()(??) + 0x50
0x102d02d0 Runnable::runThread(void*)(??) + 0x30
---------- tid# 28967291 (pthread ID: 1029) ----------
...
0x102cc564 VmMonitor::threadMain()(??) + 0x3c4
---------- tid# 26870267 (pthread ID: 772) ----------
...
0x100bb870 VmNetIntfHandler::threadMain()(??) + 0x2b0
---------- tid# 22872321 (pthread ID: 515) ----------
...
0x10263d04 AppReport::threadMain()(??) + 0x1a4
---------- tid# 15860149 (pthread ID: 258) ----------
...
0x1021b910 AppsMngt::doXML()(??) + 0xd0
0x10247424 AppsMngt::doMainLoop()(??) + 0x44
0x100ed1ec AppsMngt::threadMain()(??) + 0x12c
#
The symbolic names of the functions in the thread stacks provide valuable hints about the
purpose and function of each thread. We removed the most recent calls in each thread stack
to make the output shorter. Let us see how these threads create at the daemon start.
The excerpts in Example 2-131 on page 167 show us some initial entries that are logged at
the daemon start in both the ksys_vmm.log and ksys_vmm.debug files. They are all logged by
the main thread (pTID 1). The first entry begins in both places with the distinctive START string
as the value of the leftmost field in the row, followed by the daemon PID value and the main
thread pTID value, and then followed by the daemon start time stamp.

Example 2-131 Initial entries in VMM log files
# cd /var/ksys/log; head -10 ksys_vmm.log
[START 14549402 1 05/19/19-02:16:43 ]
[2 14549402 1 05/19/19-02:16:43 main.C:main(int, char **) : 191]
VmMonitor main
[2 14549402 1 05/19/19-02:16:43 VmUtils.C:vmutils::runSystemCommand(string) : 561]
Command: uptime> /var/ksys/log/cmdop_14549402_1 2>&1,
Command output:
02:16AM up 26 days, 12:38, 1 user, load average: 0.04, 0.04, 0.03
, returnCode: 0
Command: who -b> /var/ksys/log/cmdop_14549402_1 2>&1,
Command output:
. system boot Apr 22 13:38
, returnCode: 0
# head -16 ksys_vmm.debug
[START 14549402 1 05/19/19-02:16:43 ]
[3 14549402 1 05/19/19-02:16:43 XMLMonitor.C:XMLMonitor::setLogLevel() : 366]
[3 14549402 1 05/19/19-02:16:43 XMLMonitor.C:XMLMonitor::XMLMonitor(string) : 441]
/var/ksys/config/HAMonitoring.xml
[3 14549402 1 05/19/19-02:16:43 XMLMonitor.C:XMLMonitor::confExist(const string &): 120]
Checking /var/ksys/config/HAMonitoring.xml XML file exist
/var/ksys/config/HAMonitoring.xml XML file exists
[3 14549402 1 05/19/19-02:16:43 XMLMonitor.C:XMLMonitor::query(XMLVmmMonitoring...: 536]
Querying all attributes of VMM
[3 14549402 1 05/19/19-02:16:43 XMLMonitor.C:XMLMonitor::query(vector<std::basi...: 553]
Querying VMM
Querying VMM m_cur!=NULL
Monitoring
key=log, sValue=2
key=period, sValue=1
key=version, sValue=1.0
key=comment, sValue=2019-05-19 02:16:43
key=rebootappstarttype, sValue=VMM
VmMonitor main
Command: uptime> /var/ksys/log/cmdop_14549402_1 2>&1,
# cat /var/ksys/config/HAMonitoring.xml
<?xml version="1.0" encoding="UTF-8"?>
<Monitoring logfile="1" log="2" period="1" version="1.0" comment="2019-05-14 01:50:26"
upperminorver="0" uppermajorver="2" rebootappstarttype="VMM"/>
#

All the other entries display in their first three fields the current log level followed by the
process PID and the user level pTID of the thread generating the entry. Then, the time stamp
of the entry, the source code file name, and the function being run follow. Then, after a “:”
separator, the source code line number of the executed instruction follows. Finally, the
detailed message describing the performed action and resulted output appears. The
ksys_vmm.debug file has more details than ksys_vmm.log.
The ksys_vmm.debug excerpt in Example 2-131 on page 167 shows that the first action that is
performed by the main thread is to check for the parameters in the
/var/ksys/config/HAMonitoring.xml configuration file. Then, the uptime and who -b system
commands ran, as logged in both the ksys_vmm.log and ksys_vmm.debug files.
Checking further both log files, you can see an entry about a kernel extension and then how
the AppsMngt (pTID 258), AppReport (pTID 515), and VmNetIntfHandler (pTID 772) threads
are started in that order. Example 2-132 shows the related entries in ksys_vmm.log, which are
bolded rows 11, 13, 14, and 24.
Example 2-132 Initial entries in the VMM log file (continued)

# sed '11,$!d;=' ksys_vmm.log|sed 'N;s/\n/:/'|head -20
11:[2 14549402 1 05/19/19-02:16:43 main.C:main(int, char **) : 240] KE logs
will be redirected to /var/ksys/log/aix_vmmke.log
12:[2 14549402 1 05/19/19-02:16:43 AppsMngt.C:AppsMngt::AppsMngt() : 157] Daemon
restart.
13:[2 14549402 1 05/19/19-02:16:43 AppsMngt.C:AppsMngt::AppsMngt() : 171]
Applications Management Engine (AME) thread launched.
14:[2 14549402 515 05/19/19-02:16:43 AppReport.C:AppReport::threadMain() : 196]
Starting thread of Application Reporting.
15:[2 14549402 1 05/19/19-02:16:43 VmMonitor.C:VmMonitor::setVmUuid() : 1221] VmUUID:
1950678c-18bf-4c87-860c-f9e6ec24b513
16:[2 14549402 1 05/19/19-02:16:43 VmUtils.C:vmutils::runSystemCommand(string) : 561] Command:
mkdev -l vio0> /var/ksys/log/cmdop_14549402_1 2>&1,
17: Command output:
18: vio0 Available
19:, returnCode: 0
lsdev -c adapter | grep 'ênt'> /var/ksys/log/cmdop_14549402_1 2>&1,
21: Command output:
22: ent0 Available Virtual I/O Ethernet Adapter (l-lan)
23:, returnCode: 0
24:[2 14549402 1 05/19/19-02:16:43 VmNetIntfHandler.C:VmNetIntfHandler::startThread(): 75] Starting
VmNetIntfHandler thread.
25:[1 14549402 772 05/19/19-02:16:43 VmNetIntfHandler.C:VmNetIntfHandler::selectFD() : 133] Unable
to find a valid interface to listen for 'Hello' messages.
/usr/sbin/chvg -r y rootvg > /var/ksys/log/cmdop_14549402_1 2>&1,
27: Command output:
28: , returnCode: 0
29:[2 14549402 1 05/19/19-02:16:43 VmMonitor.C:VmMonitor::check_and_set_crit_VG_at...: 1697] We
successfully set critvg attr for 'rootvg'
30:
#

The Ethernet adapters are scanned by the newly started thread VmNetIntfHandler (pTID
772), which logs as its initial entry that it cannot find a valid interface to listen for 'Hello'
messages from the HM. This entry is expected for our case because the VM is not enabled
for monitoring by KSYS, so the virtual Ethernet adapters connecting the VM to the HMs are
not created. The next step sets rootvg to Critical by using the chvg system command. In
Example 2-133, we see how the vmmke kernel extension is loaded and then the VmMonitor
thread (pTID 1029) is started.
Example 2-133 VMM steady state: Listening for the Hello message
sed '31,$!d;=' ksys_vmm.log|sed 'N;s/\n/:/'|head -20
31:[2 14549402 1 05/19/19-02:16:43 VmMonitor.C:VmMonitor::VmMonitor() :
213] Critical VG was set for rootvg
32:
226] Kernel Extension vmmke loaded successfully
34:
276] VmMonitor thread successfully started.
286] List of network interfaces created
37:[2 14549402 1 05/19/19-02:16:43 VmUtils.C:vmutils::runSystemCommand(string) :
561] Command: lscfg -vl ent* > /var/ksys/log/cmdop_14549402_1 2>&1,
38: Command output:
39: ent0 U8408.E8E.21282FW-V4-C32-T1 Virtual I/O Ethernet Adapter (l-lan)
40:
41: Network Address.............FA858F9A8C20
42: Displayable Message.........Virtual I/O Ethernet Adapter (l-lan)
43: Hardware Location Code......U8408.E8E.21282FW-V4-C32-T1
44:
45:, returnCode: 0
46:[1 14549402 772 05/19/19-02:16:48 VmNetIntfHandler.C:VmNetIntfHandler::selectFD() :
133] Unable to find a valid interface to listen for 'Hello' messages.
#
Finally, we see in Example 2-133 how VMM enters a steady state loop where the
VmNetIntfHandler thread is looking repeatedly for interfaces on which to listen for Hello
messages, at every 5 seconds.
Because VMM kernel extension-related records appeared twice already, we take a quick look
at its log, as shown in Example 2-134.
Example 2-134 VMM kernel extension initial log entries

# head -25 aix_vmmke.log
May 19 02:16:43 vmraix2 kern:debug unix: Enter vmmke_entry_point:: command = 0x1
May 19 02:16:43 vmraix2 kern:debug unix: Initializing vmmke_entry_point KernExt
May 19 02:16:43 vmraix2 kern:debug unix: Creating a kernel process..

Sun May 19 02:16:43 2019 [LIB_WRAPPER] : Kernel module '/usr/lib/drivers/vmmke' Successfully
loaded with id '2686439424'
May 19 02:16:43 vmraix2 kern:debug unix: Waiting for the main process:15860098 to initialize the
modules..
May 19 02:16:43 vmraix2 kern:debug unix: Entered into main process.
May 19 02:16:43 vmraix2 kern:debug unix: Initializing communication and monitor control blocks
...
May 19 02:16:43 vmraix2 kern:debug unix: Initializing Communication Control Block @
'0xF1000000C040E528'
May 19 02:16:43 vmraix2 kern:debug unix: Successfully created the communication kernel thread.
tid=16449985
May 19 02:16:43 vmraix2 kern:debug unix: Initializing monitor control block @
'0xF1000000C040E548'
May 19 02:16:43 vmraix2 kern:debug unix: Successfully created the monitor kernel thread.
tid=24576269
May 19 02:16:43 vmraix2 kern:debug unix: Successfully created the kernel thread. pid=15860098
ppid=1
May 19 02:16:43 vmraix2 kern:debug unix: Signaling main process for successful initialization of
modules
May 19 02:16:43 vmraix2 kern:debug unix: 'shutdown_notify_reg' registered successfully.
sn:F1000000C040E648
May 19 02:16:43 vmraix2 kern:debug unix: Main thread going to sleep till Termination
requested...
May 19 02:16:43 vmraix2 kern:debug unix: In commuication_func....
May 19 02:16:43 vmraix2 kern:debug unix: In monitor_func....
May 19 02:16:43 vmraix2 kern:debug unix: woken up by main thread mcbP->mcb_flags=0x3
May 19 02:16:43 vmraix2 kern:debug unix: Successfully created and initialized main process.
pid=15860098
May 19 02:16:43 vmraix2 kern:debug unix: Initialization completed with rc: 0
May 19 02:16:43 vmraix2 kern:debug unix: vmmkeHB_syscall invoked by pid:14549402
May 19 02:21:41 vmraix2 kern:debug unix: vmmkeHB_syscall invoked. Received Heartbeat from pid :
14549402.
14549402.
14549402.
14549402.
#
After the initialization steps, the kernel extension enters a steady state in which it monitors the
VMM daemon by using a local heartbeat mechanism and is ready to take appropriate actions
if the daemon fails unexpectedly.
Let us now look at the complete thread structure of a fully functional VM Agent case where
communication with the underlying HMs is established. In Example 2-135, we see that our
agent daemon, with PID 6029594 and running the /usr/sbin/ksys_vmmd, has more threads
than in Example 2-129 on page 165.
Example 2-135 Kernel threads of the ksys_vmm daemon

TID : 7995671 (pTID : 1)
6029594 /usr/sbin/ksys_vmmd
TID : 15925555 (pTID : 1)

TID : 14025205 (pTID : 1286)
TID : 22741497 (pTID : 1029)
TID : 30671139 (pTID : 2828)
TID : 11862375 (pTID : 2571)
TID : 21954873 (pTID : 2314)
TID : 16187685 (pTID : 2057)
TID : 17236301 (pTID : 1800)
TID : 17039663 (pTID : 1543)
TID : 16253349 (pTID : 772)
TID : 23593327 (pTID : 515)
TID : 16122169 (pTID : 258)
#
In the process stack snapshot that is taken in Example 2-136, we again removed the most
recent calls in each thread stack and even the whole stack for some threads to make the
output shorter and more readable.
Example 2-136 VM Agent threads and associated stacks

# procstack 6029594
6029594: /usr/sbin/ksys_vmmd
---------- tid# 15925555 (pthread ID: 1) ----------
...
0x10007eac main(??, ??) + 0xf2c
0x100001e0 __start() + 0x68
---------- tid# 14025205 (pthread ID: 1286) ----------
...
0x100e74a8 UnixSocket::waitForClient()(??) + 0x28
0x100e5f40 UnixSocket::threadMain()(??) + 0x160
0x102d0f90 Runnable::runThread(void*)(??) + 0x30
---------- tid# 22741497 (pthread ID: 1029) ----------
...
0x102cd2d0 VmMonitor::threadMain()(??) + 0x3d0
---------- tid# 30671139 (pthread ID: 2828) ----------
...
0x1007180c HvncpSender::threadMain()(??) + 0x90c
---------- tid# 11862375 (pthread ID: 2571) ----------
...
0x100855ac HvncpReceiver::threadMain()(??) + 0x1fec
---------- tid# 21954873 (pthread ID: 2314) ----------
...
0x100a99cc readEthernetPacket(void*)(??) + 0x8c
0x1008efa4 VlanCommunication::threadMain()(??) + 0x44
---------- tid# 16187685 (pthread ID: 2057) ----------
...
0x100bb56c VmNetIntfHandler::threadMain()(??) + 0x2cc

---------- tid# 23593327 (pthread ID: 515) ----------
...
0x102643e4 AppReport::threadMain()(??) + 0x1a4
---------- tid# 16122169 (pthread ID: 258) ----------
...
0x102639ac AppsMngt::doSleep(timespec&)(??, ??) + 0x6c
0x1024751c AppsMngt::doMainLoop()(??) + 0x7c
0x100ece08 AppsMngt::threadMain()(??) + 0x88
#
You can easily identify in the output of Example 2-136 on page 171 all the expected initial
threads of the process (pTID 1) by running the main() function. Then, you see that a specific
class method, <class>::threadMain(), was started for each of the remaining threads. The
class names in the filtered list are suggestions for each thread, as shown in Example 2-137.
Example 2-137 Main thread calls of the ksys_vmmd daemon

# procstack 6029594|grep -E "main|threadMain"
0x10007eac main(??, ??) + 0xf2c
0x100e5f40 UnixSocket::threadMain()(??) + 0x160
0x102cd2d0 VmMonitor::threadMain()(??) + 0x3d0
0x100bb56c VmNetIntfHandler::threadMain()(??) + 0x2cc
0x102643e4 AppReport::threadMain()(??) + 0x1a4
0x100ece08 AppsMngt::threadMain()(??) + 0x88
#
Comparing with the threads in Example 2-130 on page 166 where VM monitoring is not
enabled, we notice here two extra triplets of communication threads: HvncpSender,
HvncpReceiver, and VlanCommunication. These triplets are created when the communication
with the underlying HMs is established, with one triplet per HM counterpart, as described in
“HM-VMM communication establishment” on page 215.
We now organize a bit of the bulk of information that is acquired by examining the daemon
logs and thread structure. Table 2-5 describes each listed thread type.
Table 2-5 Threads of the ksys_vmmd daemon

Thread type Thread class name Description
Initial process thread N/A Initial thread of the process, pTID=1, running the
main() function call.
VmMonitor VmMonitor Thread mainly sending heartbeats with HMs and

coordinating with the kernel extension.
UnixSocket UnixSocket Communicates with the CLI.

Thread type Thread class name Description
InterfaceHandler VmNetIntfHandler Searches for interfaces to look for Hello packets.
VlanCommunication VlanCommunication Thread reading from an interface.
HvncpSender HvncpSender Sends packets to the interface connected to the

corresponding HM.
HvncpReceiver HvncpReceiver Receives packets from that interface connected to

the corresponding HM.
Application AppsMngt Manages application star, stop, and monitoring.

Management Engine
(AME)
AR AppReport Informs the KSYS controller about local application

details.
We also identified the kernel extension component and the HAMonitoring.xml configuration
file. To better understand the way these VMM daemon threads interact with each other and
with related components, we placed them all in the diagram in Figure 2-12.
Figure 2-12 VM Agent interaction with related components
Threads responsible for communication with the HMs are grouped to simplify the diagram. Let
us first summarize what we know at this moment about the setup of communication with the
HMs.

The VmMonitor and VmNetIntfHandler threads are created among others at agent daemon
start. The VmNetIntfHandler thread scans for newly created or already available Virtual
Ethernet adapters and selects those with no IP addresses that are configured on their child
interfaces. The thread opens a raw socket per each selected interface, binds to it, and listens
for the Hello packets that are broadcast by the HMs on the VIOS side. We also see in “HM
startup sequence” on page 159 how the HM starts broadcasting these Hello packets. We set
up the scene this way to elaborate further on the communication topic.
Two scenarios are possible for the adapter detection happening during the VmNetIntfHandler
scan:
򐂰 KSYS discovery creates the adapters before VM Agent is installed.
Virtual Ethernet adapters that are created at the LPAR container level are discovered by
the first subsequent operating system device scan. During the VM Agent fileset
installation, the VMM daemon starts and detects the adapters, which are either already
discovered by the operating system or first discovered by its own device scan.
򐂰 VM Agent is installed first, and then KSYS discovery creates the adapters.
VMM scans repeatedly until the adapters are created. When host group discovery creates
the adapters at the LPAR container level, they show up at the OS level and are detected by
the first subsequent VM Agent scan.
In both scenarios, the Hello packets that are broadcast by an HM eventually reach an
interface on the VMM side. This situation is described in “HM-VMM communication
establishment” on page 215. We summarize it here.
If a Hello packet from the HM side arrives at an interface on the VMM side, then
VlanCommunication, HvncpSender, and HvncpReceiver created threads take over the
communication through that interface and a communication session is established with the
HM by a handshake protocol. VMM uses this session to send heartbeat messages to KSYS
and exchange application management-related messages with the KSYS side.
The VMM heartbeats are sent once every 10 seconds and coordinated by the VmMonitor
thread. The kernel extension stays in sync with the VmMonitor thread and takes over this
heartbeating task with the HM in case the VMM daemon hangs or fails. We delimit the VMM
subsystems that are responsible for the heartbeats: the VmMonitor thread and the kernel
extension. The kernel extension does not cover the application management messaging.
VMM and HM must use for this communication a protocol stack on top of the physical link
Ethernet layer that is similar to the TCP/IP stack. Because the communication volume
requirements are not that high and the media is reliable (hypervisor switch), we use a
simplified proprietary protocol stack that is called the HVNCP protocol. For more information
about this stack, see 2.2.9, “HVNCP protocol” on page 199.
In the upper left area of Figure 2-12 on page 173, we placed the ksysvmmgr command and the
HAMonitoring.xml configuration file. The ksysvmmgr command and VMM daemon
communicate directly, when needed, by UNIX sockets. On the VMM side, this communication
is the responsibility of the UnixSocket thread. Example 2-138 shows some samples for the
ksysvmmgr command that are used for VMM daemon management.
Example 2-138 Some ksysvmmgr usage samples

[vmraix1:root:/home/root:] ksysvmmgr status vmm
[vmraix1:root:/home/root:] ksysvmmgr stop vmm
0513-044 The ksys_vmm Subsystem was requested to stop.

[vmraix1:root:/home/root:] ksysvmmgr status
ksys_vmm inoperative
[vmraix1:root:/home/root:]
-------
# ksysmgr -v q vm vmraix1|grep HA_monitor_state

HA_monitor_state: SUSPENDED
root:vmrksys: /home/root >
We also notice in Example 2-138 on page 174 that the VMM status change that is performed
at the VM level is propagated to KSYS.
Example 2-139 lists the last records of the VMM daemon (PID 6095308), which stopped its
threads before exiting.
Example 2-139 The VMM log during graceful shutdown

[vmraix1:root:/var/ksys/log:] more ksys_vmm.debug
...
[3 6095308 2057 08/30/19-10:20:19 HvncpSender.C:HvncpSender::threadMain() :
814] Sending Packet to addr: 6A6A01581F11
[3 6095308 2057 08/30/19-10:20:19 HvncpPacket.C:HvncpPacket::display() const :
294] +Packet Frame (3 bytes) HB:
814] Sending Packet to addr: 6A6A0884EE11
...
VmMonitor daemon has been requested to SHUTDOWN
[3 6095308 1 08/30/19-10:20:28 VmMonitor.C:VmMonitor::~VmMonitor() : 351]
Entering destructor.
...
[3 6095308 1 08/30/19-10:20:28 UnixSocket.C:UnixSocket::disconnectClient() : 93]
Unix Socket: Closing connnection to client
[3 6095308 1 08/30/19-10:20:28 UnixSocket.C:UnixSocket::disconnectSocket() : 86]
Unix Socket: Closing connnection.
...
Kernel Extension vmmke unloaded successfully
Send SHUTDOWN notification to HsMon '6A6A01581F11'
Send SHUTDOWN notification to HsMon '6A6A0884EE11'
[3 6095308 1 08/30/19-10:20:30 AppsMngt.C:AppsMngt::stopThread() : 3132]
[3 6095308 1 08/30/19-10:20:30 AppsMngt.C:AppsMngt::finish() : 3058]
AME doing finish.
294] +Packet Frame (11 bytes) SHDN:DAEMON

294] +Packet Frame (11 bytes) SHDN:DAEMON
[3 6095308 1 08/30/19-10:20:30 AppsMngt.C:AppsMngt::stopThread() : 3148]
Stopping thread result is 0.
[2 6095308 1 08/30/19-10:20:32 NetworkController.C:NetworkController::stop() : 72]
Stopping the vLan thread = 1543 !!!
[ERROR 6095308 1543 08/30/19-10:20:32 HvncpPacket.C:HvncpPacket::checkVersion(con...:
319] Received an invalid packet (incomplete headers). Packet dropped.
Stopping the receiver thread = 1800 !!!
Stopping the sender thread= 2057 !!!
Interface ent1 : disconnected from VmMonitor.
...
VmMonitor has been shutdown
ksys_vmm.debug (69%)
We also notice how the last heartbeats ((3 bytes) HB) that were sent to the HMs before
entering the shutdown sequence are followed by the SHDN:DAEMON packets ((11 bytes),
which eventually cause the VM status to become SUSPENDED.
In terms of configuration updates, the ksysvmmgr command applies the changes that are
entered by the user to the HAMonitoring.xml configuration file and can send a notification to
the VMM daemon only through the UNIX socket interface (seethe -s flag or sync option for
ksysvmmgr). The VMM daemon reloads the XML configuration when it receives such a
notification.
For more information about the ksysvmmgr command, see 3.7, “Setting up the VM Agent” on
page 309.
To conclude, we covered in this section the default functions that are obtained by a blind
installation of the VM Agent on a managed VM. When the VM is enabled for monitoring at the
KSYS level, heartbeat-based VM monitoring is activated after the HM to VMM communication
is established. We also introduced the thread structure and some other involved components.
The group of communication threads handles the low-level communication housekeeping,
which supports higher-level heartbeats and exchange of application-related messages with
the neighboring HMs. The ksysvmmgr command that was demonstrated earlier in
Example 2-138 on page 174 can also be used for application-level management.
We are now ready to approach the other core functions of the VM Agent, which are
application monitoring and management.
Application Management Engine

The remaining VMM daemon threads in Figure 2-12 on page 173 are the AppsMngt and
AppReport threads, which are also known as the AME and AR threads if we use the identifiers
that are introduced in Table 2-5 on page 172.
AME normally refers to the thread that is the core component that handles the application
management and monitoring.

The AR thread is the VMM-side entity that is responsible for the exchange of application
status and configuration updates with the KSYS side. The information exchange uses the AR
Messaging protocol. An AR communication flow consists of a specific sequence of packets,
messages, and requests, which is denoted by dbell, PMSG_REQ, PMSG_RSP, GMSG, and Clear_DB
on the green line in Figure 2-12 on page 173. The flow is generated or conveyed by AME
threads, communication threads, HMs, HSDB, and KSYS. For more information about the AR
Messaging protocol, see 2.3.3, “Application Reporting flow” on page 262.
During the initial VM Agent installation, the VMM daemon starts with some default parameter
values and with no application configured, as shown in Example 2-140.
Example 2-140 VMM and application status just after the agent installation
[vmraix2:root:/home/root:] ksysvmmgr q
VmMonitor
log=2
period=1
version=1.0
comment=2019-05-19 02:16:43
rebootappstarttype=VMM
VmUUID=1950678c-18bf-4c87-860c-f9e6ec24b513
[vmraix2:root:/home/root:] ksysvmmgr q app
[vmraix2:root:/home/root:] ksysvmmgr status
[vmraix2:root:/home/root:] tail -3 /var/ksys/log/ksys_vmm_ame.log
[START 6881590 1 07/22/19-14:26:17 ]
[1 6881590 258 07/22/19-14:26:17 AppsMngt.C:AppsMngt::doXML() : 1322]
No application to monitor into configuration (into "/var/ksys/config/HAMonitoring.xml" file).
This is not an error. You can add application into configuration using "ksysvmmgr add" command,
and synchronize configuration with "ksysvmmgr sync" command.
No app at startup : AME thread going to sleep.
[vmraix2:root:/home/root:] procstack 6881590|sed '/(pthread ID: 258) ----------/,$!d'
---------- tid# 9568555 (pthread ID: 258) ----------
0xd057a534 _event_sleep(??, ??, ??, ??, ??, ??) + 0x4f4
0xd057b21c _event_wait(??, ??) + 0x35c
0xd058a8bc _cond_wait_local(??, ??, ??) + 0x19c
0xd058b1d4 _cond_wait(??, ??, ??) + 0x34
0xd058bc2c pthread_cond_wait(??, ??) + 0x1ac
0x1021b910 AppsMngt::doXML()(??) + 0xd0
0x10247424 AppsMngt::doMainLoop()(??) + 0x44
In Example 2-140, the AME thread (pTID 258) goes to sleep in the doXML() call, which is
inside the doMainLoop() main call.

The relevant VMM parameters for our purpose here are log and period, which are the log
verbosity level and the interval between two consecutive AME checks. They are documented
by the ksysvmmgr man command, as shown in Example 2-141.
Example 2-141 Raising the VMM log level

[vmraix1:root:/home/root:] man ksysvmmgr
...
log
Specifies the log level of the VM monitor
daemon. This attribute can have the following
values:
* 0: Only errors are logged. It is the

default value.
* 1: Warnings are also logged.
* 2: Informational messages are also
logged.
* 3: Details of the operation are logged.
This information is used for debugging.
period
Specifies the time duration in seconds between
two consecutive occurrences of checks that are
performed by the Application Management Engine
(AME). By default, the value of this attribute
is 1 second. The value of this attribute must
be in the range 0 - 6. For best monitoring
performance, do not modify the default value.
...
Our purpose here is to come up with a simple application setup and obtain an AME thread log
that is detailed enough to capture relevant insights about the AME internal operation. A
pattern of records is expected to show up repeatedly in the AME log at each second for the
AME checks because of the period parameter default value of 1. So, we opt for in
Example 2-142 a close monitor_period value of our test application to limit the length of the
log section that we must examine. Also, the log level is raised to 3 at this moment to capture
all possible details.
Example 2-142 Registering the app1 test application

[vmraix1:root:/tmp/HA:] ksysvmmgr stop vmm
[vmraix1:root:/tmp/HA:] ksysvmmgr add app app1 start_script=/tmp/HA/app1_start.sh
stop_script=/tmp/HA/app1_stop.sh monitor_script=/tmp/HA/app1_mon.sh
[vmraix1:root:/tmp/HA:] ksysvmmgr mod app app1 monitor_period=5
KSYSVMMGR-057(ERROR): monitor_period value must be greater than monitor_timeout
value(monitor_period > monitor_timeout).
Check cmdline or existing app monitor_period,monitor_timeout values then retry your "ksysvmmgr
modify app" operation with proper values
[vmraix1:root:/tmp/HA:] ksysvmmgr mod app app1 monitor_timeout=3
[vmraix1:root:/home/root:] ksysvmmgr modify log=3
[vmraix1:root:/tmp/HA:]

The ksysvmmgr command syntax and utilization for application management are covered in
3.7.2, “Application Management Engine” on page 312. Here, we register a simple dummy test
application with a monitoring script, as show in Example 2-142 on page 178, and play a bit
with its attributes.
Example 2-143 Monitoring script for the app1 test application

[vmraix1:root:/tmp/HA:] cat app1_mon.sh
#!/usr/bin/ksh
dt=`date`; echo "$dt:$$:$0 \c" >>/data/mon.log
echo "Entering Monitor..." >>/data/mon.log
sleep 1
dt=`date`; echo "$dt:$$:$0: \c" >>/data/mon.log
if [ -f /data/FAILAPP1MON ]; then
echo "Leaving Monitor, RC=1" >> /data/mon.log
exit 1
fi
echo "Leaving Monitor, RC=0">>/data/mon.log
exit 0
We ignore for the moment the start and stopped scripts. This test monitoring script returns
after a sleep of 1 second, either a 0 value, if the /data/FAILAPP1MON file exists, or 1 otherwise.
In a normal steady state for the application, the script returns 0. If the application is in a
normal state and we want to simulate a failure, we can create manually the FAILAPP1MON file
by running the >/data/FAILAPP1MON command. To simulate a recovery, we remove the file by
running a rm /data/FAILAPP1MON command.
At one moment in time after it was started, the script can be found in four possible states:
򐂰 Still running before its monitor_timeout timeout interval
򐂰 Already returned OK
򐂰 Already returned with an error
򐂰 Not returned after a timeout
We can force manually this last timeout state by editing online the script and increasing the
sleep interval. For our monitor_timeout=3 case, a value of 7 for the sleep parameter enforces
the timeout state.
Normal steady state application monitoring scenario

The output that is logged by the monitoring script at startup is shown in Example 2-144.
Example 2-144 Output that is logged by the monitoring script

[vmraix1:root:/tmp/HA:] >/data/mon.log
[vmraix1:root:/tmp/HA:] date;ksysvmmgr start vmm
Sat Aug 31 16:22:01 CUT 2019
0513-059 The ksys_vmm Subsystem has been started. Subsystem PID is 5570942.
[vmraix1:root:/tmp/HA:] tail -f /data/mon.log
Sat Aug 31 16:22:01 CUT 2019:6291810:/tmp/HA/app1_mon.sh Entering Monitor...
Sat Aug 31 16:22:02 CUT 2019:6291810:/tmp/HA/app1_mon.sh: Leaving Monitor, RC=0
^C[vmraix1:root:/tmp/HA:]

We see that the monitoring script starts at 16:22:01 and returns after 1 second. Then, 5
seconds later it is called again and returns again in 1 second, and so on. So, the AME log
records in the first 5 or 6 seconds after start reveal enough information for the AME internal
operation in this normal steady state scenario.
Example 2-145 shows how the application configuration is loaded from the .xml file at the
thread startup.
Example 2-145 Loading the configuration from the .xml file

[vmraix1:root:/home/root:] more /var/ksys/log/ksys_vmm_ame.log
[3 7995782 258 08/31/19-08:26:26 AppsMngt.C:AppsMngt::threadMain() : 1182]
Going out of AME thread.
[START 5570942 1 08/31/19-16:22:01 ]
[3 5570942 258 08/31/19-16:22:01 AppsMngt.C:AppsMngt::threadMain() : 1167]
Starting thread of AME.
m_bInitPhase "1", m_bNotificationReceived "0"
Loading XML configuration.
[3 5570942 258 08/31/19-16:22:01 AppsMngt.C:AppsMngt::loadFromXML() : 672]
[3 5570942 258 08/31/19-16:22:01 XMLMonitor.C:XMLMonitor::XMLMonitor(string) : 441]
/var/ksys/config/HAMonitoring.xml
[3 5570942 258 08/31/19-16:22:01 XMLMonitor.C:XMLMonitor::query(map<std::basic_s...: 609]
Querying all applications
...
For app "1567267349133928000": adding from XML configuration.
Application : '1567267349133928000' has added
Populating map of applications.
...
258 08/31/19-16:22:01 AppsMngt.C:AppsMngt::doXML() : 1328] Valid
configuration (into "/var/ksys/config/HAMonitoring.xml" file): normally starting or continuing
AME.
[3 5570942 258 08/31/19-16:22:01 AppsMngt.C:AppsMngt::doLaunch(int &) : 1405]
AME counter is 0 (at 3624862.469372535). List of 1 app uuid managed is "1567267349133928000".
...
You can see that all records are logged in the same second and that we skipped most of the
parsing and processing that the loaded XML configuration file logged further in the doXML()
method until the first occurrence of the doLaunch() method. Note the Populating map of
applications entry logged about our application with the 1567267349133928000 unique
identifier.

Examining the AME log further and looking for a 1-second basis repeating pattern of
subsequent method calls, we can delimit, by the time stamp field, blocks of entries that are
logged in the same second. Example 2-146 presents the first five of these blocks, with one
block for each of the first five consecutive seconds, from the start moment. The blocks of
records are marked in alternative bolding and normal typeset for easier identification. We
filtered the line code details on the right side for easier reading.
Example 2-146 Repeating pattern at each second

[vmraix1:root:/var/ksys/log:] cat ksys_vmm_ame.log|cut -c 1-85|more
[3 5570942 258 08/31/19-16:22:01 AppsMngt.C:AppsMngt::doXML() :
[3 5570942 258 08/31/19-16:22:01 AppsMngt.C:AppsMngt::doLaunch(int &) :
[3 5570942 258 08/31/19-16:22:01 AppsMngt.C:AppsMngt::dependencyCheck(AppEntry &) :
[3 5570942 258 08/31/19-16:22:01 AppsMngt.C:AppsMngt::doNonBlockingWait() :
[3 5570942 258 08/31/19-16:22:02 AppsMngt.C:AppsMngt::doSendNotification2Ar() :
[2 5570942 258 08/31/19-16:22:02 AppReport.C:AppReport::notify(string, HvncpMess...:
[2 5570942 258 08/31/19-16:22:04 AppReport.C:AppReport::notify(string, HvncpMess...:
Standard input

In the function calls in Example 2-146 on page 181, the doLaunch() and
doNonBlockingWait() methods are the dominant occurrences in each block. The doLaunch()
call shows up first, and then follows the doNonBlockingWait() call. This sequence pattern of
log records is noticed repeatedly in the steady state normal operation of the test application
app1.
With app1 application monitoring that is activated in this way, we examined repeatedly the
AME thread stack that is shown in Example 2-147. In all our captures, the thread showed up
in the doSleep() call under the doMainLoop() call, which suggests that the AME thread stays
most of the time in this sleep mode, and the other method calls showing up as logging
records in the AME log show up in the stack for such short periods that we can hardly capture
them.
Example 2-147 Checking repeatedly the AME thread stack

[vmraix1:root:/home/root:] while true; do procstack 5964236|sed '/(pthread ID:
258) ----------/,$!d'; sleep 2; done
---------- tid# 10289421 (pthread ID: 258) ----------
0xd057a534 _event_sleep(??, ??, ??, ??, ??, ??) + 0x4f4
0xd057b21c _event_wait(??, ??) + 0x35c
0xd058abd8 _cond_wait_local(??, ??, ??) + 0x4b8
0xd058b1d4 _cond_wait(??, ??, ??) + 0x34
0xd058b860 pthread_cond_timedwait(??, ??, ??) + 0x200
0x102632cc AppsMngt::doSleep(timespec&)(??, ??) + 0x6c
0x1024745c AppsMngt::doMainLoop()(??) + 0x7c
^C[vmraix1:root:/home/root:]
In Example 2-148 through Example 2-154 on page 185, we extracted excerpts from the AME
log by using the called method and the line code details fields only for the first
7 seconds from the start of our normal steady state scenario. AME uses its own counter for
these rounds, starting from a 0 value, and is logging this counter value each time as a first
action when entering the doLaunch() method. The doLaunch() method also logs a high
precision decimal time stamp at this moment. The round counter is initialized to zero at each
VMM start.
Example 2-148 Round 0: One application normal steady state scenario

[vmraix1:root:/var/ksys/log:] grep "08/31/19-16:22:01" ksys_vmm_ame.log|sed
'/doLaunch/,$!d'|while read lglvl pid tid ts method rest; do a=${method##*:}; b=${a%%\(*}; echo
"${b}() :${rest#*\]}"; done
doLaunch() : AME counter is 0 (at 3624862.469372535). List of 1 app uuid managed is
"1567267349133928000".
doLaunch() : For app "1567267349133928000": this application monitoring was already active.
dependencyCheck() : Dependency list is empty
doLaunch() : For app "1567267349133928000": application script "/tmp/HA/app1_mon.sh" being
forked ! pidParent(5570942) forks pidChild(6291810).
doLaunch() : For app "1567267349133928000": computing time of timeout by adding xmlAppTimeout(3)
to appEntry.m_timeScriptTimeoutAt => 3624865.469372535.
doLaunch() : Synthesis: list of 1 processes forked during this loop is
"(uuid=1567267349133928000;monitor;/tmp/HA/app1_mon.sh) ".
doNonBlockingWait() : AME waiting for 1 script(s) (at 3624862.470544447).
doNonBlockingWait() : +Synthesis of all (1) applications:

doNonBlockingWait() : | UUID STATE RESULT RUN MON DEL FAIL
STOP START RESTART DESCRIPTION
doNonBlockingWait() : | 1567267349133928000 1( UNSET) 99(NOTSET) 1 1 0 0/ 5 0/
3 0/ 3 0/3
doNonBlockingWait() : +
[vmraix1:root:/var/ksys/log:]
In Example 2-148 on page 182, we see how the doLaunch() call does its core job, which is to
check the internal application status in memory (various maps like the application map that is
mentioned in Example 2-145 on page 180) and start the appropriate script. It might be one of
the monitoring, stop, or start scripts, depending on the retrieved application status from
memory structures. The expected timeout moment for the started script is also computed and
saved for later use by adding the monitor_timeout value (here 3) to the call entering time,
3624865.469372535=3+3624862.469372535. The doNonBlockingWait() method is called
after a return from doLaunch(), here at 3624862.470544447, which means doLaunch() was
short (around 0.00117191 seconds). The doNonBlockingWait() checks the started children
processes and whether they returned; if so, then the application status is updated. The last
records print the status of the applications in the synthetic form of a table with 11 columns and
one row per monitored application.
In the next round, 1 second later, as shown in Example 2-149, doLaunch() and
doNonBlockingWait() are called but leave no relevant trace because there is no script to start
or a returned script to update afterward. The status remains unchanged.

[vmraix1:root:/var/ksys/log:] grep "08/31/19-16:22:02" ksys_vmm_ame.log|while read lglvl pid tid
ts method rest; do a=${method##*:}; b=${a%%\(*}; echo "${b}() :${rest#*\]}"; done
doSendNotification2Ar() : AME Notificaton Vector :
'0000000000000000000000000000000000000000000000000000000000001000'
notify() : Received notification from AME :
'0000000000000000000000000000000000000000000000000000000000001000'
doSendNotification2Ar() : ShortMessage vector reinitialized to :
'0000000000000000000000000000000000000000000000000000000000000000'
"1567267349133928000".
3 0/ 3 0/3

In the next round, which is shown in Example 2-150, doLaunch() does nothing again but
doNonBlockingWait() must update the status because the monitoring script returned OK. It
updates the status of the single running application, state to 2(NORMAL) from 1(UNSET), and
result to 0(NORMAL) from 99(NOTSET), with their counterpart fields also updated in the table,
STATE and RESULT. In the table, the RUN field is 0, which means that no script is running at this
moment.

"1567267349133928000".
doNonBlockingWait() : For app "1567267349133928000": rc is "0", result set to "NORMAL"
doNonBlockingWait() : For app "1567267349133928000": app state is "1(UNSET)", previous result
was "99(NOTSET)", current result is "0(NORMAL)".
doNonBlockingWait() : For app "1567267349133928000": monitoring is ok, result set to "NORMAL".
doNonBlockingWait() : Application : '1567267349133928000' status changed
doNonBlockingWait() : app '1567267349133928000' restartcount '0' appEntry.m_statsCollectionDone
: 'false'
doNonBlockingWait() : appEntry.m_lastStatsCollectedAt : '0' currentTime : '3624864'
doNonBlockingWait() : | 1567267349133928000 2( NORMAL) 0(NORMAL) 0 1 0 0/ 5 0/
3 0/ 3 0/3
In the next two rounds, as shown in Example 2-151 and Example 2-152 on page 185,
doLaunch() and doNonBlockingWait() return without performing any relevant actions.

doSendNotification2Ar() : AME Notificaton Vector :
'0000000000000000000000000000000000000000000000000000000000100000'
notify() : Received notification from AME :
'0000000000000000000000000000000000000000000000000000000000100000'
doSendNotification2Ar() : ShortMessage vector reinitialized to :
'0000000000000000000000000000000000000000000000000000000000000000'
"1567267349133928000".
3 0/ 3 0/3

[vmraix1:root:/var/ksys/log:] grep "08/31/19-16:22:05" ksys_vmm_ame.log|while read lglvl pid tid ts method
rest; do a=${method##*:}; b=${a%%\(*}; echo "${b}() :${rest#*\]}"; done
doLaunch() : AME counter is 4 (at 3624866.469566548). List of 1 app uuid managed is "1567267349133928000".
doNonBlockingWait() : | UUID STATE RESULT RUN MON DEL FAIL STOP START
RESTART DESCRIPTION
doNonBlockingWait() : | 1567267349133928000 2( NORMAL) 0(NORMAL) 0 1 0 0/ 5 0/ 3 0/ 3
0/3
In round 5, as shown in Example 2-153, doLaunch() discovers that it is time for the monitoring
script to start again and starts it. It also computes and saves the related timeout moment. The
doNonBlockingWait() method notices that the script is running, so we see that the RUN field
changed to 1. At this moment, we are in the same status as at the end of round 0, so we enter
a new similar cycle.

"1567267349133928000".
3 0/ 3 0/3
The last round, round 6 in Example 2-154, is identical to round 1 in terms of the doLaunch()
and doNonBlockingWait() actions and resulting status.

"1567267349133928000".
3 0/ 3 0/3

There is one more observation we make here that is related to the records about the AME
notifying the AR. They show up logged at the beginning of the next immediate round following
a round with configuration or status changes.
Example 2-155 lists the script files also for a second application, app2, and the content of its
monitoring script. You notice the similarity of the monitoring scripts.
Example 2-155 Monitoring scripts for dummy test applications

[vmraix1:root:/tmp/HA:] ls app1* app2*
app1_mon.sh app1_start.sh app1_stop.sh app2_mon.sh app2_start.sh
app2_stop.sh
[vmraix1:root:/tmp/HA:] cat app2_mon.sh
#!/usr/bin/ksh
dt=`date`; echo "$dt:$$:$0 \c" >>/data/mon.log
echo "Entering Monitor..." >>/data/mon.log
sleep 2
dt=`date`; echo "$dt:$$:$0: \c" >>/data/mon.log
if [ -f /data/FAILAPP2MON ]; then
echo "Leaving Monitor, RC=1" >> /data/mon.log
exit 1
fi
echo "Leaving Monitor, RC=0">>/data/mon.log
exit 0
In Example 2-156, we show how app2 is also registered in the AME.
Example 2-156 Registering the second test application

[vmraix1:root:/tmp/HA:] ksysvmmgr stop vmm
[vmraix1:root:/tmp/HA:] ksysvmmgr add app app2 start_script=/tmp/HA/app2_start.sh
stop_script=/tmp/HA/app2_stop.sh monitor_script=/tmp/HA/app2_mon.sh
monitor_timeout=10
[vmraix1:root:/tmp/HA:] ksysvmmgr mod app app2 monitor_period=10 monitor_timeout=5
[vmraix1:root:/tmp/HA:] >/data/mon.log
[vmraix1:root:/tmp/HA:] date;ksysvmmgr start vmm
Sat Aug 31 08:25:34 CUT 2019
The output is logged by both monitoring scripts in the same file, as shown in Example 2-157.
Example 2-157 Output that is logged by the monitoring scripts

[vmraix1:root:/home/root:] tail -f /data/mon.log

We see that both monitoring scripts start at 08:25:34, with app1_mon.sh returning after
1 second and app2_mon.sh after 2 seconds. Then, 10 seconds later, app2_mon.sh is called
again and returns in 2 seconds, as expected. After another 10-second interval, everything
repeats, which is consistent with our monitor_period settings and sleep intervals that are
inside the scripts.
Example 2-158 shows the records of the first round for this new scenario with two
applications that are configured and started simultaneously by VMM start after a prior VMM
stop.
Example 2-158 First round: AME counter 0 for a scenario with two applications
[vmraix1:root:/home/root:] grep "08/31/19-08:25:34" /var/ksys/log/ksys_vmm_ame.log|sed
'/doLaunch/,$!d'|while read lglvl pid tid ts method rest; do a=${method##*:}; b=${a%%\(*}; echo
">
"1567235918072992000 1567235971589285000".
doLaunch() : For app "1567235918072992000": computing time of timeout by adding
xmlAppTimeout(10) to appEntry.m_timeScriptTimeoutAt => 3596285.926736902.
"(uuid=1567235918072992000;monitor;/tmp/HA/app1_mon.sh)
(uuid=1567235971589285000;monitor;/tmp/HA/app2_mon.sh) ".
3 0/ 3 0/3
3 0/ 3 0/3

In Example 2-158 on page 187, doLaunch() and doNonBlockingWait() now do their jobs for
two applications, and the synthetic table at the end has a row for each application.
Up to 16 application can be configured, as shown in Example 2-159.
Example 2-159 Reaching the maximum number of allowed applications

[vmraix1:root:/home/root:] i=2; while true; do ksysvmmgr add app app$i
start_script=/tmp/HA/app2_start.sh stop_script=/tmp/HA/app2_stop.sh
monitor_script=/tmp/HA/app2_mon.sh; if [[ $? != 0 ]]; then break; fi; ((i=i+1)); done
Adding application "app2" into configuration successfully performed.
'-s' option is not provided.Please execute 'ksysvmmgr sync' to update VMM Daemon.
...
Application ADD failed, Maximum count of "16" applications reached.
Monitoring script failure scenario

Let us examine a monitoring script failure case that is enforced for our test app1, as shown in
Example 2-160.
Example 2-160 Enforcing a monitoring script failure for the app1 application
[vmraix1:root:/home/root:] ksysvmmgr mod app app1 monitor_failure_threshold=2 max_restart=1
[vmraix1:root:/home/root:] date;ksysvmmgr start vmm
Sun Sep 1 17:28:49 CUT 2019
[vmraix1:root:/home/root:] date; > /data/FAILAPP1MON
Sun Sep 1 17:28:57 CUT 2019
[vmraix1:root:/home/root:] ksysvmmgr stop vmm
In Example 2-161, we probed the application status from ksysvmmgr on a

1-second basis and kept only the captures before and after the state transitions.
Example 2-161 State changes during the monitoring script failure scenario for the app1 application
[vmraix1:root:/home/root:] while true; do dt=`date`; echo "$dt :\c"; ksysvmmgr -a status q app
app1|grep status; sleep 1; done
Sun Sep 1 17:28:49 CUT 2019 : status=NOT_MONITORED
Sun Sep 1 17:28:50 CUT 2019 : status=UNSET (YELLOW)
Sun Sep 1 17:28:52 CUT 2019 : status=NORMAL (GREEN)
...
Sun Sep 1 17:29:01 CUT 2019 : status=FAILING (YELLOW)
...
Sun Sep 1 17:29:07 CUT 2019 : status=TO_STOP (YELLOW
...
Sun Sep 1 17:29:11 CUT 2019 : status=TO_STOP (YELLOW)
Sun Sep 1 17:29:12 CUT 2019 : status=TO_START (YELLOW)

...
Sun Sep 1 17:29:17 CUT 2019 : status=FAILURE (RED)
From the AME log, we extracted only the relevant rounds, as shown in Example 2-162
through Example 2-165 on page 191. In round 12, as shown in Example 2-162, the
monitoring script is found to have failed the first time, so some status fields are updated from
normal as follows: applicate state (STATE) to 3(FAILING), last script return code (RESULT) to
1(ABNORM), RUN to 0 because we have no script running, and the current number of failures
counter (FAIL) to 1/2. Then, the next launch of the monitoring script, second here, happens in
round 16. Only the RUN field changes to 1 because we now have a script running.
Example 2-162 First two relevant AME log rounds: 12 and 16

09/01/19-17:29:01
"1567267349133928000".
doNonBlockingWait() : For app "1567267349133928000": rc is "1", result set to "ABNORM"
doNonBlockingWait() : For app "1567267349133928000": app state is "2(NORMAL)", previous result
was "0(NORMAL)", current result is "1(ABNORM)".
doNonBlockingWait() : For app "1567267349133928000": monitor failures(1)/max(2).
: 'false'
doNonBlockingWait() : | 1567267349133928000 3(FAILING) 1(ABNORM) 0 1 0 1/ 2 0/
3 0/ 3 0/1
...
09/01/19-17:29:05
"1567267349133928000".
doNonBlockingWait() : | 1567267349133928000 3(FAILING) 1(ABNORM) 1 1 0 1/ 2 0/
3 0/ 3 0/1

In Example 2-163, we see the monitoring script failing the second time in round 18. The
monitor_failure_threshold is reached, so a first restart is initiated. STATE changes to
4(TO_STOP), RUN to 0, and FAIL to 2/2.
Example 2-163 The next relevant AME log rounds for the monitoring script failure scenario: 18 and 19
09/01/19-17:29:07
"1567267349133928000".
doNonBlockingWait() : For app "1567267349133928000": rc is "1", result set to "ABNORM"
doNonBlockingWait() : For app "1567267349133928000": app state is "3(FAILING)", previous result
was "1(ABNORM)", current result is "1(ABNORM)".
doNonBlockingWait() : For app "1567267349133928000": monitor failures(2)/max(2). Needs to be
stopped.
: 'false'
doNonBlockingWait() : | 1567267349133928000 4(TO_STOP) 1(ABNORM) 0 1 0 2/ 2 0/
3 0/ 3 0/1
09/01/19-17:29:08
"1567267349133928000".
doLaunch() : For app "1567267349133928000": application script "/tmp/HA/app1_stop.sh" being
"(uuid=1567267349133928000;stop;/tmp/HA/app1_stop.sh) ".
doNonBlockingWait() : | 1567267349133928000 4(TO_STOP) 1(ABNORM) 1 1 0 2/ 2 0/
3 0/ 3 0/1
In the next round (19), as shown in Example 2-163, the stop script is started and RUN is
changed to 1.
The next relevant round, 23, is shown in Example 2-164. It finds the stop script returned OK,
and then updates STATE to 6(TO_START) because we are in a restart stage, RESULT to
0(NORMAL), and RUN to 0. So the next round, 24, starts the start script and updates RUN to 1.
Example 2-164 The next two relevant AME log rounds for a monitoring script failure scenario: 23 and 24
09/01/19-17:29:12
"1567267349133928000".
doLaunch() : For app "1567267349133928000": script running.

doNonBlockingWait() : For app "1567267349133928000": app state is "4(TO_STOP)", previous result
was "1(ABNORM)", current result is "0(NORMAL)".
doNonBlockingWait() : For app "1567267349133928000": stop ok, needs now to be started.
: 'false'
doNonBlockingWait() : | 1567267349133928000 6(TO_START) 0(NORMAL) 0 1 0 2/ 2 0/
3 0/ 3 0/1
09/01/19-17:29:13
"1567267349133928000".
doLaunch() : For app "1567267349133928000": application script "/tmp/HA/app1_start.sh" being
"(uuid=1567267349133928000;start;/tmp/HA/app1_start.sh) ".
doNonBlockingWait() : | 1567267349133928000 6(TO_START) 0(NORMAL) 1 1 0 2/ 2 0/
3 0/ 3 0/1
The start script returned OK in round 28, as shown in Example 2-165. As max_restart, 1 here,
is reached, the STATE goes to 9(FAILURE), RUN goes to 0, and RESTART to 1/1. This is the final
state for our failure scenario.
Example 2-165 The last relevant AME log round for the monitoring script failure scenario: 28
09/01/19-17:29:17
doLaunch() : AME counter is 28 (at 3715298.700496138). List of 1 app uuid managed is "1567267349133928000".
doLaunch() : For app "1567267349133928000": script running.
doNonBlockingWait() : For app "1567267349133928000": app state is "6(TO_START)", previous result was
"0(NORMAL)", current result is "0(NORMAL)".
doNonBlockingWait() : For app "1567267349133928000": start ok
...
doNonBlockingWait() : Application : '1567267349133928000' status changed
doNonBlockingWait() : For app "1567267349133928000": cycles(1)/max(1). State is now FAILURE.
doNonBlockingWait() : app '1567267349133928000' restartcount '1' appEntry.m_statsCollectionDone : 'false'
doNonBlockingWait() : | UUID STATE RESULT RUN MON DEL FAIL STOP START
RESTART DESCRIPTION
doNonBlockingWait() : | 1567267349133928000 9(FAILURE) 0(NORMAL) 0 1 0 2/ 2 0/ 3 0/ 3
1/1

Monitoring script timeout scenario
We enforce the timeout by increasing the sleep interval value from 1 to 7 in our test
monitoring script. The script change is made online while the application is in the normal
steady state, as shown in Example 2-166.
Example 2-166 Enforcing the timeout for the monitoring script while in the normal steady state
Sun Sep 1 20:15:47 CUT 2019
[vmraix1:root:/home/root:] ksysvmmgr q app app1|grep status
status=NORMAL (GREEN)
[vmraix1:root:/home/root:] vi /tmp/HA/app1_mon.sh
[vmraix1:root:/home/root:] date
Sun Sep 1 20:16:07 CUT 2019
[vmraix1:root:/home/root:] grep sleep /tmp/HA/app1_mon.sh
sleep 7
In Example 2-167, we probed the application state by using ksysvmmgr on a 1-second basis
and kept only the captures before and after the state transitions.
Example 2-167 State transitions during the monitoring script timeout scenario for the app1 application
[vmraix1:root:/home/root:] while true; do dt=`date`; echo "$dt :\c"; ksysvmmgr -a
status q app app1|grep status; sleep 1; done
Sun Sep 1 20:15:47 CUT 2019 : status=NOT_MONITORED
...
...
...
...
Sun Sep 1 20:16:28 CUT 2019 : status=FAILURE (RED)
From the AME log, we extracted only the relevant rounds, as shown in Example 2-168,
Example 2-169 on page 193, and Example 2-170 on page 194. In round 20, as shown in
Example 2-168, the monitoring script is started after a prior normal return and the time of
timeout is computed and saved.
Example 2-168 The first two relevant AME log rounds for the monitoring script timeout scenario: 20 and 23
09/01/19-20:16:07
"1567267349133928000".

3 0/ 3 0/1
...
09/01/19-20:16:10
"1567267349133928000".
3 0/ 3 0/1
doInspectForTimeout() : For app "1567267349133928000": timeout was set to 3725312.045299853.
doInspectForTimeout() : For app "1567267349133928000": script was in timeout, killing now script
process (pid "11075936").
The next relevant round, which is shown in Example 2-168 on page 192, is 23, where a new
method, doInspectForTimeout(), shows up with two records mentioning that the script is
found in timeout and that it kills the script process.
The next round, 24, shown in Example 2-169, finds the script that is killed and updates the
application status fields. STATE is changed to 3(FAILING), RESULT to 2(TIMOUT), RUN to 0, and
FAIL to 1/ 2.
Example 2-169 The next two relevant AME log rounds for the monitoring script timeout scenario: 24 and 25
09/01/19-20:16:11
"1567267349133928000".
doNonBlockingWait() : For app "1567267349133928000": pid "11075936" killed, result set to
"TIMOUT"
doNonBlockingWait() : For app "1567267349133928000": app state is "2(NORMAL)", previous result
was "0(NORMAL)", current result is "2(TIMOUT)".
doNonBlockingWait() : For app "1567267349133928000": monitor failures(1)/max(2).
: 'false'
doNonBlockingWait() : | 1567267349133928000 3(FAILING) 2(TIMOUT) 0 1 0 1/ 2 0/
3 0/ 3 0/1

09/01/19-20:16:12
"1567267349133928000".
3 0/ 3 0/1
The next round in Example 2-169 on page 193, 25, starts the monitoring script after the first
failure, computes its timeout time, and updates the status, which is RUN set to 1.
Example 2-170 shows the next relevant rounds, 29, 30, and 31. In round 29, the monitoring
script that restarted earlier in round 25 is found again in timeout by doInspectForTimeout(),
so its process is killed again.
Example 2-170 The next three relevant AME log rounds for the monitoring script timeout scenario: 29, 30, and 31
09/01/19-20:16:16
"1567267349133928000".
3 0/ 3 0/1
doInspectForTimeout() : For app "1567267349133928000": timeout was set to 3725317.046365310.
doInspectForTimeout() : For app "1567267349133928000": script was in timeout, killing now script
process (pid "11731218").
09/01/19-20:16:17
"1567267349133928000".
doLaunch() : For app "1567267349133928000": script was killed because of timeout.
doNonBlockingWait() : For app "1567267349133928000": pid "11731218" killed, result set to
"TIMOUT"
doNonBlockingWait() : For app "1567267349133928000": app state is "3(FAILING)", previous result
was "2(TIMOUT)", current result is "2(TIMOUT)".
doNonBlockingWait() : For app "1567267349133928000": monitor failures(2)/max(2). Needs to be
stopped.
: 'false'

doNonBlockingWait() : | 1567267349133928000 4(TO_STOP) 2(TIMOUT) 0 1 0 2/ 2 0/
3 0/ 3 0/1
09/01/19-20:16:18
"1567267349133928000".
doLaunch() : For app "1567267349133928000": application script "/tmp/HA/app1_stop.sh" being
"(uuid=1567267349133928000;stop;/tmp/HA/app1_stop.sh) ".
doNonBlockingWait() : | 1567267349133928000 4(TO_STOP) 2(TIMOUT) 1 1 0 2/ 2 0/
3 0/ 3 0/1
In the next round of Example 2-170 on page 194, 30, the script is discovered as killed while
running, so it fails a second time. The monitor_failure_threshold is reached, so a first
restart must be initiated. STATE changes to 4(TO_STOP), RUN to 0, FAIL to 2/2. The subsequent
round, 31, starts the stop script as the first restart step and sets RUN to 1. We do not continue
here with the subsequent rounds evolving toward the final FAILURE state because they are
similar to the rounds following to round 19 of the scenario in “Monitoring script timeout
scenario” on page 192.
Monitor script not usable scenario

We enforce this scenario by changing the name of the monitoring script while the application
is in the normal steady state, as shown in Example 2-171.
Example 2-171 Enforcing a script not usable case for the monitoring script while in the normal steady
state
Mon Sep 2 03:22:29 CUT 2019
[vmraix1:root:/home/root:] ksysvmmgr q app app1|grep status; mv
/tmp/HA/app1_mon.sh /tmp/HA/app1_mon.shh; date
Mon Sep 2 03:22:39 CUT 2019
In Example 2-172, we probed the application state by using ksysvmmgr on a 1-second basis
and captured the sequence of state transitions similar to the way we did in previous
scenarios.
Example 2-172 State transitions during the monitoring script not usable scenario for the app1
application
[vmraix1:root:/home/root:] while true; do dt=`date`; echo "$dt :\c"; ksysvmmgr -a
status q app app1|grep status; sleep 1; done
Mon Sep 2 03:22:29 CUT 2019 : status=NOT_MONITORED

Mon Sep 2 03:22:30 CUT 2019 : status=UNSET (YELLOW)
Mon Sep 2 03:22:31 CUT 2019 : status=UNSET (YELLOW)
Mon Sep 2 03:22:32 CUT 2019 : status=NORMAL (GREEN)
...
Mon Sep 2 03:22:39 CUT 2019 : status=NORMAL (GREEN)
Mon Sep 2 03:22:40 CUT 2019 : status=ABNORMAL (RED)
You must check for AME log rounds around 03:22:39. Example 2-173 shows the first round,
9, logged at 03:22:39, which shows a normal application state and a prior successful return.
Example 2-173 Checking the AME log

09/02/19-03:22:38
"1567267349133928000".
3 0/ 3 0/1
09/02/19-03:22:39
"1567267349133928000".
doLaunch() : For app "1567267349133928000": this application script "/tmp/HA/app1_mon.sh" is not
executable. Needs repair !
doLaunch() : Application : '1567267349133928000' status changed
doLaunch() : For app "1567267349133928000": this application gets E_ABNORMAL_STATE.
doNonBlockingWait() : | 1567267349133928000 8(ABNORMAL) 0(NORMAL) 0 1 0 0/ 2 0/
3 0/ 3 0/1
09/02/19-03:22:40
"1567267349133928000".
doLaunch() : For app "1567267349133928000": app state is "ABNORMAL". Needs repair !
3 0/ 3 0/1
09/02/19-03:22:41
"1567267349133928000".

3 0/ 3 0/1
The next round in Example 2-173 on page 196, 10, shows how the script that will be started is
not found in an appropriate state and how the application state is set to 8(ABNORMAL). This is a
final state, as shown in the next rounds.
Similar findings can be easily obtained for the start and stop script not usable scenarios.
Application state machine diagram

Considering all the findings so far in this section about the AME thread internal operation, we
aggregate the examined rounds into a pseudocode sketch for its doMainLoop() main call, as
Figure 2-13 AME thread main loop

We describe some of the methods inside the AME main loop that is shown in Figure 2-13 on
page 197:
򐂰 doXML()
At startup, the AME thread runs the doXML() method. The .xml config file is parsed for
defined applications. If no applications are defined, the thread goes to sleep from which it
can return by receiving a sync notification from ksysvmmgr. In such a situation, it parses the
.xml file again and looks for defined applications.
If defined applications are found in the .xml file, then some memory data structures, which
are called maps, are populated or updated with configuration and status details about
each found application as follows:
– If no entries exist in the memory maps for an application, then they are created.
– If entries exist, then they are updated as derived from the counterpart attributes that
are retrieved from the .xml file in case they are different.
After reaching this point, the thread returns from doXML() and calls doLaunch().
򐂰 doLaunch()
For each app in the app map, check the state and decide whether the mon, stop, or start
script must be started. If yes, start it and update the maps. Compute the time of the
timeout and update map entries. After reaching this point, the thread returns from
doLaunch() and calls doNonBlockingWait().
򐂰 doNonBlockingWait()
For each app, look for children processes (started scripts that are not marked as returned
yet) and check whether they exited or not. For those that returned, update the states.
򐂰 doInspectForTimeout()
For each app, if a timeout condition is ascertained on the current running script, the
blocking script is killed and the application status updates.
Similarly, putting together all noticed state transitions of our test application in the presented
scenarios, we come up with the state machine diagram in Figure 2-14.
Figure 2-14 Application state machine diagram

The abnormal and failure final states in the left side of the diagram must be resolved by
manual remediation of the root cause for the errors. To restart the application monitoring for
the affected application, the repair must be followed by an XML configuration reload.
2.2.9 HVNCP protocol

HM and VMM use the HVNCP proprietary protocol to communicate over the hypervisor
switch and the Ethernet layer in between. HVNCP implements a limited set of the capabilities
that are found in TCP/IP:
Pairing A point-to-point communication is established between the HM
and each VMM.
Communication closure Communications remain active unless a new communication is
opened between the same HM and VMM or an error occurs.
Flow control There is no flow control, so you avoid the processing impact of
communication for each message. We do not expect high packet
rates between the VMMs and the HM to justify the need for a flow
control that itself reduces the maximum packet rate that is
supported.
Message fragmentation If a message is too long to be encapsulated in a single packet, it
can be divided into several ones.
Message building A message is composed of one or more packets. The first and
last packets are identified so that messages can be separated.
Packet ordering Packets have a packet number, which is increased at each
packet sent, so that the receiver can restore the correct packet
order if needed.
Acknowledgment A command or response packet can request an acknowledgment
so that you can detect lost packets if an acknowledgment was
required but not received.
Checksum If there is a checksum error when receiving a packet, the packet
is discarded.
Error management If a packet that did not require an acknowledgment is lost, then it
is not retransmitted. If an acknowledgment was required and was
not received, then after reaching a timeout value the packet is
resent.

Figure 2-15 presents the HVNCP packet structure with the two 32-bit word header and
remaining payload.
Figure 2-15 HVNCP packet structure
Here are the details of the header fields:

Protocol version, major and minor (8 bits each) The first fields of the packet header
contain the protocol version and might
be used for future HVNCP extensions.
The minor number is increased in
case of modifications that do not
break compatibility. Otherwise, the
major number is increased.
Packet/Ack number (8 bits) A packet number field is used to
identify the packets of a message. The
number is increased at each packet
sent. When 0xFF is reached, the next
packet number is 0x00. If the ACK flag
is set to 1, it indicates the number of
the acknowledged packet. If the SYN
flag is set to 1, it indicates the number
of the first packet that is sent.
ACKR (1 bit) - acknowledgment request ACKR is set to 1 if the sent packet
requires an acknowledgment. A
packet that does not need to be
acknowledged cannot be linked to
another packet. Its corresponding
message must be only one packet
long. Otherwise, we have no way to
rebuild the message in case of an
error.
ACK (1 bit) – acknowledgment When set to 1, the packet number is
now an acknowledgment number.
BEG (1 bit) – Begin of message If BEG is set to 1, the packet is the first
packet of a message. If SYN is also set,
this flag has a special meaning. See
the SYN description.

END (1 bit) – End of message If END is set to 1, the packet is the last
packet of a message. If both BEG and
END are set to 1, the complete
message is in the frame. If SYN is also
set, this flag has a special meaning.
See the SYN description.
PRI (1 bit) – Priority of the packet 0: Normal
1: Urgent
An urgent packet suspends the
sending of normal packets such that it
is sent itself immediately.
Acknowledgements have the same

PRI value as their corresponding
message and have higher priority than
other packets with the same PRI
value.
When a multi-packets message is

suspended in the middle of its packets
transmission, it is the receiver’s
responsibility to correctly reconstruct
the message.
SYN (1 bit) – Synchronization packet A packet with the SYN flag set to 1 is a
special synchronization packet. It is
used to establish a communication or
to refresh a closed one. To establish a
communication, a Hello message
must be received first. If the
communication is not opened, a
synchronization packet is sent with the
ACKR flag set to 1. The receiver of the
packet must then acknowledge the
synchronization packet.
The BEG and END flags have a special

meaning if the SYN flag is set. In that
case, the BEG flag requires the receiver
to also ask its local sender to establish
communication in the opposite
direction by using the same
mechanism (with END instead of BEG to
avoid infinite recursions), but END does
not. The packet number of a
synchronization packet is the packet
number of the first packet that is sent
after the synchronization packet. A
synchronization packet cannot be sent
in parallel with other packets. When
the ACKR flag is set, several properties
of the local partition and of the
daemon are given in the frame.

HEL (1 bit) – Hello packet When HEL is set to 1, all other fields
except Checksum are ignored. This
special packet describes a Hello
message that is periodically broadcast
by HM to VMMs to signal that HM to
VMM communication can be
established.
Checksum (8 bits) The checksum is obtained by
calculating the sum of all bytes of the
header and the frame except for the
checksum field itself. Any packet with
an invalid checksum is automatically
discarded.
The HVNCP protocol provides the low-level communication means so that on top of the
Ethernet layer, the HM and VMM exchange messages for the following three main types of
communication flows:
򐂰 Communication session establishment
򐂰 Heartbeat messages from VMM toward KSYS
򐂰 Application management messages between the VMM and KSYS sides
2.3 Communication flows

This section shows the communication flows between various components of a VM Recovery
Manager HA deployment.
Example 2-65 on page 103 shows the complete list of possible KSYS to VIOS requests, as
documented in the /usr/lib/vioHADR2.00.xsd file at the VIOS level. Some of them are used
by KSYS to trigger communication flows by itself. Others are used in intermediate stages of
the flows that are initiated by other components. If the stages of different flows are similar, we
describe the first occurrence in detail and refer to it later. Section “CAA event monitoring” on
page 128 shows an example of a communication flow that is initiated outside of KSYS at the
VIOS CAA subsystem level underneath the host group SSP cluster.
From the KSYS subsystem perspective, first we consider all the flows running periodically
during the normal steady state operation of the cluster.
As described in “Quick Discovery and Deep Discovery scheduling records” on page 72,
periodic discovery actions are scheduled to identify configuration changes that might have
occurred in the whole infrastructure. A Quick Discovery action is performed hourly and a
Deep Discovery action every 24 hours. Detailed flow examples for each of these actions are
shown in 2.3.1, “Discovery of a VM that is enabled for monitoring” on page 203. To shorten
the explanations, we use the simplest possible environment of a host group with two hosts
and two VMs, one on each host. VMs themselves are configured in the simplest possible way,
which still allows us to capture the aspects of interest for the scenario under study.
Conversely, as described in “Failure Detection Engine” on page 94, two QuickQuery requests
and one NeedsAttention request are submitted at 20 seconds, one after the other, by the host
group FDE thread. The flows of these requests are described in 2.3.2, “Failure Detection
Engine: QuickQuery and NeedsAttention flows” on page 234.

2.3.1 Discovery of a VM that is enabled for monitoring
Let us now examine the flow of actions on the KSYS and VIOS sides for a VM enabled for
monitoring during a host group discovery that is performed immediately after the enablement.
To simplify the presentation, first we cover the case when the VM Agent is not installed on the
VM itself. Then, we examine the specifics of the case when the VM Agent is installed after the
monitoring enablement and the discovery of the VM in “HM-VMM communication
establishment” on page 215.
The status of the ms, vm, map, and trans tables in the HSDB before we start this exercise is
shown in Example 2-174.
Example 2-174 The ms, vm, map, and trans tables in HSDB just before the discovery command
vios=# select * from ms;
msid | clusterid | mtm | machinetype | model | serial
----------------------+-----------+-----------------+-------------+-------+---------
7926057748306591969 | 1 | 8408-E8E21282FW | 8408 | E8E | 21282FW
-5878939440565042805 | 1 | 8408-E8E212424W | 8408 | E8E | 212424W
(2 rows)
vios=# select * from vm;
vmuuid | name | msid | misshistory | state | up_major | up_minor | lo_major | lo_minor |
shortmsg | msgsync | msgsdeleted | misshistoryts | suspendts
--------+------------+------+-------------+-------+----------+----------+----------+----------+-
---------+---------+-------------+---------------+-----------
| default_vm | | 0 | 1 | 0 | 0 | 0 | 0 |
0 | 0 | 0 | |
(1 row)
vios=# select * from map;
vmuuid | viosuuid | misshistory | hmrecovery | viosrecovery | vmhbmissed
--------+----------+-------------+------------+--------------+------------
(0 rows)
--------+----------+------+-----+--------+-------+------+---------+-----------
(0 rows)
vios=#
The ms table is already populated with host entries, and the other tables are empty.
Example 2-175 shows the output of the manual discovery that is performed shortly after the
VM monitoring enablement command.
Example 2-175 Host group discovery after enabling monitoring for the vmraix1 VM
# ksysmgr q vm vmraix1|grep -e UUID -e HA_monitor
UUID: 61BE20F0-09A8-4753-B5E5-564F8A545ED5
HA_monitor: disable
# date; ksysmgr mod vm vmraix1 ha_monitor=enable
Tue Apr 30 08:58:25 CUT 2019
For VM vmraix1 attribute(s) 'ha_monitor' was successfully modified.
# date;ksysmgr -t discover hg rbHG
Tue Apr 30 08:59:54 CUT 2019
09:00:01 Running discovery on Host_group rbHG, this may take few minutes...
09:00:32 Existing HA trunk adapter found for VIOS usaxvib063ccpx1
09:00:37 Existing HA trunk adapter found for VIOS usaxvia063ccpx1

09:00:51 Existing HA trunk adapter found for VIOS usaxvia053ccpx1
09:00:51 Existing HA trunk adapter found for VIOS usaxvib053ccpx1
09:01:15 Preparing VIOS in Server-8408-E8E-SN21282FW for HA management
09:01:15 VIOS in Server-8408-E8E-SN21282FW prepared for HA management
09:01:15 Preparing VIOS in Server-8408-E8E-SN212424W for HA management
09:01:15 VIOS in Server-8408-E8E-SN212424W prepared for HA management
09:01:15 Preparing VM vmraix1 in Host Server-8408-E8E-SN212424W for HA management
09:01:15 VM vmraix1 in Host Server-8408-E8E-SN212424W Prepared for HA management
09:01:20 Creating first HA client adapter for VM vmraix1
09:01:25 Finished creating first HA client adapter for VM vmraix1
09:01:25 Creating second HA trunk client for VM vmraix1
09:01:44 Finished creating second HA client adapter for VM vmraix1
09:01:44 Starting HA monitoring for VM vmraix1
09:01:48 Discovery has started for VM vmraix2
09:01:48 Configuration information retrieval started for VM vmraix2
09:01:48 Discovery has started for VM vmraix1
09:01:48 Configuration information retrieval started for VM vmraix1
09:01:48 HA monitoring for VM vmraix1 started successfully
09:01:53 Configuration information retrieval completed for VM vmraix2
09:01:53 Discovery for VM vmraix2 is complete
09:01:53 Configuration information retrieval completed for VM vmraix1
09:01:53 Discovery for VM vmraix1 is complete
09:03:56 VM monitor state is not 'STARTED' for VM vmraix1
ERROR: Discovery has encountered an error for Host_group rbHG please investigate
09:03:57 Discovery has finished for rbHG
Following error is returned for VM vmraix1:

VM [vmraix1] state is not moved to started
Please review errors. The "ksysmgr query system status" command may provide additional details.
#
From the time stamps, we see that the whole discovery took about 4 minutes. We are
interested in the VM-related actions (the underlying infrastructure-related actions are covered
in 2.3.1, “Discovery of a VM that is enabled for monitoring” on page 203). The test
environment consists of a simple dual-VIOS configuration on two hosts with one managed
VM on each host. At least one previous discovery was performed since the rbHG host group
was added, so the SSP cluster is already created at the VIOS level and the hypervisor switch
on each component host. The presence of the hypervisor switches on each host is confirmed
in the output of Example 2-175 on page 203 by the virtual Ethernet trunk adapter found
messages for each VIOS apart.
The vmraix1 VM on the first host was enabled for monitoring before the current discovery, but
the vmraix2 VM remained disabled. You can see in the output of the discovery command how
vmraix1 is Prepared for HA management by the creation of two HA client adapters followed by
the action of Starting HA monitoring for VM vmraix1. Also, we see that VIOS HA trunk
adapters are found as existing from previous host group discovery. The final message that
vmraix1 VMM state is not moved to started shows up because the VM Agent is not yet
installed on the VM.
The KSYS trace files that we intend to examine are collected and shown in Example 2-176.
Example 2-176 Collecting trace files after the VM discovery

# ksysmgr trace log=krestlong > krestlong.log.posthamonitorenable
# ksysmgr trace log=krest > krest.log.posthamonitorenable

# ksysmgr trace log=ksys > ksys.log.posthamonitorenable
# ksysmgr trace log=fde > fde.log.posthamonitorenable
# ksysmgr trace log=fdelong > fdelong.log.posthamonitorenable
# ksysmgr trace log=user > user.log.posthamonitorenable
# ls
fde.log.posthamonitorenable krestlong.log.posthamonitorenable
fdelong.log.posthamonitorenable ksys.log.posthamonitorenable
krest.log.posthamonitorenable user.log.posthamonitorenable
#
Checking the entries in the user trace file that are logged after the discovery command start
moment, Tue Apr 30 08:59:54 CUT 2019, we see a first trace of the discovery flow in an
abridged form that is more detailed than the command output in Example 2-141 on page 178.
We divide this user trace file content, as shown in Example 2-177 and Example 2-178 on
page 206. In Example 2-177, we see that the host group discovery starts with a successful
verification block for the HMC, followed by a first CEC verification block, inside which VIOS
and trunk adapter verifications are passed successfully.
Example 2-177 The user trace file records for VM monitoring enable discovery
# more user.log.posthamonitorenable
[09] 04/30/19 T(203) _VMR 08:59:13.165979 [ INFO]: SCHEDULER initiated QUICK discoverVerify
Completed
[09] 04/30/19 T(9113) _VMR 08:59:57.241032 DEBUG VMR_HG.C[4919]: ------------- HG Verification
process START for HG : rbHG ------------
[09] 04/30/19 T(9113) _VMR 08:59:57.324091 [ INFO,VMR_HMCRccp,] Starting verify_HMCRccp
[09] 04/30/19 T(9113) _VMR 08:59:57.324110 [ INFO,VMR_HMCRcp,usaxhmc013ccpf1] Verify_HMCRcp
for HMC = usaxhmc013ccpf1, IP = 129.41.127.3
[09] 04/30/19 T(9113) _VMR 08:59:57.324116 [ INFO,VMR_HMCRcp,usaxhmc013ccpf1] Start
discover_HMC usaxhmc013ccpf1
[09] 04/30/19 T(9113) _VMR 08:59:57.636845 [ INFO,VMR_HMCRcp,usaxhmc013ccpf1] End
discover_HMC usaxhmc013ccpf1
[09] 04/30/19 T(9113) _VMR 08:59:57.636887 [ INFO,VMR_HMCRcp,usaxhmc013ccpf1] CEC List &
SessionID is same. So no update is required.
[09] 04/30/19 T(9113) _VMR 08:59:57.636894 [ INFO,VMR_HMCRcp,usaxhmc013ccpf1] Verify_HMCRcp
for HMC = usaxhmc013ccpf1 completed
[09] 04/30/19 T(9113) _VMR 08:59:57.636901 [ INFO,VMR_HMCRccp,] Completed verify_HMCRccp
[09] 04/30/19 T(9113) _VMR 09:00:13.643946 [

INFO,VMR_CECRcp,3a53a6fc-8dcb-3671-a853-70526651f83d] verify_CECRcp
CEC:Server-8408-E8E-SN212424W(3a53a6fc-8dcb-3671-a853-70526651f83d), cluster_type:HA
[09] 04/30/19 T(9113) _VMR 09:00:30.660046 [
INFO,VMR_VIOSRcp,5C3196A3-C41F-4B5C-9861-DEE6077081C6] Performing verify_VIOSRcp on VIOS
usaxvib063ccpx1
[09] 04/30/19 T(9113) _VMR 09:00:31.654865 DEBUG VMR_VIOS.C[5751]: verifyTrunkAdapter passed
on usaxvib063ccpx1
[09] 04/30/19 T(9113) _VMR 09:00:31.664472 [
INFO,VMR_VIOSRcp,5C3196A3-C41F-4B5C-9861-DEE6077081C6] verify_VIOSRcp done for VIOS
usaxvib063ccpx1
[09] 04/30/19 T(9113) _VMR 09:00:31.664495 [
INFO,VMR_VIOSRcp,0AB5001F-2C3E-4FBB-97EA-41D3F8C4F898] Performing verify_VIOSRcp on VIOS
usaxvia063ccpx1
on usaxvia063ccpx1

[09] 04/30/19 T(9113) _VMR 09:00:33.433325 [
INFO,VMR_VIOSRcp,0AB5001F-2C3E-4FBB-97EA-41D3F8C4F898] verify_VIOSRcp done for VIOS
usaxvia063ccpx1
[09] 04/30/19 T(9113) _VMR 09:00:35.406103 [
INFO,VMR_CECRcp,3a53a6fc-8dcb-3671-a853-70526651f83d] Leaving verify_CECRcp
CEC:Server-8408-E8E-SN212424W
user.log.posthamonitorenable (99%)
The sequence of actions continues with a similar CEC verification block for the second host in
the configuration, as shown in Example 2-178.
Example 2-178 The user trace file records for VM monitoring enable discovery (continued)
# more user.log.posthamonitorenable
[09] 04/30/19 T(9113) _VMR 09:00:35.406256 [
INFO,VMR_CECRcp,33613bcd-7ca1-3558-874a-c1e1d3ceee32] verify_CECRcp
CEC:Server-8408-E8E-SN21282FW(33613bcd-7ca1-3558-874a-c1e1d3ceee32), cluster_type:HA
[09] 04/30/19 T(9113) _VMR 09:00:48.562443 [
INFO,VMR_VIOSRcp,72DEE902-1210-4BD7-A35F-3A6C771C6453] Performing verify_VIOSRcp on VIOS
usaxvib053ccpx1
on usaxvib053ccpx1
[09] 04/30/19 T(9113) _VMR 09:00:49.826664 [
INFO,VMR_VIOSRcp,72DEE902-1210-4BD7-A35F-3A6C771C6453] verify_VIOSRcp done for VIOS
usaxvib053ccpx1
[09] 04/30/19 T(9113) _VMR 09:00:49.826680 [
INFO,VMR_VIOSRcp,10875B47-D737-44F9-A745-554F4DF4ADF8] Performing verify_VIOSRcp on VIOS
usaxvia053ccpx1
on usaxvia053ccpx1
[09] 04/30/19 T(9113) _VMR 09:00:51.207185 [
INFO,VMR_VIOSRcp,10875B47-D737-44F9-A745-554F4DF4ADF8] verify_VIOSRcp done for VIOS
usaxvia053ccpx1
[09] 04/30/19 T(9113) _VMR 09:00:53.173723 [
INFO,VMR_CECRcp,33613bcd-7ca1-3558-874a-c1e1d3ceee32] Leaving verify_CECRcp
CEC:Server-8408-E8E-SN21282FW
[09] 04/30/19 T(9113) _VMR 09:01:14.254966 DEBUG VMR_HG.C[12360]: Add managed system success for
CEC Server-8408-E8E-SN21282FW(33613bcd-7ca1-3558-874a-c1e1d3ceee32)
[09] 04/30/19 T(9113) _VMR 09:01:14.523511 DEBUG VMR_HG.C[12360]: Add managed system success for
CEC Server-8408-E8E-SN212424W(3a53a6fc-8dcb-3671-a853-70526651f83d)
[09] 04/30/19 T(9113) _VMR 09:01:14.785362 DEBUG VMR_CEC.C[13243]: add_VM() passed for host
Server-8408-E8E-SN212424W so now setting the status to VIOs managed for VM vmraix1
[09] 04/30/19 T(9113) _VMR 09:01:38.928457 DEBUG VMR_CEC.C[14972]: ClientEthernetAdapter
state:OK for VM:vmraix1(61BE20F0-09A8-4753-B5E5-564F8A545ED5)
[09] 04/30/19 T(9113) _VMR 09:01:44.801030 DEBUG VMR_CEC.C[14588]: StartVM() passed for host
Server-8408-E8E-SN212424W, VM vmraix1 so Moving VMMonitor state to starting
[09] 04/30/19 T(9113) _VMR 09:03:54.849762 DEBUG VMR_HG.C[5741]: ------------- HG Verification
process END for HG : rbHG ------------
user.log.posthamonitorenable: END

Both hosts are recorded as managed systems added successfully after these CEC
verifications. We expect that these hosts were already added after some previous discovery
and this CEC verification is just a reconfirmation. The discovery sequence continues with two
actions, which are add_VM() and StartVM() function calls for the vmraix1 VM, so we continue
to check the log files for more details.
The krest trace file reveals counterpart pass-through API requests to involve VIOSs, as
shown in Example 2-179, so we know what to look for at the VIOS level.
Example 2-179 The krest trace file records for VM monitoring enabled discovery
# egrep -i "addms|addvm|startvm" krest.log.posthamonitorenable
...
kriSubmitaddMS:hmc->(129.41.127.3),
[00] 04/30/19 T(9113) _VMR 09:01:14.119470 DEBUG libkrest.c[6712]: kriSubmitaddMS returning 0
with jobid->(1555954664653).
[00] 04/30/19 T(9113) _VMR 09:01:14.395984 DEBUG libkrest.c[6712]: kriSubmitaddMS returning 0
with jobid->(1555954664654).
kriSubmitaddVM:hmc->(129.41.127.3),
[00] 04/30/19 T(9113) _VMR 09:01:14.670640 DEBUG libkrest.c[7036]: kriSubmitaddVM returning 0
with jobid->(1555954664655).
kriSubmitstartVM:hmc->(129.41.127.3),
[00] 04/30/19 T(9113) _VMR 09:01:39.612644 DEBUG libkrest.c[10158]: kriSubmitstartVM returning 0
with jobid->(1555954664660).
#
In Example 2-180, we select only krest API requests that are submitted during the discovery
interval and then filter the less relevant of them for our purposes. To shorten the output, we
also remove the beginning fields.
Example 2-180 The krest requests during VM monitoring enabled discovery

# grep " 04/30/19 T(" krest.log.posthamonitorenable|sed '/_VMR\ 08:59:5/,/_VMR\
09:04:/!d'>krest.log.posthamonitorenable.discovery
# grep -c " kri" krest.log.posthamonitorenable.discovery
251
# grep " kri" krest.log.posthamonitorenable.discovery|grep -Ev
"krigetJobResult|kriSubmitQuickQuery|kriSubmitNeedAttn|kriSubmitExchange|krigetHmc|krigetCec|kri
getVios|krigetLpar|kriSubmitQueryNodeData|kriSubmitAck|returning"|awk '{$1=$2=$3=$4=$5=$6=$7="";
sub(" ",""); print}'
kriSwitchExists: START
kriTrunkAdapterExists on VIOS with UUID 5C3196A3-C41F-4B5C-9861-DEE6077081C6
kriTrunkAdapterExists: Calling getTrunkAdapterFromRawxml
kriTrunkAdapterExists on VIOS with UUID 0AB5001F-2C3E-4FBB-97EA-41D3F8C4F898
kriModifySwitch: START
kriSwitchExists: START
kriTrunkAdapterExists on VIOS with UUID 72DEE902-1210-4BD7-A35F-3A6C771C6453
kriTrunkAdapterExists on VIOS with UUID 10875B47-D737-44F9-A745-554F4DF4ADF8

kriModifySwitch: START
krigetSSPInfo:hmc->(129.41.127.3),sspid->(33dd9c03-6899-349c-901a-7b178bfc742e)
krisetComHMversion:hmc->(129.41.127.3),
kriSubmitaddVM:hmc->(129.41.127.3),
kriAdapterExists: START
kriAdapterExists: Checking Adapter with vlan:101,on switchid:2,for vm
61BE20F0-09A8-4753-B5E5-564F8A545ED5(cecuuid:3a53a6fc-8dcb-3671-a853-70526651f83d)
kriCreateAdapter: START
kriAdapterExists: START
kriAdapterExists: Checking Adapter with vlan:102,on switchid:2,for vm
61BE20F0-09A8-4753-B5E5-564F8A545ED5(cecuuid:3a53a6fc-8dcb-3671-a853-70526651f83d)
kriCreateAdapter: START
kriSubmitqueryVM:hmc->(129.41.127.3),
kriSubmitstartVM:hmc->(129.41.127.3),
kriSubmitGetLCB:hmc->(129.41.127.3),vm->(61BE20F0-09A8-4753-B5E5-564F8A545ED5)
kriSubmitGetLCB:hmc->(129.41.127.3),vm->(1950678C-18BF-4C87-860C-F9E6EC24B513)
kriSubmitqueryVM:hmc->(129.41.127.3),
#
For the first kriSubmitaddMS call (the Server-8408-E8E-SN21282FW host), we identify the
counterpart VIO_HS_ADD_MS pass-through API request at the VIOS level, as shown in
Example 2-181.
Example 2-181 VIO_HS_ADD_MS request trace in viosvc.log

# uname -uL
IBM,0221282FW 3 usaxvib053ccpx1
# alog -f /home/ios/logs/viosvc.log -o > viosvc.log.posthamonitorenable
#more viosvc.log.posthamonitorenable
[START 25756024 59834737 04/30/19-09:01:14.128 vioservice.c 1.17 306] /usr/ios/sbin/vioservice
lib/libviopass/passthru
version="1.0"?>
author="LIBKREST" title="Add Managed System">
<Request action_str="VIO_HS_ADD_MS" hsTag="0">
<addMS>
<viosList machine_type="8408" model="E8E" serial="21282FW">
<vios uuid="72dee902-1210-4bd7-a35f-3a6c771c6453" isHM="true" partNum="3"
macAddr="16DC6A23650F"/>
<vios uuid="10875b47-d737-44f9-a745-554f4df4adf8" isHM="true" partNum="2"
macAddr="16DC662CB90E"/>
</viosList>
</addMS>
</Request>
</VIO>
]
[0 25756024 59834737 04/30/19-09:01:14.161 viosvc_res.c 1.26 464] vio_response.result=[<?xml
version="1.0"?>
version="2.00"
title="VIOS ADD_MS"
published="2019-04-30T01:17:00Z"

>
<Response>
</Response>
</VIO>
]
viosvc.log.posthamonitorenable (92%)
#
Here is an excerpt from the /usr/lib/vioHADR2.00.xsd file on VIOS detailing the

VIO_HS_ADD_MS request:
- VIO_HS_ADD_MS: KSYS sends to a VIOS on the managed system in an SSP cluster that
is to be added to the list of managed systems in the cluster participating in the
HA solution. The XML document includes the list of VIOS on the system 2 of which
must be designated as HMs.
Using the vioservice thread PID to TID combination, we identify the health library log trace,
as shown in Example 2-182.
Example 2-182 The VIO_HS_ADD_MS trace in the health library log

# alog -f /home/ios/logs/health/viohs.log -o > viohs.log.posthamonitorenable
# grep "25756024 59834737" viohs.log.posthamonitorenable
[START 25756024 59834737 04/30/19-09:01:14.129 ha_util.c 1.43 230]
[3 25756024 59834737 04/30/19-09:01:14.129 hs_util.c hs_libvio_traces 1.66 60] HEALTH:59834737
-- violibExchange.c initTransaction 1.27 646 Got cluster info from
vioGetClusterInfoFromODM! cluster name='KSYS_rbRMHA_1' id=e4c3993266fd11e9801efa242424f21c
[3 25756024 59834737 04/30/19-09:01:14.130 hs_ms.c hsParseAddMsInputs 1.83 1758] Found '1'
elements for '/VIO/Request/addMS'.
elements for '/VIO/Request/addMS/viosList'.
[3 25756024 59834737 04/30/19-09:01:14.130 hs_ms.c hsParseSysProperties 1.83 1850] Input managed
system machine_type is '8408'
system model is 'E8E'
system serial is '21282FW'
elements for '/VIO/Request/addMS/viosList/vios'.
[3 25756024 59834737 04/30/19-09:01:14.130 hs_ms.c hsParseViosList 1.83 1650] Found '2' elements
for '/VIO/Request/addMS/viosList/vios'.
-- violibDB.c _allocHandleDB 1.132.12.4 589 HDBC = 2001a758
[3 25756024 59834737 04/30/19-09:01:14.160 hs_ms.c hsCurMsPendop 1.83 1429] MS exists=1,
pendop=0, type=8408, model=E8E, serial=21282FW
[3 25756024 59834737 04/30/19-09:01:14.160 hs_ms.c vioHsAddMs 1.83 2274] Managed system is
already existing.
#

The result of the vioHsAddMs call is the Managed system is already existing message in the
HSDB, which is an expected result. A similar sequence occurs for the second host, so we skip
it and examine the VIOS logs for the kriSubmitaddVM call. Example 2-183 shows the
VIO_HS_ADD_VM request for the vmraix1 VM as received at VIOS level.
Example 2-183 VIO_HS_ADD_VM request for the vmraix1 VM as received at the VIOS level
version="1.0"?>
author="LIBKREST" title="Add VM">
<Request action_str="VIO_HS_ADD_VM" hsTag="0">
<addVM>
<vmList machine_type="8408" model="E8E" serial="212424W">
<vm uuid="61be20f0-09a8-4753-b5e5-564f8a545ed5"/>
</vmList>
</addVM>
</Request>
</VIO>
]
[0 20644150 79626541 04/30/19-09:01:14.714 viosvc_res.c 1.26 464] vio_response.result=[<?xml
version="1.0"?>
version="2.00"
title="VIOS ADD_VM"
published="2019-04-30T01:17:00Z"
>
<Response>
</Response>
</VIO>
]
Using the vioservice thread PID - TID combination, we identify the health library log trace, as
Example 2-184 VIO_HS_ADD_VM trace in health library log

[START 20644150 79626541 04/30/19-09:01:14.675 ha_util.c 1.43 230]
[3 20644150 79626541 04/30/19-09:01:14.676 hs_vm.c vioHsAddVm 1.26 324] Begin.
-- violibDB.c _allocHandleDB 1.132.12.4 589 HDBC = 20018f18
[3 20644150 79626541 04/30/19-09:01:14.709 hs_vm.c hsParseMsVmInputs 1.26 181] Found '1'
elements for '/VIO/Request/addVM/vmList'.
system machine_type is '8408'

system model is 'E8E'
system serial is '212424W'
[3 20644150 79626541 04/30/19-09:01:14.709 hs_vm.c hsParseVmList 1.26 147] Found
61be20f0-09a8-4753-b5e5-564f8a545ed5 in addvm document
[3 20644150 79626541 04/30/19-09:01:14.713 hs_vm.c vioHsAddVm 1.26 389] END: rc = 0.
#
The hsParseVmList entry before vioHsAddVm successful return is rather vague: Found
61be20f0-09a8-4753-b5e5-564f8a545ed5 in addvm document. But, the PostgreSQL log entry
around this return moment, as shown in Example 2-185, clarifies it.
Example 2-185 SQL INSERT into the VM table for the vmraix1 VM
# pwd
/home/ios/logs/pg_sspdb
# more pg_sspdb-30-08-40.log
...
2019-04-30 09:01:14.712 CUT|5ccdbfa7.ed017a|LOG: statement: RELEASE
_EXEC_SVP_20122d28;SAVEPOINT _EXEC_SVP_20122d28;INSERT INTO vioshs.vm(vmuuid, name, state)
VALUES('61be20f0-09a8-4753-b5e5-564f8a545ed5', '', 1)
...
So, an entry for the vmraix1 VM is inserted into the vm table as a result of the VIO_HS_ADD_VM
request from KSYS. Note the value 1 for the state field at this insert moment. The vm table
status at the end of this discovery command in shown in Example 2-188 on page 213, where
the state field value is 2 at that moment.
The next action that we investigate is the kriSubmitstartVM call with its counterpart
VIO_HS_START_VM at the VIOS level, as shown in Example 2-186.
Example 2-186 VIO_HS_START_VM request for the vmraix1 VM as received at the VIOS level
version="1.0"?>
author="LIBKREST" title="Start VM">
<Request action_str="VIO_HS_START_VM" hsTag="0">
<startVM>
<vm uuid="61be20f0-09a8-4753-b5e5-564f8a545ed5"/>
</vmList>
</startVM>
</Request>
</VIO>
]
[0 22282714 72745423 04/30/19-09:01:39.761 viosvc_res.c 1.26 464] vio_response.result=[]

Using the vioservice thread PID - TID combination, we further identify the health library log
trace, as shown in Example 2-187.
Example 2-187 VIO_HS_START_VM trace in the health library log

[START 22282714 72745423 04/30/19-09:01:39.619 ha_util.c 1.43 230]
[3 22282714 72745423 04/30/19-09:01:39.670 hs_start_vm.c derive_msid_from_request 1.79 340]
request intended for managed system 8408-E8E-212424W
-- violibDB.c _allocHandleDB 1.132.12.4 589 HDBC = 20016648
[3 22282714 72745423 04/30/19-09:01:39.688 hs_start_vm.c vioHsStartVm 1.79 141] received StartVM
request for managed system AE69CFBA0D16418B
[3 22282714 72745423 04/30/19-09:01:39.707 hs_start_vm.c vioHsStartVm 1.79 173] StartVM request
tag is D7E3D316D0D88C47
[3 22282714 72745423 04/30/19-09:01:39.707 hs_start_vm.c vioHsStartVm 1.79 186] StartVM
requested for (1) machines
requested for 61be20f0-09a8-4753-b5e5-564f8a545ed5
-- violibDB.c _allocHandleDB 1.132.12.4 589 HDBC = 20018c88
registered for 61be20f0-09a8-4753-b5e5-564f8a545ed5
registered for (1/1) machines
-- violibDB.c _allocHandleDB 1.132.12.4 589 HDBC = 20018c88
[3 22282714 72745423 04/30/19-09:01:39.748 hs_start_vm.c tell_monitors 1.79 838] Telling host
monitors about StartVM tag D7E3D316D0D88C47
[3 22282714 72745423 04/30/19-09:01:39.755 hs_start_vm.c tell_monitors 1.79 850] Told host
monitor on VIOS 5c3196a3-c41f-4b5c-9861-dee6077081c6 about StartVM tag D7E3D316D0D88C47
[3 22282714 72745423 04/30/19-09:01:39.760 hs_start_vm.c tell_monitors 1.79 850] Told host
monitor on VIOS 0ab5001f-2c3e-4fbb-97ea-41d3f8c4f898 about StartVM tag D7E3D316D0D88C47
[3 22282714 72745423 04/30/19-09:01:39.760 hs_start_vm.c tell_monitors 1.79 865] Told (2/2) host
monitors about StartVM tag D7E3D316D0D88C47
#
The sequence of health library records in Example 2-187 reveals that actions are performed
at both the HSDB and HM levels.
A tag with a D7E3D316D0D88C47 hexadecimal value is generated at the health library level for
the StartVM request, and the HSDB is updated with details about this request, including this
tag value. Example 2-188 on page 213 shows the ms, vm, map, and trans tables at the end of
the discovery command.

Example 2-188 ms, vm, map, and trans tables in HSDB just after the discovery command
vios=# select * from ms;
msid | clusterid | mtm | machinetype | model | serial
----------------------+-----------+-----------------+-------------+-------+---------
7926057748306591969 | 1 | 8408-E8E21282FW | 8408 | E8E | 21282FW
-5878939440565042805 | 1 | 8408-E8E212424W | 8408 | E8E | 212424W
(2 rows)
vios=# select * from vm;
vmuuid | name | msid | misshistory | state
| up_major | up_minor | lo_major | lo_minor | shortmsg | msgsync | msgsdeleted | misshistoryts |
suspendts
--------------------------------------+------------+----------------------+-------------+-------
+----------+----------+----------+----------+----------+---------+-------------+---------------+
-----------
| default_vm | | 0 | 1
| 0 | 0 | 0 | 0 | 0 | 0 | 0 | |
61be20f0-09a8-4753-b5e5-564f8a545ed5 | | -5878939440565042805 | 0 | 2
| 0 | 0 | 0 | 0 | 0 | 0 | 0 | |
(2 rows)
vmuuid | viosuuid | misshistory |
hmrecovery | viosrecovery | vmhbmissed
--------------------------------------+--------------------------------------+-------------+----
--------+--------------+------------
61be20f0-09a8-4753-b5e5-564f8a545ed5 | 5c3196a3-c41f-4b5c-9861-dee6077081c6 | 0|
0 | 0 |
61be20f0-09a8-4753-b5e5-564f8a545ed5 | 0ab5001f-2c3e-4fbb-97ea-41d3f8c4f898 | 0|
0 | 0 |
(2 rows)
vmuuid | viosuuid | msid
| tag | opcode | state | data | integer | txstarted
--------------------------------------+--------------------------------------+------------------
----+-----------------------+--------+-------+------+---------+----------------------------
61be20f0-09a8-4753-b5e5-564f8a545ed5 | 5c3196a3-c41f-4b5c-9861-dee6077081c6 |
-5878939440565042805 | -2890234440895132601 | 1 | 9 | | 0 | 2019-04-30
09:01:39.712969
61be20f0-09a8-4753-b5e5-564f8a545ed5 | 0ab5001f-2c3e-4fbb-97ea-41d3f8c4f898 |
-5878939440565042805 | -2890234440895132601 | 1 | 9 | | 0 | 2019-04-30
09:01:39.739899
(2 rows)
to_hex
------------------
d7e3d316d0d88c47
(1 row)
vios=#

The state value in the vm table changed from 1 to 2 because it was after the addVM flow (see
Example 2-185 on page 211). We present in this context a relevant excerpt from the
/usr/lib/vioHADR2.00.xsd file on VIOS that mentions the addVM flow and possible state
values for a VM:
...
vmStateType
===========
ADDED: The VM has been added via the addVM flow.
STARTING: HM is attempting to start monitoring of the VM in response to a request
from KSYS
STARTED: KSYS has requested that a particular VM is monitored on a managed system
and both HMs on that system have received acknowledgements from the respective VMM.
VIOS asserts need attention when VM moves from starting to started state. KSYS can
choose to fail adding a VM after some reasonable time period if it has not moved to
started state.
..
In our case, the vmraix1 VM changed from the ADDED (1) state to the STARTING (2) state,
where it remained after the completion of the discovery command. To understand why it
remained in STARTING state, we look into the HM logs for traces that are related to the last
entries in Example 2-187 on page 212. The HM log records around the same interval for the
VIOS with the 5c3196a3-c41f-4b5c-9861-dee6077081c6 UUID are shown in Example 2-189.
Example 2-189 UNIX Socket command that is received by an HM to start monitoring on the vmraix1 VM
# uname -uL
IBM,02212424W 3 usaxvib063ccpx1
# lsattr -El vios0 -a vios_uuid
vios_uuid 5c3196a3-c41f-4b5c-9861-dee6077081c6 VIOS Unique Identifier False
# alog -f /home/ios/logs/health/host_monitor.log -o > host_monitor.log.posthamonitorenable
# more host_monitor.log.posthamonitorenable
[3 16122210 91554293 04/30/19-09:01:36.356 Processing.C:Processing::cleanMsgBuffer(const s...:
833] Removing message to VmMon at macAddr 'FFFFFFFFFFFF' from send queue. Reason: Fully sent
[3 16122210 91095433 04/30/19-09:01:39.756 HostMonitor.C:HostMonitor::socketReceiveMessage...:
134] Unix Socket: Received command 0x00000101 (tag 0xD7E3D316D0D88C47)
[3 16122210 91095433 04/30/19-09:01:39.758 HostMonitor.C:HostMonitor::socketReceiveMessage...:
137] Received Valid Request Payload.
[2 16122210 91095433 04/30/19-09:01:40.798 DatabaseAccess.C:DatabaseAccess::getVmList(vect...:
219] vioHmStartVMs: VMs to Start=1, rc=0
[2 16122210 91095433 04/30/19-09:01:40.798 Processing.C:Processing::controlVm(const string...:
505] HostMonitor is asked to START monitoring VM '61be20f0-09a8-4753-b5e5-564f8a545ed5' (command
from SSP Database)
[3 16122210 91095433 04/30/19-09:01:40.798 VmHealthStatusMgr.C:VmHealthStatusMgr::monitorV...:
183] Start monitoring VM '61be20f0-09a8-4753-b5e5-564f8a545ed5'.
209] Monitoring won't start for VM '61be20f0-09a8-4753-b5e5-564f8a545ed5' : VmMonitor has not
yet been discovered.
540] VM '61be20f0-09a8-4753-b5e5-564f8a545ed5' has not yet been discovered. Cannot process
action.
[3 16122210 91095433 04/30/19-09:01:40.798 UnixSocket.C:UnixSocket::disconnectClient() :
93] Unix Socket: Closing connnection to client
host_monitor.log.posthamonitorenable (99%)

This HM receives through a UNIX socket a command, here identified as 0x00000101 (tag
0xD7E3D316D0D88C47). This command results immediately in a vioHmStartVMs call, which in
turn results in a tentative start monitoring action for the vmraix1 VM. We also note that the
associated hexadecimal tag value is the same as in the related health library log records. But,
even if the HA monitoring is enabled at KSYS, the actual VM monitoring does not start and
remains marked as STARTING at the HSDB level because the VM Agent has not yet been
discovered by our HM instance. This is expected because we did not install the VM Agent
fileset on the vmraix1 VM, so there is no appropriate counterpart endpoint there to interact
with the HM instance.
Install the fileset and see what happens. As described “HM startup sequence” on page 159,
the running HM instances are already broadcasting continuously Hello messages, waiting for
the agents on the managed VMs to respond. A complete HM to VMM handshake flow is
detailed in “HM-VMM communication establishment”.
HM-VMM communication establishment

We covered in “VM Agent activation” on page 164 the blind startup sequence of the VMM
daemon for the case of an isolated VM when VM monitoring is not enabled at the KSYS level
and the VM does not have the HA client adapters created. We saw in that case how VMM
enters a steady state loop where it is looking continuously for the HA client adapters on which
to listen for Hello messages.
Here we present the case where the VMM agent is deployed on a VM enabled for monitoring
and discovered by KSYS after enablement, as described in 2.3.1, “Discovery of a VM that is
enabled for monitoring” on page 203. The focus here is on the handshake protocol between
the VMM and the HMs, but we are also interested in the HSDB updates about the VM status.
The running HMs on the VIOS side are broadcasting repeatedly Hello messages and waiting
for the possible agents on the managed VMs to respond, as described in “HM activation” on
page 150.
In Example 2-190, we filter the output of the procstack command on our test VM, vmraix1, so
that we can easily identify later the thread type for each TID.
Example 2-190 VMM threads for the HM to VMM communication establishment scenario
# uname -uL
IBM,02212424W 5 vmraix1
[vmraix1:root:/var/ksys/log/hmvmmhs:] lssrc -s ksys_vmm
# procstack 10682746|egrep "tid|main|threadMain"
---------- tid# 17563989 (pthread ID: 1) ----------
0x10007e9c main(??, ??) + 0xf1c
---------- tid# 19202453 (pthread ID: 1286) ----------
0x100e6110 UnixSocket::threadMain()(??) + 0x50
---------- tid# 20513041 (pthread ID: 1029) ----------
0x102cc564 VmMonitor::threadMain()(??) + 0x3c4
---------- tid# 27918627 (pthread ID: 2828) ----------
0x1007185c HvncpSender::threadMain()(??) + 0x8fc
---------- tid# 23724513 (pthread ID: 2571) ----------
0x10083e24 HvncpReceiver::threadMain()(??) + 0x6a4
---------- tid# 17826247 (pthread ID: 2314) ----------
0x1008ee44 VlanCommunication::threadMain()(??) + 0x44
---------- tid# 17760557 (pthread ID: 2057) ----------
0x1007185c HvncpSender::threadMain()(??) + 0x8fc
---------- tid# 24183173 (pthread ID: 1800) ----------

0x10083e24 HvncpReceiver::threadMain()(??) + 0x6a4
---------- tid# 21168565 (pthread ID: 1543) ----------
0x1008ee44 VlanCommunication::threadMain()(??) + 0x44
---------- tid# 18940395 (pthread ID: 772) ----------
0x100bb870 VmNetIntfHandler::threadMain()(??) + 0x2b0
---------- tid# 22806881 (pthread ID: 515) ----------
0x10263d04 AppReport::threadMain()(??) + 0x1a4
---------- tid# 19792237 (pthread ID: 258) ----------
#
The initial steps of the VMM daemon startup sequence until the scan for the Virtual Ethernet
adapters are omitted in Example 2-191 because the steps are similar the ones that are
described in “VM Agent activation” on page 164. For the case we present here, the prior host
group discovery created the HA client Virtual Ethernet adapters at the LPAR container level,
as shown in Example 2-175 on page 203. So, the VmNetIntfHandler thread (pTID 772) can
identify its counterpart logical devices at the operating system level, ent1 and ent2, among
the other scanned adapters.
Example 2-191 Discovery of the Virtual Ethernet adapters for HM to VMM communication
# uname -uL; pwd
/var/ksys/log
# more ksys_vmm.debug
[START 10682746 1 05/14/19-01:50:26
...
Command: lsdev -c adapter | grep 'ênt'> /var/ksys/log/cmdop_10682746_1 2>&1,
Command output:
ent0 Available Virtual I/O Ethernet Adapter (l-lan)
, returnCode: 0
[3 10682746 1 05/14/19-01:50:26 HACommon.C:hacommon::runSystemCommand(string) : 117]
Execution of command 'lsdev -c adapter | grep 'ênt' | grep -w Defined | awk '{print $1
}' > /var/ksys/log/interfaceList.txt' succeeded (rc=0).
[2 10682746 1 05/14/19-01:50:26 VmNetIntfHandler.C:VmNetIntfHandler::startThread(): 75]
Starting VmNetIntfHandler thread.
[3 10682746 1 05/14/19-01:50:26 VmMonitor.C:VmMonitor::check_and_set_crit_VG_at...: 1636]
get CRITICAL VG value: LC_ALL=C /usr/sbin/lsvg rootvg | /usr/bin/grep "CRITICAL VG:" |
/usr/bin/sed 's/.*CRITICAL VG:[ ]*//g' 2>/dev/null
[3 10682746 772 05/14/19-01:50:26 VmNetIntfHandler.C:VmNetIntfHandler::selectFD() :
125] Found 7 possible interfaces to listen (not filtered).
[3 10682746 772 05/14/19-01:50:26 VmUtils.C:vmutils::getNetIntf(int, list<std::ba...:
253] intfAlias=en0
254] intfName=ent0
277] Interface en0 has IP address, discard it.
253] intfAlias=en1
254] intfName=ent1
265] Interface en1 has no IP address, keep it.

253] intfAlias=en2
254] intfName=ent2
265] Interface en2 has no IP address, keep it.
[3 10682746 772 05/14/19-01:50:26 VmUtils.C:vmutils::bindNetIntfName(string) :
454] Successful bind on interface ent1 (fd returned 12)
175] Interface ent1 : Now listening to find a HostMonitor.
182] Successful bind on interface ent1 . Listening for 'Hello' message.
#
The child interfaces, en1 and en2, have no IP configured, so ent1 and ent2 are good
candidates to be the wanted HA client adapters. On the contrary, en0 has an IP configured, so
it is discarded.
The last three records in Example 2-191 on page 216 show how the same thread (pTID 772)
binds to the ent1 interface to listen for expected Hello packets coming from the counterpart
HM on the VIOS side. Though not shown, the same happens for the identified ent2 interface
and the other HM instance on the second VIOS of the host.
In Example 2-192, we collected the Ethernet connectivity details for the HA client adapters of
the considered VM, vmraix1, and also for their counterpart VIOS trunk adapters. These
details are mentioned here to help with easier identification of the communication endpoints
for the handshake protocol we present.
Example 2-192 HA client adapters and counterpart VIOS trunk adapters

# uname -uL
# entstat -d ent1|egrep "^Hardware Address|^Switch ID|^Port VLAN ID"
Hardware Address: 6a:6a:0f:fd:f7:07
Port VLAN ID: 101
# entstat -d ent2|egrep "^Hardware Address|^Switch|^Port VLAN ID"
Hardware Address: 6a:6a:0f:fd:f7:08
Port VLAN ID: 102
#
-------------------------
# uname -uL; entstat -d ent22|egrep "^Hardware Address|^Switch|^Port VLAN ID"
Hardware Address: 6a:6a:08:84:ee:11
Port VLAN ID: 101
Switch Mode: VEPA
#
-------------------------
# uname -uL; entstat -d ent22|egrep "^Hardware Address|^Switch|^Port VLAN ID"
IBM,02212424W 2 usaxvia063ccpx1
Hardware Address: 6a:6a:01:58:1f:11
Port VLAN ID: 102

Switch Mode: VEPA
#
The VLAN ID of the ent1 interface on vmraix1 is 101, so the Hello packets that are expected
on ent1 are sent from the VIOS that is connected to the same VLAN, that is,
usaxvib063ccpx1. The excerpt that is shown in Example 2-193 from the HM log on the
usaxvib063ccpx1 VIOS shows the Hello packets that are broadcast during the time interval of
interest, once every 30 seconds. Clocks on the client VM and VIOSs are synced by the NTP
protocol for easier examination of the sequencing of events.
Each Hello packet contains as payload a specific identifier string that is generated randomly
at the HM level based on the system clock. This string is used by the VMM to identify the
originator HM of the received messages. If the HM restarts for any reason, then a new and
unique identifier is generated for this identification purpose. In the case of an already
established session, the VMM notices the new identity of the counterpart communication
endpoint, drops the current session, and triggers a new protocol handshake sequence.
Example 2-193 Hello broadcast records in the HM log on the VIOS side
# uname -uL
# alog -f /home/ios/logs/health/host_monitor.log -o > host_monitor.log.hmvmmhandshake
# more host_monitor.log.hmvmmhandshake
...
[3 16122210 91554293 05/14/19-01:50:13.351 Processing.C:Processing::sendMessage(bool, stri...:
682] Broadcasting 'Hello' message with identifier '1556303553.811512000.38'
[3 16122210 89719261 05/14/19-01:50:13.351 HvncpSender.C:HvncpSender::threadMain() :
814] Sending Packet to addr: FFFFFFFFFFFF
[3 16122210 89719261 05/14/19-01:50:13.351 HvncpPacket.C:HvncpPacket::display() const :
292] +Packet Frame (23 bytes) 1556303553.811512000.38
833] Removing message to VmMon at macAddr 'FFFFFFFFFFFF' from send queue. Reason:
Fully sent
292] +Packet Frame (84 bytes) uuid=61be20f0-09a8-4753-b5e5-564f8a545ed5:ostype=AI
X:verVmmLower=2.0:verVmmUpper=2.0
[3 16122210 77726013 05/14/19-01:50:43.796 HvncpReceiver.C:HvncpReceiver::threadMain() :
390] Received Packet from macAddr '6A6A0FFDF707'.
292] +Packet Frame (84 bytes) uuid=61be20f0-09a8-4753-b5e5-564f8a545ed5:ostype=AI
X:verVmmLower=2.0:verVmmUpper=2.0
411] Received SYN packet 1 from macAddr '6A6A0FFDF707'.
host_monitor.log.hmvmmhandshake (99%)
#

Two Hello messages with the same 1556303553.811512000.38 identifier value (23-byte
payload) are shown in Example 2-193 on page 218 as sent by the same HM daemon at
01:50:13.351 and 01:50:43.339. Note the 30-second interval. The VmNetIntfHandler thread
(pTID 772) is created in between, around 01:50:26, as shown in Example 2-191 on page 216,
and starts shortly afterward to look repeatedly for interfaces on which to listen for Hello
messages every 5 seconds. So, the first Hello packet that is has a chance to receive is that
one that is sent at 01:50:43.339. Around 01:50:43, after receiving this first Hello packet, the
VmNetIntfHandler thread performs with the VmMonitor thread (pTID 1029) the whole
sequence of actions that is shown in Example 2-194.
Example 2-194 Receiving the Hello packet on the VMM side and sending back the SYN packet
...
[3 10682746 772 05/14/19-01:50:43 VmNetIntfHandler.C:VmNetIntfHandler::checkMessa...:
387] Ethertype 1536
[3 10682746 772 05/14/19-01:50:43 VmNetIntfHandler.C:VmNetIntfHandler::checkMessa...:
402] Received a 'Hello' packet from macAddr '6A6A0884EE11' with helloIdentifier
'1556303553.811512000.38'.
[3 10682746 772 05/14/19-01:50:43 VmNetIntf.C:VmNetIntf::getIntfName() const :
58] getIntfName()
300] Interface ent1 : Found a HostMonitor with MacAddr '6A6A0884EE11'
302] Interface ent1 : Now used by the VmMonitor and won't be listened anymore
[3 10682746 772 05/14/19-01:50:43 VmNetIntf.C:VmNetIntf::~VmNetIntf() :
45] ~VmNetIntf()
[3 10682746 772 05/14/19-01:50:43 VmNetIntf.C:VmNetIntf::~VmNetIntf() :
45] ~VmNetIntf()
[3 10682746 772 05/14/19-01:50:43 VmNetIntfHandler.C:VmNetIntfHandler::threadMain() :
461] New HostMonitor '6A6A0884EE11' is found
[3 10682746 772 05/14/19-01:50:43 VmNetIntfHandler.C:VmNetIntfHandler::threadMain() :
466] Pipe is being establshed between Interface 'ent1' and HostMonitor '6A6A0884EE11'
[3 10682746 772 05/14/19-01:50:43 ethernet.C:configureEthernetSocketAddr(socketHa...:
255] +Socket '12' parameters (SNDD_8022)
256] | family : 23
257] | len : 36
258] | filtertype : 4
259] | ethertype : 1536
260] | filterlen : 12
261] | nddname : ent1
262] +
304] Found interface 'ent1' : macAddr '6A6A0FFDF707'.
[3 10682746 772 05/14/19-01:50:43 VmMonitor.C:VmMonitor::addAdapter(string &, str...:
767] Interface ent1 : successfully connected to VmMonitor.
[2 10682746 1029 05/14/19-01:50:43 VmMonitor.C:VmMonitor::threadMain() :
1289] Connect to the found HostMonitor at macAddr '6A6A0884EE11'.

[3 10682746 1029 05/14/19-01:50:43 VmMonitor.C:VmMonitor::threadMain() :
1311] Network controller successfully started for macAddr '6A6A0884EE11'.
[3 10682746 1029 05/14/19-01:50:43 HvncpSender.C:HvncpSender::addCommTableEntry(co...:
300] Sender opened communication table with macAddr '6A6A0884EE11'.
[3 10682746 1029 05/14/19-01:50:43 HvncpReceiver.C:HvncpReceiver::linkCommTables(c...:
275] Receiver opened communication table with macAddr '6A6A0884EE11'.
[2 10682746 2057 05/14/19-01:50:43 HvncpSender.C:HvncpSender::revokeSendSyn(const ...:
401] Delete any communication pending to 6A6A0884EE11 (except discovery)
[3 10682746 2057 05/14/19-01:50:43 HvncpSender.C:HvncpSender::sendSyn(const string...:
523] Send SYN packet to 6A6A0884EE11
294] +Packet Frame (84 bytes)
uuid=61be20f0-09a8-4753-b5e5-564f8a545ed5:ostype=AIX:verVmmLower=2.0:verVmmUpper=2.0
A socket is configured on the ent1 interface that has the MAC address 6A6A0FFDF707 to
establish a pipe with the originator HM of the received Hello message, and acts at the remote
counterpart VIOS endpoint interface that is identified by the MAC address 6A6A0884EE11. A
network controller starts for this pipe on the VMM side by creating three threads:
HvncpSender, VlanCommunication, and HvncpReceiver with pTIDs 2057, 1543, and 1800. The
pTID values are confirmed by the final log entries in Example 2-194 on page 219 and the
subsequent entries in Example 2-197 on page 222.
This triplet takes over the communication and an initial synchronization SYN packet is sent by
the HvncpSender thread to the HM endpoint at the address 6A6A0884EE11 as a reaction to the
received Hello packet. The payload of this initial SYN packet has 84 bytes and contains
various VMM details, among which is the VM UUID with a value of
61be20f0-09a8-4753-b5e5-564f8a545ed5, as shown in Example 2-194 on page 219.
On the HM side, we have a similar thread triplet setup handling the HM to VMM
communication at the trunk adapter endpoint. In Example 2-195, note this triplet among all
the HM daemon threads: the HvncpSender thread, the HvncpReceiverer thread, and the
VlanCommunication thread, with kernel thread IDs (tid#) 89719261, 77726013, and
91881983.
Example 2-195 HM threads for HM to VMM communication establishment scenario

# uname -uL
# lssrc -s ksys_hsmon
# procstack 16122210|egrep "tid|main|threadMain"
---------- tid# 91029851 (pthread ID: 1) ----------
0x100013c8 main(0x2, 0x2ff22d60) + 0xc88
---------- tid# 91095433 (pthread ID: 2057) ----------
---------- tid# 91554293 (pthread ID: 1800) ----------
---------- tid# 90243567 (pthread ID: 1543) ----------
---------- tid# 91619721 (pthread ID: 1286) ----------
---------- tid# 89719261 (pthread ID: 1029) ----------

---------- tid# 77726013 (pthread ID: 772) ----------
---------- tid# 91881983 (pthread ID: 515) ----------
---------- tid# 82969031 (pthread ID: 258) ----------
#
The SYN packet that is sent to the 6A6A0884EE11 MAC address from the vmraix1 VM MAC
address 6A6A0FFDF707 is received as “SYN packet 1” by the HvncpReceiver thread on the
usaxvib063ccpx1 VIOS HM side, as shown by the first record in Example 2-196.
Example 2-196 Receiving the SYN packet on the HM side and replying with the SYN ACK and SYN packets
...
411] Received SYN packet 1 from macAddr '6A6A0FFDF707'.
444] SYN begin, invalidate the previous communcaiton.On the sender. mac: 6A6A0FFDF707.
[3 16122210 77726013 05/14/19-01:50:43.796 Processing.C:Processing::notify(string, HvncpMe...:
384] Received a SYN payload
'uuid=61be20f0-09a8-4753-b5e5-564f8a545ed5:ostype=AIX:verVmmLower=2.0:verVmmUpper=2.0'
398] VmMonitor '6A6A0FFDF707' connection> HostMonitor is now able to receive messages
...
425] +
[2 16122210 89719261 05/14/19-01:50:43.797 HvncpSender.C:HvncpSender::revokeSendSyn(const ...:
401] Delete any communication pending to 6A6A0FFDF707 (except discovery)
[3 16122210 89719261 05/14/19-01:50:43.797 HvncpSender.C:HvncpSender::sendSyn(const string...:
523] Send SYN packet to 6A6A0FFDF707
726] Send SYN acknowledgment 1 to 6A6A0FFDF707
814] Sending Packet to addr: 6A6A0FFDF707
292] +Packet Frame (58 bytes) uuid=5c3196a3-c41f-4b5c-9861-dee6077081c6:verHmRunning=2.0

The HM replies with a SYN packet (58-byte payload with various HM details) that is sent by the
HvncpSender thread (TID# 89719261) at 01:50:43.797 (Example 2-196 on page 221). It is
followed by a 0-byte payload SYN acknowledgment 1 packet that is sent by the same
HvncpSender thread, and both are received on the VMM side (pTIDs 1543 and 1800), as
Example 2-197 Receiving SYN ACK and SYN packets on the VMM side and replying with SYN ACK
...
uuid=61be20f0-09a8-4753-b5e5-564f8a545ed5:ostype=AIX:verVmmLower=2.0:verVmmUpper=2.0
[3 10682746 1800 05/14/19-01:50:43 HvncpReceiver.C:HvncpReceiver::threadMain() :
394] Received Packet from macAddr '6A6A0884EE11'.
415] Received SYN packet 9 from macAddr '6A6A0884EE11'.
[3 10682746 1800 05/14/19-01:50:43 VmMonitor.C:VmMonitor::notify(string, HvncpMess...:
1121] Received a SYN payload 'uuid=5c3196a3-c41f-4b5c-9861-dee6077081c6:verHmRunning=2.0'.
1127] HostMonitor '6A6A0884EE11' connection> VmMonitor is now able to receive messages.
1141] vmm major version '2' and HM major version '2' matches, ignoring vmm minor version '0' and
HM minor version '0' processing packet
1155] +Parameters of the HostMonitor being connected:
1156] | MacAddr : 6A6A0884EE11
1157] | UUID : 5c3196a3-c41f-4b5c-9861-dee6077081c6
1158] | HsMon running version : 2.0
1159] +VmMon <> HsMon new relation status :
1160] | Version compatibility : YES
1164] | VmMon running version : 2.0
1166] +
469] Received SYN acknowledgment from macAddr '6A6A0884EE11'.
470] Notify main thread when comm is fully established.

631] Communication pipe to 6A6A0884EE11 is now opened
676] Deleting the packet from commWindow. src: 6A6A0884EE11.
726] Send SYN acknowledgment 9 to 6A6A0884EE11
815] HostMonitor '6A6A0884EE11' connection> VmMonitor is now able to send messages
294] +Packet Frame (8 bytes) START_HB
976] Received message 'START_HB' from HostMonitor '5c3196a3-c41f-4b5c-9861-dee6077081c6'.
The VlanCommunication thread (pTID 1543) on the VMM side receives the frames and passes
them to the HvncpReceiver thread (pTID 1800). The communication pipe with the HM is now
opened from the VMM perspective, and a final SYN acknowledgment 9 is sent to the HM as a
reply to the 0-byte payload SYN packet 9 received. At this moment, the VMM expects a
START_HB message to start sending heartbeats to the HM at the other end of the pipe.
In Example 2-198, the HM receives the SYN ACK packet and the communication pipe is now
opened also from this HM perspective, so the HM further checks and updates the VM status
in the HSDB by using a vioHmVmDiscover() call.
Example 2-198 Receiving SYN ACK on the HM side and replying with START_HB
...
465] Received SYN acknowledgment from macAddr '6A6A0FFDF707'.
466] Notify main thread when comm is fully established.
631] Communication pipe to 6A6A0FFDF707 is now opened
676] Deleting the packet from commWindow. src: 6A6A0FFDF707.
268] VmMonitor '6A6A0FFDF707' connection> HostMonitor is now able to send messages
271] HM Running version: 2.0
VM Running version: 2.0

273] Pushing to the SYN queue for processing.
[3 16122210 77726013 05/14/19-01:50:43.799 Processing.C:ProcessingMsg::pushMessageToQueue(...:
1345] INFO: Pushed SYN to queue. Waking up thread.
[3 16122210 82969031 05/14/19-01:50:43.799 Processing.C:ProcessingMsg::threadMain() :
1395] INFO: Got a SYN packet!
[2 16122210 82969031 05/14/19-01:50:43.799 DatabaseAccess.C:DatabaseAccess::vmDiscovered(s...:

304] Called vioHmVmDiscover interface for VM 61be20f0-09a8-4753-b5e5-564f8a545ed5 (vers
ion matching)
305] VM OS Type : AIX
308] VmMon Version range : 2.0 - 0.0
310] VmMon Running version : 2.0
314] Called vioHmVmDiscover rc=0.
505] HostMonitor is asked to START monitoring VM '61be20f0-09a8-4753-b5e5-564f8a545ed5' (command
from SSP Database)
183] Start monitoring VM '61be20f0-09a8-4753-b5e5-564f8a545ed5'.
193] VM '61be20f0-09a8-4753-b5e5-564f8a545ed5' is already tracked. Restart monitoring.
547] Monitoring of VM '61be20f0-09a8-4753-b5e5-564f8a545ed5' requires sending a command to start
Heartbeats to the VmMonitor
671] Sending message 'START_HB' to VmMonitor of VM '61be20f0-09a8-4753-b5e5-564f8a545ed5' at
MacAddr '6A6A0FFDF707'.
The vioHmVmDiscover() callback (performed by ProcessingMsg TID# 82969031) enters at

01:50:43.799 and returns successfully at 01:50:43.828. Its main input parameter must be the
VM UUID received in the initial SYN packet from the VMM. To see what it is doing at the HSDB
level, we isolated all the related actions that were performed during the callback as they were
registered in the database log (Example 2-199).
Example 2-199 Excerpt from HSDB ProgreSQL log for a vioHmVmDiscover callback
2019-05-14 01:50:43.727 CUT|5cda1ef3.16801a2|LOG: duration: 0.018 ms
2019-05-14 01:50:43.810 CUT|5cda1ef3.16801a4|LOG: statement: BEGIN;SET search_path TO vios
2019-05-14 01:50:43.811 CUT|5cda1ef3.16801a4|LOG: statement: BEGIN;SELECT msid FROM vioshs.vios
where viosuuid='5c3196a3-c41f-4b5c-9861-dee6077081c6'
2019-05-14 01:50:43.818 CUT|5cda1ef3.16801a4|LOG: statement: SAVEPOINT _EXEC_SVP_31256208;BEGIN
WORK; LOCK TABLE vioshs.locked IN EXCLUSIVE MODE NOWAIT
2019-05-14 01:50:43.818 CUT|5cda1ef3.16801a4|WARNING: there is already a transaction in
progress
2019-05-14 01:50:43.818 CUT|5cda1ef3.16801a4|LOG: statement: RELEASE
_EXEC_SVP_31256208;SAVEPOINT _EXEC_SVP_31256208;SELECT viosuuid, tid, count, CAST ((EXTRACT(
epoch FROM CLOCK_TIMESTAMP()::timestamp ) - EXTRACT( epoch FROM
vioshs.locked.lockts::timestamp)) AS INT) FROM vioshs.locked
_EXEC_SVP_31256208;SAVEPOINT _EXEC_SVP_31256208;UPDATE vioshs.locked SET viosuuid
='5c3196a3-c41f-4b5c-9861-dee6077081c6', tid=82969031, count=1, lockTs=NOW()

2019-05-14 01:50:43.821 CUT|5cda1ef3.16801a4|LOG: statement: COMMIT
2019-05-14 01:50:43.822 CUT|5cda1ef3.16801a4|LOG: statement: BEGIN;SELECT vmuuid, name, msid,
missHistory, state, up_major, up_minor, lo_major, lo_minor, shortMsg, missHistoryTS, suspendTS
FROM vioshs.vm where vmuuid='61be20f0-09a8-4753-b5e5-564f8a545ed5'
2019-05-14 01:50:43.825 CUT|5cda1ef3.16801a4|LOG: statement: SAVEPOINT
_EXEC_SVP_31256208;UPDATE vioshs.vm SET state=3 WHERE
vmuuid='61be20f0-09a8-4753-b5e5-564f8a545ed5'
_EXEC_SVP_31256208;SAVEPOINT _EXEC_SVP_31256208;INSERT INTO vioshs.trans(vmuuid, viosuuid, msid,
tag, opcode, state, data, integer, txstarted)
VALUES('61be20f0-09a8-4753-b5e5-564f8a545ed5','5c3196a3-c41f-4b5c-9861-dee6077081c6',
-5878939440565042805, 0, 4, 1, 'old=8:new=3', 0, NOW())
_EXEC_SVP_31256208;SAVEPOINT _EXEC_SVP_31256208;SELECT count(1) from vioshs.map where
vmuuid='61be20f0-09a8-4753-b5e5-564f8a545ed5' AND
viosuuid='5c3196a3-c41f-4b5c-9861-dee6077081c6'
_EXEC_SVP_31256208;SAVEPOINT _EXEC_SVP_31256208;UPDATE vioshs.map SET vmHbMissed=NULL WHERE
vmuuid='61be20f0-09a8-4753-b5e5-564f8a545ed5' AND
2019-05-14 01:50:43.827 CUT|5cda1ef3.16801a4|LOG: statement: BEGIN;BEGIN WORK; LOCK TABLE
vioshs.locked IN EXCLUSIVE MODE NOWAIT
2019-05-14 01:50:43.827 CUT|5cda1ef3.16801a4|WARNING: there is already a transaction in
progress
2019-05-14 01:50:43.827 CUT|5cda1ef3.16801a4|LOG: statement: SAVEPOINT
_EXEC_SVP_31256208;SELECT viosuuid, tid, count FROM vioshs.locked
_EXEC_SVP_31256208;SAVEPOINT _EXEC_SVP_31256208;UPDATE vioshs.locked SET tid=0, count=0,
lockTs=NULL
2019-05-14 01:50:43.831 CUT|5cda1ef3.14e01f2|LOG: statement: SET DateStyle = 'ISO';SET
extra_float_digits = 2;show transaction_isolation
Only the 5cda1ef3.16801a4 ProgreSQL session logged records during that interval, so we are
sure that they were performed by our callback run. The relevant actions that we observe there
are as follows:
1. Retrieve the entry in the vm table where the vmuuid field is equal to the VM UUID that was
passed by the VMM in the initial SYN packet.
2. Update the state field of the row corresponding to the identified VM in the vm table to a
value of 3 (STARTED).
3. Update the vmHbMissed field to NULL for the entry in the map table corresponding to our VM
and involved VIOS (WHERE vmuuid='61be20f0-09a8-4753-b5e5-564f8a545ed5' AND
viosuuid='5c3196a3-c41f-4b5c-9861-dee6077081c6').

Immediately after the return from the vioHmVmDiscover callback, the same HM ProcessingMsg
thread (TID# 82969031) sends a START_HB message to its VMM side through the established
pipe (see the last five entries in Example 2-198 on page 223). This message is received on
the VMM side, as shown in the last four entries of Example 2-197 on page 222, and we
continue in Example 2-200 with the analysis of the VMM log.
Example 2-200 Receiving the START_HB message on the VMM side

976] Received message 'START_HB' from HostMonitor '5c3196a3-c41f-4b5c-9861-dee6077081c6'.
985] ALL apps start stablization time : 0
986] Initial DBELL will be send after 4 HB cycle's
993] Will now send Heartbeats to HostMonitor '5c3196a3-c41f-4b5c-9861-dee6077081c6' every 10.00
seconds.
1007] Ethname:ent1, vmAdr:6A6A0FFDF707, hmAdr:6A6A0884EE11
1009] Sending Config data to vmmKE
726] Send acknowledgment 9 to 6A6A0884EE11
125] Found 7 possible interfaces to listen (not filtered).
VMM reacts to the received START_HB message by setting up its scene for sending the
heartbeat to the HM every 10 seconds. The kernel extension is also informed about this setup
for the case that it must take over. We are also informed that an initial AR flow is going to be
triggered by sending a DBELL frame, but this topic is covered in 2.3.3, “Application Reporting
flow” on page 262, so we skip it for now. The cycle is closed by sending an acknowledgment
(Packet Frame (0 bytes)) back to the HM, which is received and processed in turn by the HM
communication thread triplet, as shown in Example 2-201.
Example 2-201 Receiving the first VMM heartbeats on the HM side

671] Sending message 'START_HB' to VmMonitor of VM '61be20f0-09a8-4753-b5e5-564f8a545ed5' at
MacAddr '6A6A0FFDF707'.

676] Deleting the packet from commWindow. src: 6A6A0FFDF707.
296] An acknowledgment has been received. Check for sent messages status.
833] Removing message to VmMon at macAddr '6A6A0FFDF707' from send queue. Reason: Fully sent
[2 16122210 77726013 05/14/19-01:50:46.580 VmHealthStatusMgr.C:VmHealthStatusMgr::setVmDat...:
424] VM '61be20f0-09a8-4753-b5e5-564f8a545ed5' state changed to DETECTED.
[2 16122210 91554293 05/14/19-01:50:48.855 DatabaseAccess.C:DatabaseAccess::writeToDatabase():
603] VM '61be20f0-09a8-4753-b5e5-564f8a545ed5' state updated in Database to : DETECTED
616] vioHmHbDetected returned with rc = 0
Shortly after receiving the SEND_HB acknowledgment, the HM also receives at 01:50:46.580
(Example 2-201 on page 226) the first VMM heartbeat (Packet Frame (3 bytes) HB) and
updates the HSDB by using a vioHmHbDetected call. Subsequent VMM heartbeats are
periodically received every 10 seconds and logged in an interlaced manner with already
known entries of HM heartbeat updates to the HSDB and Hello broadcasts toward VMMs,
both every 30 seconds, in a kind of a steady state pattern.

On the VMM side, Example 2-202, we can see how these VM heartbeats are sent and logged
interlaced with records about received Hello broadcasts in a similar steady state manner.
Example 2-202 Sending the heartbeat on the VMM side

[3 10682746 1029 05/14/19-01:50:46 VmMonitor.C:VmMonitor::doHeartbeatJobCycle() :
1395] Send Heartbeat to MacAddr '6A6A0884EE11'.
...
[3 10682746 1029 05/14/19-01:51:16 VmMonitor.C:VmMonitor::doHeartbeatJobCycle() :
1395] Send Heartbeat to MacAddr '6A6A01581F11'.
...
...
...

394] Received Packet from macAddr '6A6A01581F11'.
...
We removed records from the log excerpt in Example 2-202 on page 228 to shorten the
output. The listing was still long because the VMM log contains the interlaced records about
messages that are exchanged both with the HM instance listening at the 6A6A0884EE11
address (which we used so far in our analysis) and with the other HM instance listening at the
6A6A01581F11 address on the partner VIOS. The removed records were mainly traces of the
similar handshake sequence that was performed by our VMM with this second HM on the
partner VIOS. The VM heartbeat and Hello broadcast messages that were exchanged with
our HM instance (listening at the 6A6A0884EE11 address) are marked in bold and italic typeset
for easier identification.
Example 2-203 shows the HSDB vm, map, and trans tables in this steady state situation after
successful HM to VMM communication establishment.
Example 2-203 The vm, map, and trans tables in HSDB after successful HM to VMM communication establishment
vios=# select vmuuid,name,msid,state from vm;
vmuuid | name | msid | state
--------------------------------------+------------+----------------------+-------
61be20f0-09a8-4753-b5e5-564f8a545ed5 | | -5878939440565042805 | 3
| default_vm | | 1
(2 rows)
vmuuid | viosuuid | misshistory |
hmrecovery | viosrecovery | vmhbmissed
--------------------------------------+--------------------------------------+-------------+----
--------+--------------+------------
61be20f0-09a8-4753-b5e5-564f8a545ed5 | 5c3196a3-c41f-4b5c-9861-dee6077081c6 | 0|
0 | 0 |
61be20f0-09a8-4753-b5e5-564f8a545ed5 | 0ab5001f-2c3e-4fbb-97ea-41d3f8c4f898 | 0|
0 | 0 |
(2 rows)
--------+----------+------+-----+--------+-------+------+---------+-----------
(0 rows)
vios=#

The state value for our VM is STARTED (3) and the vmhbmissed value is NULL, as expected. It
is the happy case state in which a VM spends most of its lifecycle. It is retrieved from the
HSDB by a typical steady state NeedsAttention request that originates on the KSYS side.
Example 2-204 illustrates the request and reply payloads for a typical happy case steady
state NeedsAttention request that is issued periodically by the FDE thread. We go to the
VIOS that is designated by the FDE thread for polling and issue a manual request.
Example 2-204 Happy case steady state NeedsAttention request and reply
# ksysmgr trace log=fde | grep -i polling|tail -1
[00] 08/25/19 T(34c) _VMR 18:16:37.891076 DEBUG FDEthread.C[208]: Use VIOS usaxvib063ccpx1:
5C3196A3-C41F-4B5C-9861-DEE6077081C6 for polling
#
--------
# uname -uL
# cat na.xml
</VIO>
<VIO><Response>
<needsAtt>
<vmStatList>
</vmList>
</vmStatList>
</needsAtt>
</Response></VIO>
#
Checking the KSYS requests that are logged on the designated VIOS around the session
establishment moment, we notice two NeedsAttention requests followed by acknowledgment
requests and also a VIO_HS_QUERY_MSG/VIO_HS_GET_MSG messaging sequence, as shown in
Example 2-205.
Example 2-205 KSYS requests for the session establishment moment

# alog -f /home/ios/logs/viosvc.log -o > viosvc.log.VMstarted.txt
# grep VIO_HS_ viosvc.log.VMstarted.txt|more
<Request action_str="VIO_HS_QUERY_MSG">
<Request action_str="VIO_HS_QUERY_MSG">
<Request action_str="VIO_HS_GET_MSG">
Standard input: END

The reference /usr/lib/vioHADR2.00.xsd VIOS file provides the following details about the
VIO_HS_NEEDS_ATTENTION and VIO_HS_ACKNOWLEDGE requests:
- VIO_HS_NEEDS_ATTENTION: KSYS uses this request on a poll period. It returns a

list VMs that have missed a heartbeat and or have an application request pending
that has not been acknowledged by KSYS
- VIO_HS_ACKNOWLEDGE: This is sent by KSYS as an acknowledgment that it has
received and successfully saved the information regarding a VM. VIOS will not
return status for a VMs after the acknowledgment until there is a further change
for that VM. VIO_HS_ACKNOWLEDGE is not applied to missed heartbeat however it is
applied to the following:
state transitions
application requests
...
We use these details to comment on the typical NeedsAttention request in Example 2-204 on
page 230 and on those two acknowledged NeedsAttention requests showing up around the
session establishment moment. Various other Need Attention request cases with their
specific response payloads are covered in 2.3.2, “Failure Detection Engine: QuickQuery and
NeedsAttention flows” on page 234.
So, coming back to the typical happy case NeedsAttention request in Example 2-204 on
page 230, VIOS returns continuously this kind of short payload with no VM status, which
means that no heartbeat was missed and no VM status change happened since the last
acknowledged NeedsAttention request. It happens periodically until there is a further change
for that VM, such as a missed heartbeat, a VM state transition, or an application-related
change.
Example 2-206 and Example 2-207 on page 232 shows occurrences of the last two cases.
Example 2-206 Acknowledged NeedsAttention request for VM state transition

# more viosvc.log.VMstarted.txt
...
version="1.0"?>
author="LIBKREST" title="Req Needs Attention">
</VIO>
]
[0 16712162 65274117 08/25/19-18:55:18.742 viosvc_res.c 1.26 464]
<needsAtt hsTag="-1234658345958918737">
<vmStatList>
<vmStat uuid="61be20f0-09a8-4753-b5e5-564f8a545ed5"><VM state="STARTED" stateChg='1' >
</VM>
</vmStat>
</vmList>
</vmStatList>
</needsAtt>
</Response></VIO>
]

version="1.0"?>
author="LIBKREST" title="acknowledgment to VIOS">
</VIO>
]
The stateChg attribute in Example 2-206 on page 231 is documented in the same reference
file, /usr/lib/vioHADR2.00.xsd:
...
stateChg: This attribute when present indicates that this VM is undergoing a state
change and state field will indicate the current state. When this attribute is
present, the payload may include a tag will need to be acknowledged. that has been
discovered.
...
Similarly in Example 2-207, we have the case of an application-related change. In such a

situation, the VMM makes the KSYS aware through a so-called DBELL or short message
notification that something happened at the application level. After the notification is received
on the KSYS side, an AR flow is initiated to obtain more details about what has happened.
The shortMsg attribute conveys the initial hardcoded DBELL in Example 2-200 on page 226 as
generated after the first four heartbeats that are sent by VMM after a successful HM to VMM
handshake. The VIO_HS_QUERY_MSG/VIO_HS_GET_MSG messaging sequence in Example 2-205
on page 230 is part of this AR flow. The AR topic is covered in 2.3.3, “Application Reporting
Example 2-207 Acknowledged NeedsAttention request for VM application-related change

# more viosvc.log.VMstarted.txt
...
version="1.0"?>
author="LIBKREST" title="Req Needs Attention">
</VIO>
]
[0 16712178 65339731 08/25/19-18:56:23.426 viosvc_res.c 1.26 464]
<needsAtt hsTag="1735421413719889094">
<vmStatList>
<vmStat uuid="61be20f0-09a8-4753-b5e5-564f8a545ed5"><VM state="STARTED" shortMsg="0x2" >
</VM>
</vmStat>
</vmList>
</vmStatList>

</needsAtt>
</Response></VIO>
]
version="1.0"?>
author="LIBKREST" title="acknowledgment to VIOS">
</VIO>
]
Example 2-208 shows the case status of our VM at the KSYS level.
Example 2-208 Case monitored VM status at the KSYS level

# lsrsrc -s 'Name="vmraix1"' IBM.VMR_LPAR|grep -v LCB|egrep -i "state|hb"
HMstate = "STARTED"
HBmissed = 0
HAState = 4
VerifyState = {["33613bcd-7ca1-3558-874a-c1e1d3ceee32","PASS",""]}
PartitionState = "running"
RmcState = "active"
MigrationState = ""
RRestartState = ""
HMstateHM1 = ""
HBmissedHM1 = 0
HMstateHM2 = ""
HBmissedHM2 = 0
# ksysmgr -v q vm vmraix1
Name: vmraix1
UUID: 61BE20F0-09A8-4753-B5E5-564F8A545ED5
State: READY
Host: Server-8408-E8E-SN212424W
Priority: Medium
Heartbeating_with: usaxvib063ccpx1
usaxvia063ccpx1
HA_monitor: enable
Homehost: Server-8408-E8E-SN212424W
Version_conflict: No
Last_response: 0 seconds
VM_restart_count: 0
VM_max_restart_count: 3
Restart_phase: none
HA_monitor_state: STARTED
RMC_state: active

LPM validation was successful for Hosts:

Server-8408-E8E-SN21282FW
#
VM is monitored for its heartbeat, as confirmed by the STARTED value of the HA_monitor_state
VM attribute at the ksysmgr level, which is the same as the HMstate attribute value for the
counterpart RMC resource instance of vmraix1. The Last_response VM attribute at the
ksysmgr level and the HBmissed attribute at the RMC level are both 0, which means that the
heartbeat is received normally. The heartbeat missed case is described in “NeedsAttention
2.3.2 Failure Detection Engine: QuickQuery and NeedsAttention flows

The first request that is logged by the KSYS daemon in the restart exercise in 2.2.5, “KSYS
daemon restart exercise” on page 62 is a QQ request. It was used as an example to introduce
the concepts in “HMC pass-through APIs” on page 101. The focus of that section is to present
the generic HMC pass-through API that is used as communication means between the KSYS
and VIOS sides. The HMC pass-through API is also used for NA requests and all the other
types of requests that are initiated by KSYS threads to communicate with VIOS components,
the HSDB and the HM, and with the VM Agent.
In “Failure Detection Engine” on page 94, we saw that the main job of an FDE thread is to
retrieve the health status of the hosts and VMs from the HSDB. Example 2-60 on page 94
reveals a normal steady state sequence of three polling requests repeated continuously by an
FDE thread: two QQ requests followed by a NA request at 20-second intervals on average.
We now examine an end-to-end QQ request flow. We then repeat the same analysis for an
NA request flow.
QuickQuery flow
A QQ request is initiated by the FDE thread of a host group, so we start our check by looking
at the fde log. Example 2-38 on page 76 shows the trace in the fde log that was left by the
first QQ request at KSYS daemon startup. Comparing that first QQ trace with the trace of our
current QQ in Example 2-209, we notice a similar pattern for the sequence of records.
Example 2-209 Excerpt of a QuickQuery request trace from the fde log
...
[05] 12/22/18 T(9192) _VMR 16:25:28.551712 DEBUG FDEthread.C[127]: Monitoring
[05] 12/22/18 T(9192) _VMR 16:25:28.551752 DEBUG FDEthread.C[145]: CEC is
4462c37c-65c6-3614-b02a-aa09d752c2ee
[05] 12/22/18 T(9192) _VMR 16:25:28.551764 DEBUG FDEthread.C[158]: VIOS
[05] 12/22/18 T(9192) _VMR 16:25:28.551773 DEBUG FDEthread.C[208]: Use VIOS
[05] 12/22/18 T(9192) _VMR 16:25:28.551777 DEBUG FDEthread.C[285]: Current scan [
38 ]
[05] 12/22/18 T(9192) _VMR 16:25:28.551796 DEBUG VMR_VIOS.C[6134]: setCAAtopology

with HMC: rthmc3.
Calling kriSubmitQuickQuery!. HMC:9.3.18.159, viosUuid:
315CE56B-9BA0-46A1-B4BC-DA6108574E7E
Job submitted. Now doing WaitTillJobCompletion() ..
[05] 12/22/18 T(9192) _VMR 16:25:28.678218 DEBUG VMR_HMC.C[3426]: Calling
,retry flag is 22
[05] 12/22/18 T(9192) _VMR 16:25:29.863592 DEBUG needAttn.C[1566]: START
...
Note the entry marking the 20-second sleep interval between the previous request
(sleepCounter 37) and current request (Current scan [ 38 ]). By comparison, there is no
sleep interval before the first QQ request (Current scan [ 1 ]), as shown in Example 2-38 on
page 76. The same VIOS, rt13v2, is used for polling in both cases. A doQuickQuery call is
performed for this VIOS, and inside this call is the HMC REST API job request pattern that is
handled at the krest library level by the kriSubmitQuickQuery and krigetJobResult() calls.
This pattern is covered in “QuickQuery asynchronous job example” on page 105. Here, we
focus on the actions at the FDE and VIOS levels and on the content of the request and
response payloads they exchange with each other.
A typical QQ request that originated at the FDE level on the KSYS side and delivered to the
vioservice command at the VIOS level together with its immediate corresponding XML
response that was obtained at the VIOS level from the HSDB are shown in Example 2-210.
Example 2-210 QuickQuery request and response in the vioservice log on the VIOS side
# alog -f /home/ios/logs/viosvc.log -o > viosvc.log.QQ.txt
# more viosvc.log.QQ.txt
...
</VIO>
]
[0 12058936 62259481 12/22/18-15:20:23.862 viosvc_res.c 1.26 464]
<quickQuery>

</quickQuery>
<quickQuery>
</quickQuery>
</Response></VIO>
]
rc=0
...
Comparing the time stamps in the vioservice log excerpt in Example 2-210 on page 235 with
the ones in the viohs.log log excerpt that is shown in Example 2-211, which were taken from
the same vioservice command execution instance (same process and thread IDs), we
discover that the HSDB was queried for the status details by the vioHsQuickQuery call.
Example 2-211 vioHsQuickQuery call that is performed by the vioservice command to query the HSDB
# alog -f /home/ios/logs/health/viohs.log -o > viohs.log.22dec.txt
# grep "12058936 62259481" viohs.log.22dec.txt
[START 12058936 62259481 12/22/18-15:20:23.790 ha_util.c 1.43 230]
id=730caed6da2211e8800498be9454b8e0
HDBC = 200176e8
HDBC = 200176e8
End, rc=0
#
The same VIOS response XML, now transferred on the KSYS side by the job request (jobid:
1543584834864), is shown as logged in the fdelong log file (Example 2-212 on page 237).

Example 2-212 QuickQuery XML response in the fdelong log file on the KSYS side
[07] 12/22/18 T(9192) _VMR 16:25:29.863556 DEBUG VMR_HMC.C[6954]: JobOutput
[07] 12/22/18 T(9192) _VMR 16:25:29.863557 DEBUG <VIO><Response>
<quickQuery>
</quickQuery>
<quickQuery>
</quickQuery>
</Response></VIO>

,retry flag is 22
...
Both Example 2-210 on page 235 and Example 2-212 show the same QQ request XML
response. The response contains a quickQuery/viosStatList/viosStatus structure of nested
subelements for each host in the host group. Inside the viosStatus element are more
self-closing viosStat subelements, one for each VIOS on the host. We examine the attributes
of the viosStat subelement. Each referred VIOS is identified by a uuid attribute. The rest of
the attributes provide health status details for the referred VIOS as follows:
state State of the VIOS as a node in the CAA cluster underlying the SSP.
Possible values are UP or DOWN.
hmResponsive A value of 1 indicates that the HM daemon on the VIOS is running and
sending heartbeats periodically as expected; otherwise, the value is 0.
hmResponseSec Gives the time elapsed in seconds since the last moment the HM on
the VIOS updated its heartbeat within the HSDB.
hasData A value of 1 indicates that HSDB was updated with some information
about a CAA event that happened on the VIOS node; 0 is used
otherwise.

reason A code from HSDB corresponding to the current CAA event reason, if
any.
syncMsg Used internally.
The values of these attributes in Example 2-212 on page 237 are for the typical happy case
where the status is OK on all monitored VIOSs. Cases with VIOS issues are covered in “CAA
event monitoring” on page 128.
We continue with our typical happy case. The XML response that is received on the KSYS
side is now parsed by the FDE thread in the handleVIOResponse call starting immediately after
the doQuickQuery call return (scan [ 38 ]). Example 2-213 lists the XML parsing-related
records of this handleVIOResponse section, which is also logged in the fdelong log. To shorten
the output, we kept only the records for the first viosStat XML element, which corresponds to
the rt13v2 VIOS.
Example 2-213 Parsing the QuickQuery response from the VIOS

[07] 12/22/18 T(9192) _VMR 16:25:29.863710 DEBUG needAttn.C[1594]: Handling VIO
[07] 12/22/18 T(9192) _VMR 16:25:29.863713 DEBUG needAttn.C[1533]: Handling
Response
...
viosStat
[07] 12/22/18 T(9192) _VMR 16:25:29.863730 DEBUG needAttn.C[714]: uuid =
315ce56b-9ba0-46a1-b4bc-da6108574e7e
[07] 12/22/18 T(9192) _VMR 16:25:29.863739 DEBUG needAttn.C[741]: state = UP for
315ce56b-9ba0-46a1-b4bc-da6108574e7e
[07] 12/22/18 T(9192) _VMR 16:25:29.866542 DEBUG needAttn.C[755]: hmResponsive = 1
for 315ce56b-9ba0-46a1-b4bc-da6108574e7e
[07] 12/22/18 T(9192) _VMR 16:25:29.868871 DEBUG needAttn.C[772]: hmResponseSec =
1 for 315ce56b-9ba0-46a1-b4bc-da6108574e7e
[07] 12/22/18 T(9192) _VMR 16:25:29.868875 DEBUG VMR_VIOS.C[4387]: setHMrespSec
HMrespSec 1 for rt13v2: 315CE56B-9BA0-46A1-B4BC-DA6108574E7E
[07] 12/22/18 T(9192) _VMR 16:25:29.871177 DEBUG needAttn.C[797]: hasData = 0
[07] 12/22/18 T(9192) _VMR 16:25:29.871181 DEBUG needAttn.C[811]: reason = string
(0x00000000) : 0x0
[07] 12/22/18 T(9192) _VMR 16:25:29.871186 DEBUG VMR_VIOS.C[5802]:
setCAAstateReason 0x0 () for rt13v2: 315CE56B-9BA0-46A1-B4BC-DA6108574E7E
rt13v2 UP
[07] 12/22/18 T(9192) _VMR 16:25:29.876316 DEBUG needAttn.C[829]: syncMsg = 0
viosStat
5ac3e0dc-13fc-4221-9617-8ef61c4cdd83
...
[07] 12/22/18 T(9192) _VMR 16:25:29.913014 DEBUG needAttn.C[1605]: FINISH
handleVIOResponse.
[00] 12/22/18 T(9192) _VMR 16:25:29.913072 DEBUG VMR_HG.C[11696]: FDE
handleVIOResponse success

All viosStat element attributes are taken one by one and checked for relevant value changes.
The corresponding attributes of the VIOS resources on the KSYS side are updated with the
new values (Example 2-214). In our case, HMrespSec was changed to 1 in Example 2-213 on
page 238 by the setHMrespSec call.
Example 2-214 VIOS health state attributes on the KSYS side

# lsrsrc -s 'Name="rt13v2"' IBM.VMR_VIOS|egrep -i
"uuid|stat|hm|resp|data|reason|sync|msg"
ViosUuid = "315CE56B-9BA0-46A1-B4BC-DA6108574E7E"
CecUuid = "4462c37c-65c6-3614-b02a-aa09d752c2ee"
State = "MANAGED"
ActiveHM = 101
CAAstate = "UP"
PoolState = "UP"
HMrespSec = 1
HMresponsive = "Yes"
WarnMsg = ""
InfoMsg = ""
HMVersions = {"2.00"}
PartitionState = ""
HMmode = "Active"
CAAstateReason = ""
CAAreposState = "UP"
RMCState = "active"
# ksysmgr -v q vios rt13v2|egrep -i "uuid|stat|hm|resp|data|reason|sync|msg"
UUID: 315CE56B-9BA0-46A1-B4BC-DA6108574E7E
RMC_state: active
State: MANAGED
HMstate: Yes
HMmode: Active
CAA_state: UP
Pool_state: UP
HM_versions: 2.00
#
Nothing abnormal is detected in this happy case example, so the flow finishes and the FDE
thread enters a new 20-second sleep period before starting a new QQ or NA check. This
happy case pattern repeats in a normal steady state operation.
The same QQ flow pattern is followed when case events happen. An abnormal situation
exercise is simulated in “CAA event monitoring” on page 128, where we simulate a VIOS
failure by shutting down a VIOS node. Then, we restart the VIOS node. We describe the way
that generated CAA events are conveyed to the HSD, the FDE, and up the stack to other
KSYS components, such as the event notification subsystem.
Fresh VIOS health state data that is obtained this way feeds continuously into the FDE
thread. The FDE thread decides on the next action, which is to do nothing if all is OK or to
notify the user and decide on host relocation if something abnormal is found, as described in
“Host relocation decision” on page 95.

NeedsAttention flow
The /usr/lib/vioHADR2.00.xsd VIOS reference file defines various used types and for most
of these types provides free text explanations in the preceding comment sections. We
identified vmStateType, discoveryInfoType, VMattenType, VIOS_StatusType, and
reqActionType as containing NA request related information. We cite selectively from their
comment sections as follows:
...
vmStateType
===========
ADDED: The VM has been added via the addVM flow.
STARTING: HM is attempting to start monitoring of the VM in response to a request
from KSYS
STARTED: KSYS has requested that a particular VM is monitored on a managed system
and both HMs on that system have received acknowledgements from the respective
VMM. VIOS asserts need attention when VM moves from starting to started state.
KSYS can choose to fail adding a VM after some reasonable time period if it has
not moved to started state.
STOPPING: HM is attempting to stop monitoring of the VM on a managed system
STOPPED: Moved from stopping to stopped is NeedsAttention case after acked KSYS
can send removeMS
REMOVING: VM is being removed using the removeVM flow
SUSPENDED: HM received suspend me request from VMM
UNKNOWN: An unmanaged VM is discovered
======================================================================
...
This is a discoveryInfoType complex object used for capturing details regarding a
newly discovered VM by HM and report it to KSYS via the NeedsAttention interface.
type: This attribute give info on the type of action/discovery and the details are
as follows:
MOVED: A managed VM moved (likely LPM or LKU action) and the VM can potentially be
associated with the same CEC after the move.
MANAGED: This was requested by KSYS, because of various VM & HM boot conditions
that KSYS is aware of and these conditions are not known to VIOS. This will be
returned always.
UNMANAGED: The discovered VM is currently not managed. Udid value is valid.
CANNOT_MANAGE: The discovered VM has an invalid udid (empty string) and hence
cannot be managed.
versionConflict: This is a Boolean attribute that indicates that there is a
version mismatch between the HM and the VMM.
======================================================================
...
This is object is returned for some number of VMs on a NeedsAttention poll if the
VM meets the criteria that KSYS must be aware of some issue.
VM
missedHB: integer regarding number of Heartbeats missed by the VM
isFenced: the VM is fenced on one or more managed systems
state: The VM can be in multiple states, the state in and of itself is not the
same as NeedsAttention.
...
stateChg: This attribute when present indicates that this VM is undergoing a state
change and state field will indicate the current state. When this attribute is
present, the payload may include a tag will need to be acknowledged.
...
======================================================================
VIOS-related info which can be returned with NeedsAttention poll if state is DOWN

or REMOVED, it NeedsAttention
VIOS UDID,
State
UP: CAA definition of up, is joined to the cluster
DOWN: CAA definition of down, is not currently joined to the cluster
REMOVED: node was removed from the cluster.
KSYS convenience managed system information
If a VIOS is a health monitor, the VIOS maybe returned in NeedsAttention state if
the HM daemon is not responsive. VIOS heartbeats the HM and may kill and restart
the daemon based on the heartbeats. The VIOS may report this as hmResponsive ==
FALSE
======================================================================
...
- VIO_HS_NEEDS_ATTENTION: KSYS uses this request on a poll period. It returns a
list VMs that have missed a heartbeat and or have an application request pending
that has not been acknowledged by KSYS
- VIO_HS_ACKNOWLEDGE: This is sent by KSYS as an acknowledgment that it has
received and successfully saved the information regarding a VM. VIOS will not
return status for a VMs after the acknowledgment until there is a further change
for that VM. VIO_HS_ACKNOWLEDGE is not applied to missed heartbeat however it is
applied to the following:
state transitions
application requests
...
This chapter has some examples of Need Attention requests:

򐂰 The typical happy case steady state NA request in Example 2-204 on page 230
򐂰 The acknowledged NA request with VM state change to STARTED in Example 2-206 on
page 231
򐂰 The acknowledged NA request with the initial DBELL sent during the HM to VMM
handshake sequence in Example 2-207 on page 232
򐂰 The NA request with VIOS-only health state after the CAA node DOWN event in
Example 2-102 on page 139
򐂰 The acknowledged NA request with both the VIOS health state and VM health state in
Example 2-109 on page 146
This section continues with the analysis of some relevant NA request cases. We start with the
case in Example 2-204 on page 230, which has a short reply payload because we used the
simple setup from “HM-VMM communication establishment” on page 215. The test
environment that we used in that section consists of only two hosts in a host group, with one
AIX VM on each host and only one of these two VMs enabled for monitoring. At the end of
that exercise, the VMM was sending heartbeats every 10 seconds to HSDB by using its
underlying HMs, and KSYS was retrieving continuously the VM consolidated status by
sending NA requests every 60 seconds. Example 2-215 shows the NA reply payload.
Example 2-215 Happy case NA reply for one VM sending heartbeats

<VIO><Response>
<needsAtt>
<vmStatList>
</vmList>
</vmStatList>
</needsAtt>
</Response></VIO>

The returned payload contains only one needsAtt/vmStatList/vmList structure of nested
subelements for one of the hosts. It is the host where the one VM sending heartbeats is
located. Migrating the VM to the neighboring host results in a similar NA request returned
payload, as shown in Example 2-216.
Example 2-216 Returned payload of the NA request after the migration of the VM
# ksysmgr lpm vm vmraix1
Running LPM on VM(S), this may take few minutes...
LPM has started for vmraix1
LPM has completed for vmraix1 on Target host Server-8408-E8E-SN21282FW
Waiting for rediscovery.
1 out of 1 VMs have been successfully performed LPM
# ksysmgr q vm state=manage|egrep "^Name|^Host|^HA_monitor"
Name: vmraix1
HA_monitor: enable
Name: vmraix2
HA_monitor: disable
------------------

<VIO><Response>
<needsAtt>
<vmStatList>
<vmList machine_type="8408" model="E8E" serial="21282FW">
</vmList>
</vmStatList>
</needsAtt>
</Response></VIO>
#
From the serial attribute value, we see that the vmList element is now on the other host
where the VM is after the LPM migration.
We enable the second VM for monitoring, run a discovery, migrate the first VM back to its
home host, and recheck the NA reply payload (Example 2-217).
Example 2-217 Reply payload for NA request: Two heartbeat VMs each on distinct hosts
# ksysmgr mod vm vmraix2 ha_monitor=enable
For VM vmraix2 attribute(s) 'ha_monitor' was successfully modified.
# ksysmgr discover hg rbHG
...

LPM has completed for vmraix1 on Target host Server-8408-E8E-SN212424W
#
---------

<VIO><Response>
<needsAtt>
<vmStatList>
</vmList>
</vmStatList>
</needsAtt>
<needsAtt>
<vmStatList>
</vmList>
</vmStatList>
</needsAtt>
</Response></VIO>
#
As expected, we get as the response payload a vmList element inside the nested
needsAtt/vmStatList structure for each host with at least one heartbeat VM. The vmList
elements contain nothing, which means that the heartbeats are received for both VMs.
We analyze the case where the heartbeat from one VM is lost on both VMM to HM paths. We
enforce this missed heartbeat event by disabling the HA client adapters for one of the
heartbeat VMs, as shown in Example 2-218.
Example 2-218 Enforcing a missed heartbeat event

hscroot@usaxhmc013ccpf1:~> lshwres -r virtualio --rsubtype eth -m
Server-8408-E8E-SN212424W --level lpar -F
lpar_name,slot_num,port_vlan_id,vswitch,mac_addr --filter lpar_names=vmraix1
vmraix1,7,101,rbRMHA_VSWITCH,6A6A0FFDF707
vmraix1,32,2030,ETHERNET0,FA643EBA1D20
hscroot@usaxhmc013ccpf1:~> date;chhwres -r virtualio -m Server-8408-E8E-SN212424W
-o d --rsubtype eth -p vmraix1 -s 7;date
Tue Aug 27 22:09:04 UTC 2019
Tue Aug 27 22:09:07 UTC 2019
hscroot@usaxhmc013ccpf1:~> date;chhwres -r virtualio -m Server-8408-E8E-SN212424W
-o d --rsubtype eth -p vmraix1 -s 8;date
Tue Aug 27 22:09:38 UTC 2019
Tue Aug 27 22:09:41 UTC 2019
hscroot@usaxhmc013ccpf1:~>

Its effect at the HSDB level is shown in Example 2-219. The vmhbmissed field in the map table
is updated with the time stamp of the moment when the HM concludes that the heartbeat
from the VM is lost.
Example 2-219 VM heartbeat missed time stamps in the HSDB map table
vios=# select vmuuid,viosuuid,vmhbmissed from map where
vmuuid='61be20f0-09a8-4753-b5e5-564f8a545ed5';
vmuuid | viosuuid |
vmhbmissed
--------------------------------------+--------------------------------------+------------------
----------
61be20f0-09a8-4753-b5e5-564f8a545ed5 | 5c3196a3-c41f-4b5c-9861-dee6077081c6 | 2019-08-27
22:09:14.258417
61be20f0-09a8-4753-b5e5-564f8a545ed5 | 0ab5001f-2c3e-4fbb-97ea-41d3f8c4f898 | 2019-08-27
22:09:54.292171
(2 rows)
vios=#
The excerpts in Example 2-220 show the records that are logged by both HMs at the moment
when each ibe notices that the missed heartbeat event happened, which is around 19
seconds after the last received VM heartbeat.
Example 2-220 VM missed heartbeat event at the HM level

# alog -f /home/ios/logs/health/host_monitor.log -o > host_monitor.log.NeedsAttn
# more host_monitor.log.NeedsAttn
...
[2 13631804 49873289 08/27/19-22:09:14.240 VmHealthStatusMgr.C:VmHealthStatusMgr::checkTim...:
819] VM '61be20f0-09a8-4753-b5e5-564f8a545ed5' state changed to MISSING.
570] VM '61be20f0-09a8-4753-b5e5-564f8a545ed5' state updated in Database to : MISSING
585] vioHmClientMissed returned with rc = 0
682] Broadcasting 'Hello' message with identifier '15667
host_monitor.log.NeedsAttn (82%)
-------------

vios_uuid 0ab5001f-2c3e-4fbb-97ea-41d3f8c4f898 VIOS Unique Identifier False
# alog -f /home/ios/logs/health/host_monitor.log -o > host_monitor.log.NeedsAttn
# more host_monitor.log.NeedsAttn
...

[2 14483764 63177039 08/27/19-22:09:54.274 VmHealthStatusMgr.C:VmHealthStatusMgr::checkTim...:
819] VM '61be20f0-09a8-4753-b5e5-564f8a545ed5' state changed to MISSING.
570] VM '61be20f0-09a8-4753-b5e5-564f8a545ed5' state updated in Database to : MISSING
585] vioHmClientMissed returned with rc = 0
host_monitor.log.NeedsAttn (82%)
The effect at the NA request level is evaluated by probing 1 second at a time, as shown in
Example 2-221.
Example 2-221 Watching for the missed heartbeat event

# while true; do sleep 1;date; /usr/ios/sbin/vioservice lib/libviopass/passthru <na.xml;done
...
Tue Aug 27 22:09:54 CUT 2019
<VIO><Response>
<needsAtt>
<vmStatList>
</vmList>
</vmStatList>
</needsAtt>
<needsAtt>
<vmStatList>
</vmList>
</vmStatList>
</needsAtt>
</Response></VIO>
Tue Aug 27 22:09:55 CUT 2019

<VIO><Response>
<needsAtt>
<vmStatList>
</vmList>
</vmStatList>
</needsAtt>
<needsAtt>
<vmStatList>
<vmStat uuid="61be20f0-09a8-4753-b5e5-564f8a545ed5"><VM state="STARTED" missedHb='42' >
</VM>
</vmStat>
</vmList>

</vmStatList>
</needsAtt>
</Response></VIO>
Tue Aug 27 22:09:56 CUT 2019

<VIO><Response>
<needsAtt>
<vmStatList>
</vmList>
</vmStatList>
</needsAtt>
<needsAtt>
<vmStatList>
<vmStat uuid="61be20f0-09a8-4753-b5e5-564f8a545ed5"><VM state="STARTED" missedHb='43' >
</VM>
</vmStat>
</vmList>
</vmStatList>
</needsAtt>
</Response></VIO>
...
We observe that a new vmStat element shows up inside the vmList for our VM only when
both the first and second HM update their missed heartbeat time stamps in the HSDB.
Example 2-221 on page 245 shows that at 22:09:55, shortly after the missed heartbeat, a
time stamp update was performed by the second HM at 22:09:54.303 (vioHmClientMissed
return). Inside the new vmStat element, a VM subelement is now present with the state and
missedHb attributes having values set as retrieved from the HSDB. The update of the VM
state to the STARTED state is described in “HM-VMM communication establishment” on
page 215. The VM state is now retrieved from the HSDB as it was set then.
We also observe that the missedHb attribute value looks like an elapsed time in seconds value
rather than an integer counter of missed heartbeats value. At 22:09:55, the value is 42,
1 second later is 43, and so on. The /usr/lib/vioHADR2.00.xsd VIOS reference file
documents this attribute as missedHB: integer regarding number of Heartbeats missed by
the VM. We check the health library and HSDB logs around 22:09:55 to get more insight into
how the value of this missedHb attribute is computed.
Example 2-222 shows the entries that are logged in vioshs.log by the NA request that is
issued at 22:09:55, the first one that returns a missedHb value, and the final entries that are
logged by the prior NA request issued a second earlier.
Example 2-222 Health library log around the start of missedHb reporting
# alog -f /home/ios/logs/health/viohs.log -o > viohs.log.NeedsAttn.missedHb
# more viohs.log.NeedsAttn.missedHb
[START 19333444 71762285 08/27/19-22:09:54.695 ha_util.c 1.43 230]
...
[3 19333444 71762285 08/27/19-22:09:54.770 hs_atten.c hsGetHighest 1.108 429] ====>>
61be20f0-09a8-4753-b5e5-564f8a545ed5: count1:40 count2=0
61be20f0-09a8-4753-b5e5-564f8a545ed5: highCnt:40 whichVios=1
[3 19333444 71762285 08/27/19-22:09:54.770 hs_atten.c hs_check_and_segregate_vms 1.108 3727]
>>Missing HB from VIOS:1 cnt=40 noEntry=0

[3 19333444 71762285 08/27/19-22:09:54.771 hs_atten.c hs_serialize_VM_for_Attn 1.108 2741]
61be20f0-09a8-4753-b5e5-564f8a545ed5 ## report=0 stateChg=0 vm->misc.shortMsg=0 shortMsg=0
[3 19333444 71762285 08/27/19-22:09:54.771 hs_atten.c vioHsNeedsAttention 1.108 4593] End, rc=0
[START 19333450 71762291 08/27/19-22:09:55.787 ha_util.c 1.43 230]
vioGetClusterInfoFromODM! cluster name='KSYS_rbRMHA_1' id=1d1bbef6c72d11e98004fa2424246026
[3 19333450 71762291 08/27/19-22:09:55.845 hs_atten.c hs_get_all_VM_short_msg 1.108 1245] Got
zero short msg entries from trans!
[3 19333450 71762291 08/27/19-22:09:55.845 hs_atten.c hs_handle_all_VM_short_msg 1.108 1056] No
short msg entries found in trans table!
[3 19333450 71762291 08/27/19-22:09:55.853 hs_atten.c hsGetMissCount 1.108 338]
[1950678c-18bf-4c87-860c-f9e6ec24b513 72dee902-1210-4bd7-a35f-3a6c771c6453] - 0 sec
[1950678c-18bf-4c87-860c-f9e6ec24b513 10875b47-d737-44f9-a745-554f4df4adf8] - 0 sec
1950678c-18bf-4c87-860c-f9e6ec24b513: count1:0 count2=0
1950678c-18bf-4c87-860c-f9e6ec24b513: highCnt:0 whichVios=0
1950678c-18bf-4c87-860c-f9e6ec24b513 ## report=0 stateChg=0 vm->misc.shortMsg=0 shortMsg=0
[61be20f0-09a8-4753-b5e5-564f8a545ed5 5c3196a3-c41f-4b5c-9861-dee6077081c6] - 2a sec
[61be20f0-09a8-4753-b5e5-564f8a545ed5 0ab5001f-2c3e-4fbb-97ea-41d3f8c4f898] - 2 sec
Missing HB from both vioses, max from :3
[START 19333456 71762297 08/27/19-22:09:56.880 ha_util.c 1.43 230]
viohs.log.NeedsAttn.missedHb (92%)
For the prior NA request, Example 2-222 on page 246 shows that one VIOS has a missed
heartbeat counter of 40, but the other VIOS received fresh heartbeats (count1:40 count2=0),
so no heartbeat missed event is returned. But for the NA request that is issued at 22:09:55,
both VIOSs have positive missed heartbeat counters (count1:42 count2=2). So, this time the
health library encounters a full missed heartbeat even reported by both HMs and returns with
the new elements and a positive value of the missedHb attribute for the affected VM
(61be20f0-09a8-4753-b5e5-564f8a545ed5). The value that is returned is selected as the
maximum among those two, which is 42 for our sample here.

You also notice in Example 2-222 on page 246 how the other VM
(1950678c-18bf-4c87-860c-f9e6ec24b513) is sending heartbeats normally through both
VIOSs (count1:0 count2=0).
To understand how these missed heartbeat counters are computed, we go to the DBN node
and look for database log entries that correspond to one of the hsGetMissCount calls. We look
at the hsGetMissCount call that was issued around 22:09:55.862, and the findings are shown
in Example 2-223.
Example 2-223 Computing the missed heartbeat counters

# /usr/ios/cli/ioscli cluster -status -verbose|grep -p DBN|egrep "Node Name|DBN"
Node Name: usaxvia053ccpx1
Node Roles: DBN
#
------------
# uname -uL
# ls -latr /home/ios/logs/pg_sspdb/*.log
...
-rw-r--r-- 1 vpgadmin bin 20971642 Aug 27 21:36 pg_sspdb-27-21-15.log
...
drwxrwxr-x 8 bin bin 4096 Aug 28 14:00 ..
drwx------ 2 vpgadmin bin 4096 Aug 28 14:30 .
# cp /home/ios/logs/pg_sspdb/pg_sspdb-27-21-56.log pg_sspdb-27-21-56.log.missedHb
# more pg_sspdb-27-21-56.log.missedHb
...
2019-08-27 22:09:55.861 CUT|5d65aa33.19e0118|LOG: statement: RELEASE
_EXEC_SVP_20122a88;SAVEPOINT _EXEC_SVP_20122a88;select DISTINCT vmuuid from vioshs.map where
viosuuid='5c3196a3-c41f-4b5c-9861-dee6077081c6' OR
viosuuid='0ab5001f-2c3e-4fbb-97ea-41d3f8c4f898'
_EXEC_SVP_20122a88;SAVEPOINT _EXEC_SVP_20122a88;SELECT CAST ((EXTRACT( epoch FROM NOW():
:timestamp) - EXTRACT( epoch FROM vioshs.map.vmHbMissed::timestamp)) AS BIGINT) FROM vioshs.map
WHERE vioshs.map.vmuuid='61be20f0-09a8-4753-b5e5-564f8a545ed5' AND
vioshs.map.viosuuid='5c3196a3-c41f-4b5c-9861-dee6077081c6'
_EXEC_SVP_20122a88;SAVEPOINT _EXEC_SVP_20122a88;SELECT CAST ((EXTRACT( epoch FROM NOW():
:timestamp) - EXTRACT( epoch FROM vioshs.map.vmHbMissed::timestamp)) AS BIGINT) FROM vioshs.map
WHERE vioshs.map.vmuuid='61be20f0-09a8-4753-b5e5-564f8a545ed5' AND
vioshs.map.viosuuid='0ab5001f-2c3e-4fbb-97ea-41d3f8c4f898'
pg_sspdb-27-21-56.log.missedHb (74%)
So, the missed heartbeat counters are computed as the difference between the current
moment and the missed heartbeat time stamp that was reported by the HM in the HSDB map
table ((EXTRACT( epoch FROM NOW()::timestamp) - EXTRACT( epoch FROM
vioshs.map.vmHbMissed::timestamp)) AS BIGINT).

We are done with the analysis of this enforced missed heartbeat event for our test VM and
have a better understanding of how such an event is registered by the HMs in the HSDB and
becomes available for NA request retrieval.
Example 2-224 shows an NA reply payload that is captured by the probing shown in
Example 2-221 on page 245 during an LPM operation that is performed by ksysmgr.
Example 2-224 NA reply payload during a lpm operation that is performed by ksysmgr
LPM has completed for vmraix1 on Target host Server-8408-E8E-SN21282FW
#
------------------------
# uname -uL
# while true; do sleep 2;date; /usr/ios/sbin/vioservice lib/libviopass/passthru <na.xml;done
...
Thu Aug 29 08:06:36 CUT 2019
<VIO><Response>
<needsAtt hsTag="-8403607219422560833">
<vmStatList>
<vmStat uuid="61be20f0-09a8-4753-b5e5-564f8a545ed5"><VM state="STARTED" stateChg='1'
missedHMEntry="10875b47-d737-44f9-a745-554f4df4adf8" >
<discoveryInfo type='MOVED'/>
</VM>
</vmStat>
</vmList>
</vmStatList>
</needsAtt>
<needsAtt hsTag="-8403607219422560833">
<vmStatList>
<vmStat uuid="61be20f0-09a8-4753-b5e5-564f8a545ed5"><VM state="STARTED" stateChg='1' >
</VM>
</vmStat>
</vmList>
</vmStatList>
</needsAtt>
</Response></VIO>
...
^C#

Inside the VM element in Example 2-224 on page 249, there is a discoveryInfo subelement
with a type attribute set to a MOVED value. As defined in the /usr/lib/vioHADR2.00.xsd VIOS
reference file, it is used for capturing details regarding a VM newly discovered by an HM and
reporting it to KSYS through the NeedsAttention interface. We already saw an NA request
case with multiple instances of this discoveryInfo subelement in a different context. The
response payload in Example 2-109 on page 146 contains more subelements of this kind,
with the type attribute value set to UNMANAGED and the state attribute of the enclosing VM
elements set to UNKNOWN. They inform KSYS that some unmanaged VMs were discovered on
the host that is mentioned in the enclosing vmList element by the serial attribute.
The NA request in Example 2-224 on page 249 leaves a trace in the health library log, as
Example 2-225 Health library log entries for an NA request that is issued during a ksysmgr lpm operation
# alog -f /home/ios/logs/health/viohs.log -o > viohs.log.NeedsAttn.LPM
# more viohs.log.NeedsAttn.LPM
[START 17498526 83362155 08/29/19-08:06:36.346 ha_util.c 1.43 230]
vioGetClusterInfoFromODM! cluster name='KSYS_rbRMHA_1' id=1d1bbef6c72d11e98004fa2424246026
[3 17498526 83362155 08/29/19-08:06:36.392 hs_atten.c hs_get_all_VM_short_msg 1.108 1245] Got
zero short msg entries from trans!
[3 17498526 83362155 08/29/19-08:06:36.392 hs_atten.c hs_handle_all_VM_short_msg 1.108 1056] No
short msg entries found in trans table!
[1950678c-18bf-4c87-860c-f9e6ec24b513 72dee902-1210-4bd7-a35f-3a6c771c6453] - 0 sec
[1950678c-18bf-4c87-860c-f9e6ec24b513 10875b47-d737-44f9-a745-554f4df4adf8] - 0 sec
1950678c-18bf-4c87-860c-f9e6ec24b513: count1:0 count2=0
1950678c-18bf-4c87-860c-f9e6ec24b513: highCnt:0 whichVios=0
[61be20f0-09a8-4753-b5e5-564f8a545ed5 72dee902-1210-4bd7-a35f-3a6c771c6453] - 0 sec
[3 17498526 83362155 08/29/19-08:06:36.400 hm_time.c tsDiff 1.39 240] Error: Query SELECT CAST
((EXTRACT( epoch FROM NOW()::timestamp) - EXTRACT( epoch FROM vios
hs.map.vmHbMissed::timestamp)) AS BIGINT) FROM vioshs.map WHERE
vioshs.map.vmuuid='61be20f0-09a8-4753-b5e5-564f8a545ed5' AND
vioshs.map.viosuuid='10875b47-d737-44f9-a745-554f4df4adf8' found no entry in the DB table.
[3 17498526 83362155 08/29/19-08:06:36.400 hs_atten.c hsGetMissCount 1.108 343] Error (rc=-2)
getting miss HB info for
61be20f0-09a8-4753-b5e5-564f8a545ed5-10875b47-d737-44f9-a745-554f4df4adf8
[3 17498526 83362155 08/29/19-08:06:36.400 hs_atten.c hsGetHighest 1.108 419] There is no map
entry with vm 61be20f0-09a8-4753-b5e5-564f8a545ed5, vios 72dee902-1210-4bd7-a35f-3a6c771c6453
rc=-2

1950678c-18bf-4c87-860c-f9e6ec24b513 ## report=0 stateChg=0 vm->misc.shortMsg=0 shortMsg=0
[3 17498526 83362155 08/29/19-08:06:36.408 hs_atten.c hs_get_vms_for_HM_pair 1.108 3658] Got
zero VMs with query=select DISTINCT vmuuid from vioshs.map where
viosuuid='0ab5001f-2c3e-4fbb-97ea-41d3f8c4f898' OR
[3 17498526 83362155 08/29/19-08:06:36.408 hs_atten.c hs_handle_msys 1.108 4019] No VMs
monitored from the CEC=-3133915808583905883
[3 17498526 83362155 08/29/19-08:06:36.408 hs_atten.c hs_serialize_VM_list_for_discovery 1.108
3206] Error getting MS info for -6300057356714549736
[START 17498532 83362161 08/29/19-08:06:38.420 ha_util.c 1.43 230]
viohs.log.NeedsAttn.LPM (99%)
Comparing to the similar trace in Example 2-222 on page 246, we observe that the same
actions are performed inside the vioHsNeedsAttention with more actions in the following
order:
1. Look for short message entries in trans table.
2. Check and report the VM missed heartbeats.
3. Look for VM state changes and report the state changes and short messages.
4. Look for VM discoveries.
To conclude our findings so far, we list the attributes that were encountered in the NA reply
payloads for the analyzed cases. These attributes describe either VM elements or adjacent
elements.
missedHb Gives the time elapsed in seconds since the HMs reported missed
heartbeats for the VM. It is returned only if both HMs report positive
values and the maximum value is chosen.
stateChg Indicates that the VM is undergoing a state change. An accompanying
state field indicates the current state. When this attribute is present,
the payload may include a tag that must be acknowledged.
shortMsg A short message that is sent by the VM toward KSYS to notify it that a
configuration or status change happened at the application level of our
VM.
state The VM can be in one of the following states:
ADDED The VM was added by the addVM flow.
STARTING The HMs are attempting to start monitoring of the VM in response
to a request from KSYS.
STARTED The KSYS has enabled the VM to be monitored, and both HMs on
the host where the VM is located received heartbeats from the
respective VMM.
STOPPING HM is attempting to stop monitoring of the VM on a managed
system.

STOPPED Moved from stopping to stopped if it is a NeedsAttention case. After
it is acknowledge, KSYS can send a removeMS call.
REMOVING VM is being removed by using the removeVM flow.
SUSPENDED HM received a suspend me request from the VMM.
UNKNOWN An unmanaged VM is discovered.
uuid The VM unique identifier that shows up as an attribute of the vmStat
element.
serial Gives the serial number of the host where the VM is as an attribute of
the enclosing vmList element.
type Attribute of an enclosed discoveryType element that is used for a
newly discovered VM by an underlying HM. Gives details about the
type of discovery by using the following values:
MOVED A managed VM is being moved by LPM. The VM can potentially be
associated with the host after the move.
MANAGED Requested by KSYS because of various VM and HM boot
conditions that KSYS is aware of. These conditions are not known
to VIOS.
UNMANAGED The discovered VM is not managed.
CANNOT_MANAGE The discovered VM has an invalid ID (empty string) and cannot be
managed.
As shown in Example 2-102 on page 139, elements and attributes describing a VIOS heath
state can appear also in NA reply payloads in special cases, like in QQ reply payload. The
information that was stated about the QQ reply payload attributes describing the VIOS heath
state in “QuickQuery flow” on page 234 applies also to this case, so we do not cover it again.
Now, we look at the NA request flow. We expect it to be similar to the steady state QQ request
flow in “QuickQuery flow” on page 234. The NA request flow starts with the request payload
that is sent by the host group FDE thread on the KSYS side to the VIOS side. It continues with
the response payload that is retrieved from HSDB and conveyed back to the FDE thread,
where it is processed and, depending on results, either triggers appropriate actions or notifies
users with appropriate messages. If nothing new or abnormal is detected in the reply payload
about a VM, such as a VM state change or application-level change or a missed heartbeat,
then we have a steady state happy case and the flow finishes without triggering any action.
The FDE thread enters a new 20-second sleep period after which it starts the next QQ
request.
Let us now approach the VM missed heartbeat scenario and present in detail its NA request
flow. Earlier in this section, we described how the missed heartbeat event is reported by the
HMs to the HSDB and how a NA request retrieves the missedHb attribute values for such a
case. Here, our focus is on the actions happening at the KSYS level. The event is enforced in
the same way by disabling the HA client adapters of the VM at the HMC level. Then, through
the probing in Example 2-226, we capture the effect at the ksysmgr level.
Example 2-226 Capturing the effect of the missed heartbeat at the ksysmgr level
hscroot@usaxhmc013ccpf1:~> chhwres -r virtualio -m Server-8408-E8E-SN212424W -o d --rsubtype eth
-p vmraix1 -s 7
-p vmraix1 -s 8;date
Thu Aug 29 11:37:41 UTC 2019

-------

# while true; do date;lsrsrc -s 'Name="vmraix1"' IBM.VMR_LPAR|egrep "HBmissed\ |RestartPhase";
ksysmgr -v q vm vmraix1|egrep "Last_response|Restart_phase"; sleep 60; done
Thu Aug 29 11:37:48 CUT 2019
HBmissed = 0
RestartPhase = ""
Restart_phase: none
Thu Aug 29 11:38:49 CUT 2019
HBmissed = 50
RestartPhase = "Missed HB count 50 for VM vmraix1 is less than its VMFDT 190"
Restart_phase: Missed HB count 50 for VM vmraix1 is less than its VMFDT 190
Thu Aug 29 11:39:50 CUT 2019
HBmissed = 111
Thu Aug 29 11:40:52 CUT 2019
HBmissed = 171
Thu Aug 29 11:41:53 CUT 2019
HBmissed = 232
RestartPhase = "VM vmraix1 is pingable ignoring missed HB count 232"
Restart_phase: VM vmraix1 is pingable ignoring missed HB count 232
^Croot:vmrksys: /home/root >
#
To get more insights about the actions at the KSYS level, we chose to check directly the most
recent NA request before 11:41:53 when our probing notified us that the VMFDT threshold was
reached. So, we look in the fdelong trace file for the records of this NA request, as shown in
Example 2-227.
Example 2-227 The fdelong trace records for the NA request with VMFDT threshold passed
# ksysmgr trace log=fdelong > fdelong.log.missedHb
# more fdelong.log.missedHb
[14] 08/29/19 T(34c) _VMR 11:41:15.477873 DEBUG FDEthread.C[709]: Sleep for 20 sec. sleepCounter
16343
[14] 08/29/19 T(34c) _VMR 11:41:35.477911 DEBUG FDEthread.C[127]: Monitoring Enabled for HG
rbHG.
[14] 08/29/19 T(34c) _VMR 11:41:35.477929 DEBUG FDEthread.C[145]: CEC is
33613bcd-7ca1-3558-874a-c1e1d3ceee32
[14] 08/29/19 T(34c) _VMR 11:41:35.477938 DEBUG FDEthread.C[158]: VIOS
72DEE902-1210-4BD7-A35F-3A6C771C6453 in CEC
[14] 08/29/19 T(34c) _VMR 11:41:35.477946 DEBUG FDEthread.C[208]: Use VIOS usaxvib053ccpx1:
72DEE902-1210-4BD7-A35F-3A6C771C6453 for polling
[14] 08/29/19 T(34c) _VMR 11:41:35.477967 DEBUG FDEthread.C[285]: Current scan [ 21522 ]
[14] 08/29/19 T(34c) _VMR 11:41:35.477970 DEBUG VMR_HG.C[11824]: FDE performing doNeedAttn
GLOBAL_DATA to 72DEE902-1210-4BD7-A35F-3A6C771C6453

[14] 08/29/19 T(34c) _VMR 11:41:35.477979 DEBUG VMR_retry.C[1174]: Doing operation with opCode:
21(VMDR_NEED_ATTN)
[14] 08/29/19 T(34c) _VMR 11:41:35.477996 DEBUG VMR_retry.C[178]: INFO: Trying with HMC:
usaxhmc013ccpf1.
[14] 08/29/19 T(34c) _VMR 11:41:35.478001 DEBUG VMR_HMC.C[6852]: getNeedAttn: Calling
kriSubmitNeedAttn!. HMC:129.41.127.3, viosUuid: 72DEE902-1210-4BD7-A35F-3A6C771C6453
[14] 08/29/19 T(34c) _VMR 11:41:35.580081 DEBUG VMR_HMC.C[6872]: getNeedAttn: Job submitted. Now
doing WaitTillJobCompletion() ..
[14] 08/29/19 T(34c) _VMR 11:41:35.580083 DEBUG VMR_HMC.C[3468]: Calling krigetJobResult(). HMC:
129.41.127.3, jobid: 1563455810649, retCnt = 1
[14] 08/29/19 T(34c) _VMR 11:41:35.671675 DEBUG VMR_HMC.C[6886]: getNeedAttn
[72DEE902-1210-4BD7-A35F-3A6C771C6453] JobStatus: COMPLETED_OK, ReturnCode: 0
[14] 08/29/19 T(34c) _VMR 11:41:35.671676 DEBUG VMR_HMC.C[6902]: JobOutput
Notice in Example 2-227 on page 253 the prior 20-second sleep interval, how the VIOS that
is used for polling is assigned, how the NA job request is submitted through the HMC to that
VIOS, and how it completed. The reply payload is shown in Example 2-228.
Example 2-228 Reply payload for the NA request as retrieved by the FDE thread
[14] 08/29/19 T(34c) _VMR 11:41:35.671676 DEBUG VMR_HMC.C[6902]: JobOutput
[14] 08/29/19 T(34c) _VMR 11:41:35.671676 DEBUG <VIO><Response>
<needsAtt>
<vmStatList>
<vmList machine_type="8408" model="E8E"
serial="21282FW">
</vmList>
</vmStatList>
</needsAtt>
<needsAtt>
<vmStatList>
<vmList machine_type="8408" model="E8E"
serial="212424W">
<vmStat
uuid="61be20f0-09a8-4753-b5e5-564f8a545ed5"><VM state="STARTED" missedHb='232' >
</VM>
</vmStat>
</vmList>
</vmStatList>
</needsAtt>
</Response></VIO>
[14] 08/29/19 T(34c) _VMR 11:41:35.671680 DEBUG VMR_retry.C[345]: In doRetry function, for
opCode = 21(VMDR_NEED_ATTN), rc = 0, retCode is 0, errstr is: ,retry flag is 22
[14] 08/29/19 T(34c) _VMR 11:41:35.671687 DEBUG VMR_HG.C[11832]: FDE doNeedAttn success
[14] 08/29/19 T(34c) _VMR 11:41:35.671690 DEBUG needAttn.C[1484]: START handleVIOResponse scan [
21522 ].
We see how the final processing stage of the reply payload is performed in the
handleVIOResponse call, as shown in Example 2-229 on page 255.

Example 2-229 Processing the NA reply payload: VMFDT threshold passed
21522 ].
[14] 08/29/19 T(34c) _VMR 11:41:35.671798 DEBUG needAttn.C[341]: Working on VM vmraix1:
61BE20F0-09A8-4753-B5E5-564F8A545ED5
[14] 08/29/19 T(34c) _VMR 11:41:35.671804 DEBUG VMR_LPAR.C[16228]: setHBmissed 232 for vmraix1:
61BE20F0-09A8-4753-B5E5-564F8A545ED5 vlanid 0 notAvail 0 scan [ 21522 ]
[14] 08/29/19 T(34c) _VMR 11:41:35.674720 DEBUG VMR_LPAR.C[16336]: setHBmissed - Able to ping
vmraix1, ignore HBmissed
[14] 08/29/19 T(34c) _VMR 11:41:35.674720 DEBUG VMR_LPAR.C[16336]: setHBmissed - Able to ping
vmraix1, ignore HBmissed
[14] 08/29/19 T(34c) _VMR 11:41:35.680021 DEBUG needAttn.C[1521]: FINISH handleVIOResponse.
[14] 08/29/19 T(34c) _VMR 11:41:35.680042 DEBUG VMR_HG.C[11848]: FDE handleVIOResponse success
GLOBAL_DATA
[14] 08/29/19 T(34c) _VMR 11:41:35.680047 DEBUG FDEthread.C[382]: Checking for LOCAL DATA
[14] 08/29/19 T(34c) _VMR 11:41:35.680052 DEBUG FDEthread.C[775]: STARTING LOCAL DATA
VERIFICATION
33613bcd-7ca1-3558-874a-c1e1d3ceee32
72DEE902-1210-4BD7-A35F-3A6C771C6453 in CEC
[14] 08/29/19 T(34c) _VMR 11:41:35.680072 DEBUG FDEthread.C[827]: VIOS usaxvib053ccpx1:
72DEE902-1210-4BD7-A35F-3A6C771C6453 is capable of doing GLOBAL DATA. Skipping the CEC
33613bcd-7ca1-3558-874a-c1e1d3ceee32.
3a53a6fc-8dcb-3671-a853-70526651f83d
5C3196A3-C41F-4B5C-9861-DEE6077081C6 in CEC
[14] 08/29/19 T(34c) _VMR 11:41:35.680085 DEBUG FDEthread.C[827]: VIOS usaxvib063ccpx1:
5C3196A3-C41F-4B5C-9861-DEE6077081C6 is capable of doing GLOBAL DATA. Skipping the CEC
3a53a6fc-8dcb-3671-a853-70526651f83d.
[14] 08/29/19 T(34c) _VMR 11:41:35.680086 DEBUG FDEthread.C[922]: FINISH LOCAL DATA VERIFICATION
[14] 08/29/19 T(34c) _VMR 11:41:35.680171 DEBUG FDEthread.C[677]: LPAR vmraix2 VM.scan 21519 <
Current HG.scan 21522
[14] 08/29/19 T(34c) _VMR 11:41:35.680190 DEBUG FDEthread.C[684]: LPAR vmraix2 VM.scanMEntry
21519 < Current HG.scan 21522
[14] 08/29/19 T(34c) _VMR 11:41:35.680198 DEBUG FDEthread.C[677]: LPAR vmrksys VM.scan 21519 <
Current HG.scan 21522
[14] 08/29/19 T(34c) _VMR 11:41:35.680208 DEBUG FDEthread.C[684]: LPAR vmrksys VM.scanMEntry
[14] 08/29/19 T(34c) _VMR 11:41:35.680213 DEBUG FDEthread.C[684]: LPAR vmraix1 VM.scanMEntry
[14] 08/29/19 T(34c) _VMR 11:41:35.680215 DEBUG FDEthread.C[709]: Sleep for 20 sec. sleepCounter
16344
The setHBmissed call inside the handleVIOResponse call evaluates the missed heartbeat
counter against the threshold and further ascertains that the VM is reachable by ping, so the
missed heartbeat is ignored, as logged before the handleVIOResponse call finish (setHBmissed
- Able to ping vmraix1, ignore HBmissed). We ignore the last local data step in this final
NA reply payload processing stage. No other action is performed afterward, and the NA
request flow is finished as marked by the logged record for the subsequent 20-second sleep
interval.

For better understanding, we also mention in Example 2-230 the records that are logged
inside the handleVIOResponse call in previous NA requests, which matches the probing output
in Example 2-226 on page 252.
Example 2-230 Final processing stage for the NA requests issued before passing the VMFDT threshold
21510 ].
GLOBAL_DATA
...
21513 ].
61BE20F0-09A8-4753-B5E5-564F8A545ED5
[14] 08/29/19 T(34c) _VMR 11:38:33.642482 DEBUG VMR_LPAR.C[16309]: setHBmissed HBmissed(50) <
VM Monitoring interval(190) No Action
GLOBAL_DATA
...
21516 ].
61BE20F0-09A8-4753-B5E5-564F8A545ED5
GLOBAL_DATA
...
21519 ].
61BE20F0-09A8-4753-B5E5-564F8A545ED5
GLOBAL_DATA

We see in the log excerpts Example 2-230 on page 256 that the setHBmissed function is
called only when HBmissed (set from the reply missedHb attribute) has a positive value and that
no action is performed if its value is less than VMFDT.
For completeness, we also simulated the case when ping is not working so that VM relocation
is decided, as shown in Example 2-231.
Example 2-231 Final processing stage for the NA request triggering the relocation
hscroot@usaxhmc013ccpf1:~> lshwres -r virtualio --rsubtype eth -m Server-8408-E8E-SN212424W
--level lpar -F lpar_name,slot_num,port_vlan_id,vswitch,mac_addr --filter lpar_names=vmraix1
vmraix1,32,2030,ETHERNET0,FA643EBA1D20
-p vmraix1 -s 32
-p vmraix1 -s 7
-p vmraix1 -s 8;date
Thu Aug 29 14:27:08 UTC 2019
-----
# while true; do date; lsrsrc -s 'Name="vmraix1"' IBM.VMR_LPAR |egrep "HBmissed\

|RestartPhase";ksysmgr -v q vm vmraix1|egrep "Last_response|Restart_phase"; >
...
Thu Aug 29 14:31:28 CUT 2019
HBmissed = 240
RestartPhase = "VM failure detected. Initiating recovery for VM vmraix1 "
Restart_phase: VM failure detected. Initiating recovery for VM vmraix1
^Croot:vmrksys: /home/root >
# ksysmgr trace log=fdelong > fdelong.log.missedHb.relocate
# more fdelong.log.missedHb.relocate
...
22023 ].
61BE20F0-09A8-4753-B5E5-564F8A545ED5
[06] 08/29/19 T(34c) _VMR 14:31:20.046111 DEBUG VMR_LPAR.C[16343]: setHBmissed - Limit reached
for vmraix1:61BE20F0-09A8-4753-B5E5-564F8A545ED5 (state =STARTED)
scan [ 22023 ]
[06] 08/29/19 T(34c) _VMR 14:31:20.047743 DEBUG VMR_LPAR.C[16361]: VM_FAILURE_DETECTED Event
notification failed.!!
[06] 08/29/19 T(34c) _VMR 14:31:20.047747 DEBUG VMR_LPAR.C[16375]: setHBmissed - Not able to
ping vmraix1
[06] 08/29/19 T(34c) _VMR 14:31:20.323656 DEBUG VMR_LPAR.C[16393]: The state of VM vmraix1 as
per HMC is "running"

[06] 08/29/19 T(34c) _VMR 14:31:20.323667 DEBUG VMR_LPAR.C[16439]: Recovering LPAR vmraix1 state
running HBmissed 240 HBoffset 0 refcode
[06] 08/29/19 T(34c) _VMR 14:31:20.323672 DEBUG VMR_LPAR.C[16440]: RECOVERY TASK ADDED for LPAR
[vmraix1-61BE20F0-09A8-4753-B5E5-564F8A545ED5]
GLOBAL_DATA
fdelong.log.missedHb.relocate (99%)
We further detail the flow for the NA request case when the shortMsg attribute in the reply
payload is set. Such an NA request flow shows up as part of the AR flow, as described in
2.3.3, “Application Reporting flow” on page 262.
Similar to the QQ request flow, we start our analysis by checking the start records in the fde
log, as shown in Example 2-232. From the time stamp and the sleepCounter value of 38 in
the first record, you see it is the immediate subsequent FDE polling request that is logged
after the QQ request that we examined in Example 2-213 on page 238.
Example 2-232 Excerpt of a NeedsAttention request trace from the fdelong log
# ksysmgr trace log=fdelong > fdelong.log.NAshortMsg
# more fdelong.log.NAshortMsg
[05] 12/22/18 T(9192) _VMR 16:25:49.913226 DEBUG FDEthread.C[127]: Monitoring
4462c37c-65c6-3614-b02a-aa09d752c2ee
[05] 12/22/18 T(9192) _VMR 16:25:49.913304 DEBUG FDEthread.C[208]: Use VIOS
[05] 12/22/18 T(9192) _VMR 16:25:49.913307 DEBUG FDEthread.C[285]: Current scan [
39 ]
operation with opCode: 21(VMDR_NEED_ATTN)
with HMC: rthmc3.
[05] 12/22/18 T(9192) _VMR 16:25:49.913344 DEBUG VMR_HMC.C[6793]: getNeedAttn:
Calling kriSubmitNeedAttn!. HMC:9.3.18.159, viosUuid:
315CE56B-9BA0-46A1-B4BC-DA6108574E7E
[05] 12/22/18 T(9192) _VMR 16:25:50.034284 DEBUG VMR_HMC.C[6813]: getNeedAttn: Job
submitted. Now doing WaitTillJobCompletion() ..
[05] 12/22/18 T(9192) _VMR 16:25:51.220926 DEBUG VMR_HMC.C[6827]: getNeedAttn
function, for opCode = 21(VMDR_NEED_ATTN), rc = 0, retCode is 0, errstr is: ,retry
flag is 22
[05] 12/22/18 T(9192) _VMR 16:25:51.220945 DEBUG VMR_HG.C[11592]: FDE doNeedAttn
success GLOBAL_DATA to 315CE56B-9BA0-46A1-B4BC-DA6108574E7E

Inside the doNeedAttn call, we recognize a krest HMC REST API job request pattern, which
is handled by kriSubmitNeedAttn and krigetJobResult() calls. This pattern is similar to the
case covered in “QuickQuery asynchronous job example” on page 105, so we do not need to
get into details. The NA request, as it is delivered at the VIOS level, together with its
corresponding response, obtained from the HSDB, are shown in Example 2-233.
Example 2-233 NeedsAttention request and response in the vioservice log

# alog -f /home/ios/logs/viosvc.log -o > viosvc.log.NAshortMsg.txt
# more viosvc.log.NAshortMsg.txt
...
START 12058942 62259487 12/22/18-15:20:45.144 vioservice.c 1.18 320]
</VIO>
]
[0 12058942 62259487 12/22/18-15:20:45.240 viosvc_res.c 1.26 464]
<needsAtt hsTag="-3026381694404356594">
<vmStatList>
<vmList machine_type="8286" model="42A" serial="21E0B2V">
<vmStat uuid="4e395e99-fffb-4345-8067-2d5296ba7d91"><VM state="STARTED"
shortMsg="0x28" >
</VM>
</vmStat>
</vmList>
</vmStatList>
</needsAtt>
<needsAtt hsTag="-3026381694404356594">
<vmStatList>
<vmList machine_type="8286" model="42A" serial="2100DEW">
</vmList>
</vmStatList>
</needsAtt>
</Response></VIO>
]
rc=0

Though we have many managed VMs on the host group, we notice that the content in the
XML response mentions only one VM. So, we can assume that the other VMs are sending
heartbeats normally, they have no VM Agent that is installed, or are not enabled for
monitoring. We also notice that this request comes with a tag value that is passed under the
hsTag attribute, so we expect that an acknowledgment request will follow. Comparing the time
stamps in the vioservice log excerpt as shown in Example 2-233 on page 259 and in the
viohs.log log excerpt in Example 2-234, which are taken for the same vioservice command
execution instance (same process and thread IDs), we see the way HSDB was queried for the
status details by a vioHsNeedsAttention call.
Example 2-234 The vioHsNeedsAttention call performed by the vioservice command to query the
HSDB
# alog -f /home/ios/logs/health/viohs.log -o > viohs.log.NAshortMsg.txt
# grep "12058942 62259487" viohs.log.NAshortMsg.txt
[START 12058942 62259487 12/22/18-15:20:45.146 ha_util.c 1.43 230]
id=730caed6da2211e8800498be9454b8e0
HDBC = 200176e8
HDBC = 200176e8
...
[3 12058942 62259487 12/22/18-15:20:45.240 hs_atten.c vioHsNeedsAttention 1.107
4587] End, rc=0
#
The VIOS response payload that is retrieved on the KSYS side by the job request is then
parsed by the FDE thread. The handleVIOResponse call section in Example 2-235 contains
the XML parsing-related records that are logged in the fde log for our NA polling request
(scan [ 39 ]). The shortMsg="0x28" attribute for the rt13001 VM makes the FDE thread add
a SHORTMSG task for that VM.
Example 2-235 Parsing the NeedsAttention response from the VIOS

# more fdelong.log.NAshortMsg
[05] 12/22/18 T(9192) _VMR 16:25:51.221027 DEBUG needAttn.C[1168]: hsTag = string
(-3026381694404356594) : longlong (-3026381694404356594)
[05] 12/22/18 T(9192) _VMR 16:25:51.221034 DEBUG needAttn.C[167]: machine_type =
8286
[05] 12/22/18 T(9192) _VMR 16:25:51.221040 DEBUG needAttn.C[176]: model = 42A
[05] 12/22/18 T(9192) _VMR 16:25:51.221043 DEBUG needAttn.C[185]: serial = 21E0B2V
4e395e99-fffb-4345-8067-2d5296ba7d91
[05] 12/22/18 T(9192) _VMR 16:25:51.221158 DEBUG needAttn.C[352]: Working on VM
rt13001: 4E395E99-FFFB-4345-8067-2D5296BA7D91
[05] 12/22/18 T(9192) _VMR 16:25:51.224155 DEBUG needAttn.C[490]: shortMsg = 0x28
[05] 12/22/18 T(9192) _VMR 16:25:51.224160 DEBUG needAttn.C[507]: SHORTMSG TASK
ADDED: LPAR rt13001: 4E395E99-FFFB-4345-8067-2D5296BA7D91, dbell 28

[05] 12/22/18 T(9192) _VMR 16:25:51.224210 DEBUG needAttn.C[1168]: hsTag = string
(-3026381694404356594) : longlong (-3026381694404356594)
[05] 12/22/18 T(9192) _VMR 16:25:51.224216 DEBUG needAttn.C[167]: machine_type =
8286
[05] 12/22/18 T(9192) _VMR 16:25:51.224219 DEBUG needAttn.C[176]: model = 42A
[05] 12/22/18 T(9192) _VMR 16:25:51.224222 DEBUG needAttn.C[185]: serial = 2100DEW
[05] 12/22/18 T(9192) _VMR 16:25:51.224225 DEBUG needAttn.C[1605]: FINISH
handleVIOResponse.
handleVIOResponse success GLOBAL_DATA
doAcknowledge on hsTag -3026381694404356594
[05] 12/22/18 T(9192) _VMR 16:25:51.224291 DEBUG VMR_HMC.C[7017]: getAcknowledge:
Calling kriSubmitAck!. HMC:9.3.18.159, viosUuid:
315CE56B-9BA0-46A1-B4BC-DA6108574E7E
[05] 12/22/18 T(9192) _VMR 16:25:51.429664 DEBUG VMR_HMC.C[7037]: getAcknowledge:
Job submitted. Now doing WaitTillJobCompletion() ..
doAcknowledge success
[05] 12/22/18 T(9192) _VMR 16:25:51.519457 DEBUG FDEthread.C[382]: Checking for
LOCAL DATA
[05] 12/22/18 T(9192) _VMR 16:25:51.519464 DEBUG FDEthread.C[775]: STARTING LOCAL
DATA VERIFICATION
4462c37c-65c6-3614-b02a-aa09d752c2ee
[05] 12/22/18 T(9192) _VMR 16:25:51.519496 DEBUG FDEthread.C[827]: VIOS rt13v2:
315CE56B-9BA0-46A1-B4BC-DA6108574E7E is capable of doing GLOBAL DATA. Skipping the
CEC 4462c37c-65c6-3614-b02a-aa09d752c2ee.
6e2f37f5-1824-347a-99fd-488f818086dd
4E0F6F60-214B-4D2C-A436-0EAC46F7F71F in CEC
[05] 12/22/18 T(9192) _VMR 16:25:51.519513 DEBUG FDEthread.C[827]: VIOS rt14v1:
4E0F6F60-214B-4D2C-A436-0EAC46F7F71F is capable of doing GLOBAL DATA. Skipping the
CEC 6e2f37f5-1824-347a-99fd-488f818086dd.
[05] 12/22/18 T(9192) _VMR 16:25:51.519515 DEBUG FDEthread.C[922]: FINISH LOCAL
DATA VERIFICATION
We handle the continuation by using the SHORTMSG task, as described in 2.3.3, “Application
Reporting flow” on page 262. Here, we notice that immediately after the return from the
handleVIOResponse call that an acknowledgment job is submitted and completes successfully
for the tag that passed the NA request under the hsTag attribute.

2.3.3 Application Reporting flow
A mechanism is needed to make the KSYS subsystem aware of the status and configuration
changes happening to the applications that are managed on each monitored VM by the local
AME instance. This mechanism is implemented by the AR Messaging protocol.
We review the application-related KSYS functions that are available at the time of writing. As
shown in Example 2-236, the ksysmgr man page documents that a query command is
available on the KSYS side to check the application status and that the application
configuration can be done only by the ksysvmmgr command at the VM level.
Example 2-236 The application status check command available from the KSYS side
# man ksysmgr
...
* To display the health status of the registered
applications:
ksysmgr query app
You can use this command to check the status of the

registered applications inside each virtual machine.
The application status value GREEN indicates a
stable application. The YELLOW value indicates an
intermediate state of application in which the
application is attempted to be restarted. The RED
value indicates permanent failure of an application.
The HA policies for an application can be configured

in the VM only by using the ksysvmmgr command.
...
# ksysmgr q app
Name: app1
Status: GREEN
AppID: 1567267349133928000
VM: vmraix1
Critical: no
MaxRestart: 1
Monitoring: 1

#
An application with the critical attribute set to yes makes the KSYS subsystem react even if
the repeated application restart cycle at the VM level fails and the application reaches the
FAILURE state, as documented by the ksysvmmgr man page shown in Example 2-237.
Example 2-237 The critical application attribute as shown by ksysvmmgr man

# man ksysvmmgr
[vmraix1:root:/home/root:] man ksysvmmgr

...
critical
Marks the application as critical. The valid
values are Yes and No (default). If you mark an

application as critical, failure of the
application is notified to the KSYS subsystem
for further action.
...
The KSYS subsystem reacts by notifying the user about the issue and restarting the VM on
the same host. If the application reaches the FAILURE (RED) state again, the local restart
repeats, but no more than a pre-determined counter value. Currently, this counter value is 3
and is hardcoded. If this counter value is reached, the KSYS subsystem restarts the VM on
another host within the host group of the host currently hosting the VM.
The application-related KSYS function is based on the AR Messaging protocol. As introduced

in “VM Agent activation” on page 164, AR is handled on the VM side by the dedicated
AppReport thread of the VMM daemon. We denote the thread interchangeably with the AR
thread.
An application configuration change that is performed by the ksysvmmgr command is

considered by the VMM daemon, mainly by the AME thread, at the subsequent load of the
XML configuration file (at the command execution if -s is specified, or at the next ksysvmmgr
sync or next VMM restart). We described how ksysvmmgr and VMM interact for command
execution at the end of “VM Agent activation” on page 164. An application state change
originates at the AME level during an AME round execution, as described for some relevant
cases in “Application Management Engine” on page 176.
We encountered this kind of situation in Example 2-149 on page 183 and Example 2-151 on
page 184. You see in those two examples the AME log entries about an AME Notification
Vector being sent to AR by the doSendNotification2Ar() method.

3
Chapter 3. Planning and deploying IBM VM

Recovery Manager High
Availability for IBM Power
Systems
This chapter describes the planning, installation, and configuration of IBM VM Recovery
Manager High Availability (HA) for IBM Power Systems.
To implement the VM Recovery Manager HA solution, you must review your current HA
recovery plan and consider how the VM Recovery Manager HA solution can be integrated
into your current environment.

򐂰 VM Recovery Manager HA requirements
򐂰 VM Recovery Manager HA coexistence with other products
򐂰 VM Recovery Manager HA restrictions
򐂰 Installing the VM Recovery Manager HA solution
򐂰 Configuring VM Recovery Manager HA
򐂰 Setting up HA policies
򐂰 Setting up the VM Agent
򐂰 Setting contacts for event notification
򐂰 Uninstalling VM Recovery Manager HA

3.1 VM Recovery Manager HA requirements
Before you plan the implementation of the VM Recovery Manager HA solution, you must
understand the other entities and resources that the VM Recovery Manager HA solution
requires for disaster recovery (DR) operations.
You must meet the following requirements before you can install the VM Recovery Manager
HA solution:
򐂰 Software requirements
򐂰 Firmware (FW) requirements
򐂰 Installation and configuration requirements
򐂰 Hardware Management Console (HMC) requirements
򐂰 Host group requirements
򐂰 Networks requirements
򐂰 GUI requirements
3.1.1 Software requirements

Table 3-1 shows the software requirements for the VM Recovery Manager HA.
Table 3-1 VM Recovery Manager HA software component requirements

Component Requirement
KSYS controller IBM AIX 7.2 with Technology Level 2 Service

Pack 1 (7200-02-01) or later
HMC HMC Version 9 Release 9.1.0 or later
Virtual I/O Server (VIOS) VIOS V3.1.0.1 or later
Logical partition (LPAR) Each host must have one of the following
operating systems:
򐂰 AIX 6.1 or later
򐂰 Red Hat Enterprise Linux (RHEL) (Little
Endian) Version 7.4 or later (kernel version
3.10.0-693)
򐂰 SUSE Linux Enterprise Server (Little Endian)
Version 12.3 or later (kernel version
4.4.126-94.22)
򐂰 Ubuntu Linux distribution Version 16.04
򐂰 IBM i Version 7.2 or later
Virtual machine (VM) Agent At the moment, the VM Agent to monitor the VM
and applications can be installed only on the
following operating systems:
򐂰 AIX Version 6.1 and later
򐂰 RHEL (Little Endian) Version 7.4 or later
(kernel version 3.10.0-693)
򐂰 SUSE Linux Enterprise Server (Little Endian)
Version 12.3 or later (kernel version
4.4.126-94.22)

Note: You must install OpenSSL V1.0.1.516 or later for the AIX operating system. You can
download the OpenSSL software from AIX Web Download Pack Programs.
The latest version of the OpenSSL software is also included in the AIX base media.
3.1.2 Firmware requirements

The following section describes the FW requirements for the VM Recovery Manager HA:
򐂰 The VM Recovery Manager HA solution requires that the PowerVM Enterprise edition is
deployed in each of the hosts.
򐂰 The VM Recovery Manager HA solution supports the following minimal levels of Power
Systems servers:
– IBM POWER7+ systems that have the following FW levels:
• FW770.90 and later
• FW780.70 or later except for MMB systems (9117-MMB models)
– IBM POWER8 systems that have the following FW levels:
– IBM POWER9™ systems that have the following FW levels:
• FW910 and later
3.1.3 Installation and configuration requirements

The following section describes the installation and configuration requirements for the VM
Recovery Manager HA:
򐂰 You must have root authority to perform any installation tasks.
򐂰 The KSYS LPAR must have at least one 1-core CPU and 8 GB of memory. These
requirements can be higher if you have a large environment of more than 100 LPARs in
the data center.
򐂰 Ensure that you have enough space in the LPAR so that KSYS filesets can be installed
successfully. You must have 30 MB of disk space in the /opt directory and 200 MB of disk
space in the /var directory.
򐂰 You must check whether a KSYS installation is already in progress by using the ksysmgr q
cl command. If the KSYS software is installed, you must uninstall the KSYS software.
򐂰 For the installation of VM Agent, ensure that each VM meets the following disk space
requirements:
– At least 100 MB disk space in the /usr directory to install the VM Agent filesets.
– At least 1 GB disk space in the /var directory for log files.
򐂰 For the VIOS, ensure that you have two VIOSs per host. You can have a maximum of 24
VIOSs in a single host group. If more than two VIOSs are in a host, you can exclude this
requirement from the KSYS configuration settings.
Chapter 3. Planning and deploying IBM VM Recovery Manager High Availability for IBM Power Systems 267
3.1.4 Host group requirements
The following section describes the host group requirements for the VM Recovery Manager
HA:
򐂰 The host group can be named with a logical name that can be up to 64 characters long.
򐂰 A single KSYS LPAR can manage up to four host groups. A host group can have up to 12
hosts.
򐂰 All the hosts on the host group must be configured for network and storage such that any
VM from any host can be migrated to any other host within the host group.
򐂰 For each host group, the KSYS subsystem requires two disks for health cluster
management. A disk of at least 10 GB, called the repository disk, is required for health
monitoring of the hosts, and another disk of at least 10 GB, called the HA disk, is required
for health data tracking for each host group. All these disks must be accessible to all the
VIOSs on each of the hosts on the host group, as shown in Figure 3-1.
Figure 3-1 KSYS disk health cluster management
3.1.5 HMC requirements

The following section describes the HMC requirements for the VM Recovery Manager HA:
򐂰 The VM Recovery Manager HA solution requires HMC Version 9 Release 9.1.0 or later. It
is a best practice to have each host managed by two HMCs.
򐂰 Ensure that you can perform a Live Partition Mobility (LPM) operation on the VMs among
the hosts that you want to be part of VM Recovery Manager HA management. You can
use HMC-based LPM validation to ensure that the VMs can move from a host to any other
host on the host group.

򐂰 For POWER7+ systems and POWER8 systems in which the Simplified Remote Restart
(SRR) attribute is not set for a VM in the HMC, when the VM is moved from a source host
to a destination host, the time-of-day change might not be updated correctly. To retain the
correct time-of-day clock for a VM across the hosts, you must set the VIOS partitions
profile of both the source host and destination host as the time reference partition by
enabling the Time Reference Partition option in the HMC, as shown in Figure 3-2.
Figure 3-2 VIO partition properties: Checking the Enable TIme Reference
򐂰 To migrate an IBM i VM from the source host to the destination host, verify that the
Restricted I/O Partition check box for the IBM i LPAR is selected in the HMC. For more
information about the steps to verify the restricted I/O mode, see IBM Knowledge Center.
򐂰 Ensure that the automatic reboot attribute is not set for any VM in the HMC. The KSYS
validates this attribute and notifies you to disable this attribute. If you set this attribute, it
can lead to unpredictable results, such as a VM restart on two hosts simultaneously.
򐂰 When you add a host or manage a VM that is co-managed by the HMC and PowerVM
NovaLink, set the HMC to the master mode. Otherwise, the discovery operation fails, and
the VMs on the host are not monitored for HA.
3.1.6 Network requirements

The following section describes the network requirements for the VM Recovery Manager HA:
򐂰 All VMs that are managed by the VM Recovery Manager HA solution must use virtual I/O
resources through VIOS. The VMs must not be connected to a physical network adapter
or any dedicated devices.
򐂰 Storage area network (SAN) connectivity and zoning must be configured so that VIOS can
access the disks that are relevant to the hosts.
򐂰 Ensure that the KSYS LPAR has HTTP Secure (HTTPS) connectivity to all the HMCs that
can manage the hosts on the host group.
򐂰 The same virtual LAN (VLAN) must be configured across the hosts.
򐂰 Ensure that redundant connections are established from the KSYS to HMC and from HMC
to VIOS LPARs, as shown in Figure 3-3. Any connectivity issues between KSYS, HMC,
and VIOS LPARs can lead to disruption in the regular data collection activity and DR
operations.
Figure 3-3 VM Recovery Manager HA for network requirements
3.1.7 GUI requirements

The following section describes the GUI requirements for the VM Recovery Manager HA:
򐂰 The LPAR in which you want to install the GUI filesets must be running IBM AIX 7.2 with
Technology Level 2 Service Pack 1 (7200-02-01) or later. You can choose to install the
GUI server fileset on one of the KSYS nodes.
򐂰 The LPAR in which you are installing the GUI server must run in an Enhanced Korn shell
that uses the /usr/bin/ksh93 shell script.
򐂰 The LPAR in which you are installing the GUI server fileset must have at least one 1-core
CPU and 8 GB of memory.

3.2 VM Recovery Manager HA coexistence with other products
The VM Recovery Manager HA solution and other products can coexist in the same
environment with the following considerations.
3.2.1 IBM Power Virtualization Center

If you use the IBM Power Virtualization Center (IBM PowerVC) solution to perform the LPM
operation of VMs, ensure that the VM remains within the host group that is defined in the
KSYS configuration settings. Otherwise, the VM might be lost from the VM Recovery
Manager HA configuration and cannot be monitored for failures.
3.2.2 IBM PowerHA SystemMirror Software

Even if you are using the PowerHA SystemMirror solution for application HA, you can use the
VM Recovery Manager HA solution for hardware failures and automated VM restart
operations. This type of deployment can bring the entire cluster back online by restarting the
failed PowerHA node or VMs on other hosts.
If you deployed the PowerHA SystemMirror and VM Recovery Manager HA solutions

together, consider the following guidelines:
򐂰 If the primary node of the PowerHA configuration fails, the PowerHA solution starts the
workload on a secondary node of the PowerHA configuration. The VM Recovery Manager
HA solution restarts the failed VM, which is the old primary LPAR of PowerHA, on another
host. The restarted VM can rejoin the cluster and continue to provide a higher level of HA.
򐂰 If you deploy the PowerHA solution and the VM Recovery Manager HA solution together,
you must not configure the VM Agent in the PowerHA VMs because PowerHA application
management is sufficient for application HA.
򐂰 Ensure that you define the anticollocation policies for nodes or LPARs of the PowerHA
cluster.
The VM Recovery Manager HA solution can coexist with other cluster technologies, such as
Oracle RAC and Veritas Cluster Server.
3.2.3 PowerVM NovaLink

Currently, the VM Recovery Manager HA solution works with the PowerVM environment that
is configured with HMC. When NovaLink and HMCs are deployed together, the VM Recovery
Manager HA solution can work only if HMCs are set to in the master mode. The VM Recovery
Manager HA solution communicates to HMC for continuous monitoring and relocation
operations.
3.3 VM Recovery Manager HA restrictions
This section describes some restrictions of the VM Recovery Manager HA solution to take
into account.
3.3.1 Migration restriction

򐂰 You cannot run the LPM operation in parallel on multiple hosts by using the ksysmgr
command. You must specify multiple VMs in a comma-separated list in the ksysmgr
command. Also, you can perform the LPM operation on a list of VMs simultaneously only if
all the VMs are present in the same host.
򐂰 The flexible capacity policy applies only for VM failover operations. The flexible capacity
function is not supported for VMs that are migrated by using the LPM operation.
򐂰 The flexible capacity reduction rules are applied even if 100% of resources are available
on the target host.
򐂰 The flexible capacity policy applies only on CPU and memory resources. It is not applied
on I/O resources. You must ensure enough I/O resources are available in the target host.
3.3.2 VM Agent restrictions

򐂰 The ksysvmmgr start|stop app command supports only one application concurrently.
򐂰 The ksysvmmgr suspend|resume command does not support application dependencies.
򐂰 For all applications that are installed on non-rootvg disks, you must enable the automatic
varyon option for volume groups (VGs) and automount for file systems after the VM is
restarted on the AIX operating system.
򐂰 To get the correct version information for a SAPHANA type of application, you must have
an instance name and database name when you add the application in the VM Agent
configuration settings. Otherwise, you might not get the version information.
򐂰 If the application is in any one of the failure states, for example, NOT_STOPPABLE,
NOT_STARTABLE, ABNORMAL, or FAILURE, you must fix the issue, and then use the ksysvmmgr
start|resume application command to start and monitor the application. You can also
modify the XML configuration file and run the ksysvmmgr sync command.
򐂰 If the KSYS cluster is deleted, or if a VM is unmanaged from the HA management, the VM
Agent daemon becomes inoperative. In such cases, you must manually start the VM
Agent daemon in the VM.
򐂰 On a critical application failure, the KSYS subsystem continues to relocate the VM from
one host to another host even if the VM reaches its home host. For example, a host group
contains two hosts (host1 and host2) and a registered critical application in the vm1_host1
VM fails. In this case, the KSYS subsystem relocates the vm1_host1 VM to host2. If the
application does not start properly with a NORMAL status, the KSYS subsystem again
moves the vm1_host1 VM to host1, which is its home host. The KSYS subsystem
continues this relocation process until the application status becomes NORMAL.

3.3.3 GUI restrictions
򐂰 The VM Recovery Manager HA GUI does not support multiple sessions on a single VM.
򐂰 The VM Recovery Manager HA GUI does not support user and role management.
򐂰 You cannot delete the KSYS cluster by using the VM Recovery Manager HA GUI.
򐂰 The Activity window under each host group, host, and VM shows the activity log
information for the current session. If you refresh the window or log in to the session again,
the data in the Activity window might be lost.
3.3.4 VM Recovery Manager HA filesets and structure

The VM Recovery Manager HA package consists of filesets for the installation of KSYS, VM
Agent, and GUI.
The following sections describe the key components of the VM Recovery Manager HA
solution.
KSYS
KSYS is base product software that must be installed on an AIX LPAR. It provides the
technical foundation of VM Recovery Manager HA and command line-based administration
by using the ksysmgr command.
Here are the KSYS filesets:

򐂰 ksys.hautils.rte
򐂰 ksys.ha.license
򐂰 ksys.main.cmds
򐂰 ksys.main.msg.en_US.cmds
򐂰 ksys.main.rte
VM Agent
The VM Agent is an optional fileset that can be installed on the guest VMs that are running
either AIX or Linux operating systems. If you install the VM Agent, you can monitor the
individual VMs and the applications that are running in the VM. Otherwise, only host level
monitoring is supported.
򐂰 AIX VM Agent fileset:
ksys.vmmon.rte
򐂰 RHEL VM Agent package:
vmagent-1.3.0-1.0.el7.ppc64le
򐂰 SUSE Linux Enterprise Server package:
vmagent-1.3.0-1.0.suse123.ppc64le
GUI
An optional fileset that can be installed on an AIX LPAR for accessing the VM Recovery
Manager HA solution by using a GUI. You can install the server on the KSYS LPAR also.
򐂰 ksys.ui.agent: The GUI agent fileset that must be installed on the KSYS nodes.
򐂰 ksys.ui.server: A GUI server fileset that must be installed on the system that manages
the KSYS nodes. This fileset can be installed on one of the KSYS nodes.
򐂰 ksys.ui.common: A GUI common fileset that must be installed along with both the
ksys.ui.server (GUI server) fileset and the ksys.ui.agent (GUI agent) fileset.
3.4 Installing the VM Recovery Manager HA solution
The VM Recovery Manager HA solution provides HA management for IBM Power Systems
servers with PowerVM virtualization. After you plan the implementation of the VM Recovery
Manager HA solution, you can install the VM Recovery Manager HA software. The VM
Recovery Manager HA solution uses other subsystems such as HMC and VIOSs that must
exist in your production environment.
The VM Recovery Manager HA software can be enabled when you have the following
subsystems in your production environment:
򐂰 VIOS Version 3.1.0.1 or later must be installed on all VIOS partitions that are part of the
host group. Host Monitor (HM), which is a key component of the VM Recovery Manager
HA solution, is installed on the VIOS by default. The HM component is enabled and used
when you install the VM Recovery Manager HA filesets. The VM Recovery Manager HA
solution requires two VIOS per host.
򐂰 HMC V9 R 9.1.0 or later must be used to manage all hosts that are part of the cluster.
Figure 3-4 shows the components of VM Recovery Manager HA.
Figure 3-4 Components of VM Recover Manager HA

To install the VM Recovery Manager HA solution, you must first install the KSYS filesets. After
the KSYS software is installed, the KSYS subsystem automatically monitors the health of
hosts by enabling the HMs in the VIOS partitions of each host that is part of the VM Recovery
Manager HA management.
You can optionally install the VM Agents in the VMs that run AIX or Linux operating systems
to monitor the health of an individual VM and applications that run in the VMs. You can also
install the GUI server for the VM Recovery Manager HA solution to use the GUI by using a
browser.
Complete the following procedures to install the VM Recovery Manager HA solution:

1. Install the VIOS interim fix.
2. Install the KSYS software.
3. Optional: Install VM Agents in the VMs.
4. Install the GUI server.
3.4.1 Installing the VIOS interim fix

You must install the VIOS interim fix IJ10896m2a.181101.epkg.Z on all VIOS instances that
are included in the KSYS subsystem. This interim fix provides fixes to VIOSs that support VM
Recovery Manager. The VIOS interim fix is in the vios_3.1.0.10_ifixes directory in the VM
Recovery Manager HA package, as shown in Example 3-1.
Example 3-1 Listing the VIOS interim fix directory in the VM Recovery Manager HA package
# pwd
/mnt/GDR_BUILDS/1844B_VMR
# ls -la
total 16
drwxr-xr-x 5 root system 256 Nov 04 01:31 .
drwxrwxrwx 143 root system 8192 Nov 04 01:32 ..
drwxr-xr-x 3 root system 256 Nov 04 01:31 installp
drwxr-xr-x 3 root system 256 Nov 04 01:31 usr
drwxr-xr-x 2 root system 256 Nov 04 01:31 vios_3.1.0.10_ifixes
Important: Install the interim fix before you initialize the KSYS subsystem.
If you already have a shared pool that is configured in your environment, ensure that any
cluster services are not active. Stop any active cluster services by running the following
command:
clstartstop -stop -n clustername -m hostname
Run the command in each of the managed VIOS instances, as shown in Example 3-2.
Example 3-2 Installing the VIOS interim fix

$ updateios -install -dev /tmp/fix -accept
*******************************************************************************
EFIX MANAGER PREVIEW START
*******************************************************************************
+-----------------------------------------------------------------------------+
Efix Manager Initialization
+-----------------------------------------------------------------------------+
Initializing log /var/adm/ras/emgr.log ...
+-----------------------------------------------------------------------------+
Processing Efix Package 1 of 1.
+-----------------------------------------------------------------------------+
Efix package file is: /tmp/fix/IJ10896m2a.181102.epkg.Z
MD5 generating command is /usr/bin/csum
MD5 checksum is 961dcf33ab5bcbd2d8b0adefcbc57f10
Accessing efix metadata ...
Processing efix label "IJ10896m2a" ...
Verifying efix control file ...
Example 3-3 shows the continuation of the installation of the VIOS interim fix.
Example 3-3 Accepting the installation of the VIOS interim fixes

*******************************************************************************
EFIX MANAGER PREVIEW END
*******************************************************************************
+-----------------------------------------------------------------------------+
Operation Summary
+-----------------------------------------------------------------------------+
Log file is /var/adm/ras/emgr.log
EPKG NUMBER LABEL OPERATION RESULT

=========== ============== ================= ==============
1 IJ10896m2a INSTALL PREVIEW SUCCESS
ATTENTION: system reboot will be required by the actual (not preview) operation.
Please see the "Reboot Processing" sections in the output above or in the
/var/adm/ras/emgr.log file.
Return Status = SUCCESS
Continue the installation [y|n]?Y
Example 3-4 shows the summary of the installation of the VIOS interim fix.
Example 3-4 Summary of the installation of VIOS interim fix

+-----------------------------------------------------------------------------+
Operation Summary
+-----------------------------------------------------------------------------+
Log file is /var/adm/ras/emgr.log
EPKG NUMBER LABEL OPERATION RESULT

=========== ============== ================= ==============
1 IJ10896m2a INSTALL SUCCESS
ATTENTION: system reboot is required. Please see the "Reboot Processing"

sections in the output above or in the /var/adm/ras/emgr.log file.
Return Status = SUCCESS

File /etc/inittab has been modified.
One or more of the files listed in /etc/check_config.files have changed.

See /var/adm/ras/config.diff for details.
$
Note: The installation of the VIOS interim fix was cropped among these three examples.
Verify whether the installation of the interim fix is successful by running lssw, as shown in
Example 3-5.
Example 3-5 Final output of the lssw command
ID STATE LABEL INSTALL TIME UPDATED BY ABSTRACT

=== ===== ========== ================= ========== ===============================
1 S IJ10896m2a 11/08/18 15:56:38 Fixes to support VM Recover Manager
STATE codes:
S = STABLE
Important: After the installation successfully completes, a system restart is required.
If the cluster services were stopped, start the cluster services by running the following
command:
clstartstop -start -n clustername -m hostname
3.4.2 Installing the KSYS software

You can use the installp command in the AIX LPAR to install KSYS filesets that are included
in the package. To install the KSYS software, complete the following steps:
1. Ensure all the prerequisites that are specified in 3.1, “VM Recovery Manager HA
requirements” on page 266 are complete.
2. Go to the directory that contains the images that you want to install and run the command
that is shown in Example 3-6.
Example 3-6 Installing the KSYS software

# installp -acFXYd . ksys.hautils.rte ksys.ha.license ksys.main.cmds
ksys.main.msg.en_US.cmds ksys.main.rte ksys.ui.agent ksys.ui.common
3. Verify whether the installation of filesets is successful by running the command that is
Example 3-7 Listing the filesets installed at the KSYS node

# lslpp -l ksys.ha.license ksys.hautils.rte ksys.main.cmds
ksys.main.msg.en_US.cmds ksys.main.rte
Fileset Level State Description
----------------------------------------------------------------------------
Path: /usr/lib/objrepos
ksys.ha.license 1.3.0.0 COMMITTED Base Server Runtime
ksys.hautils.rte 1.3.0.0 COMMITTED Base Server Runtime
ksys.main.cmds 1.3.0.0 COMMITTED Base Server Runtime
ksys.main.msg.en_US.cmds 1.3.0.0 COMMITTED Base Server Runtime
ksys.main.rte 1.3.0.0 COMMITTED Base Server Runtime
Path: /etc/objrepos
ksys.hautils.rte 1.3.0.0 COMMITTED Base Server Runtime
ksys.main.cmds 1.3.0.0 COMMITTED Base Server Runtime
ksys.main.rte 1.3.0.0 COMMITTED Base Server Runtime
4. To check the command-line utility of the KSYS subsystem, run the

/opt/IBM/ksys/ksysmgr command, as shown in Example 3-8. The KSYS subsystem
might take a few minutes to run the command for the first time. You can add the
/opt/IBM/ksys directory to your PATH environment variable so that you can access the
ksysmgr command easily.
Example 3-8 Checking the ksysmgr command

# /opt/IBM/ksys/ksysmgr
No command arguments found
Here is a list of available actions for ksysmgr:

Available action
add
delete
discover
help
manage
unmanage
modify
move
query
recover
restore
restart
resync
report
cleanup
sync
pair
verify
monitor
lpm
5. After the successful installation of the KSYS filesets, check whether the class IDs are
reserved by running the command that is shown in Example 3-9.
Example 3-9 Checking the ID class

# cat /usr/sbin/rsct/cfg/ct_class_ids | egrep
"IBM.VMR_HMC|IBM.VMR_CEC|IBM.VMR_LPAR|IBM.VMR_VIOS|IBM.VMR_SSP|IBM.VMR_SITE|IBM.VM
R_SA|IBM.VMR_DP|IBM.VMR_DG|IBM.VMR_KNODE|IBM.VMR_KCLUSTER|IBM.VMR_HG|IBM.VMR_APP"
IBM.VMR_HMC 510
IBM.VMR_CEC 511
IBM.VMR_LPAR 512
IBM.VMR_VIOS 513
IBM.VMR_SSP 514

IBM.VMR_SITE 515
IBM.VMR_SA 516
IBM.VMR_DP 517
IBM.VMR_DG 518
IBM.VMR_KNODE 519
IBM.VMR_KCLUSTER 520
IBM.VMR_HG 521
IBM.VMR_APP 522
6. If the IBM.VMR_APP class is not available in the output, manually add the IBM.VMR_APP 522
entry into the /usr/sbin/rsct/cfg/ct_class_ids file and refresh the Reliable Scalable
Cluster Technology (RSCT) subsystem by running the command that is shown in
Example 3-10.
Example 3-10 Refreshing the RSCT subsystem

# rmcctrl -z
#
# rmcctrl -s
0513-059 The ctrmc Subsystem has been started. Subsystem PID is 11534722.
#
3.4.3 Installing VM Agents

VM Agents are components that are installed in VMs or LPARs. These optional agents offer
robust monitoring of the VMs and applications that are running in VMs. You can manage HA
applications in VMs by using a lightweight application monitoring framework.
Installing a VM Agent in an AIX VM

To install a VM Agent in an AIX VM, complete the following steps:
1. Ensure that all the prerequisites that are specified in 3.1, “VM Recovery Manager HA
requirements” on page 266 are complete.
2. Run the following command in the AIX VM, as shown in Example 3-11.
Example 3-11 Installing a VM Agent in an AIX VM

# installp -acFXYd . ksys.vmmon.rte
3. To verify whether the installation of VM Agent is successful, run the lslpp command
Example 3-12 Checking the VM Agent installation

# lslpp -l ksys.vmmon.rte
Fileset Level State Description
----------------------------------------------------------------------------
Path: /usr/lib/objrepos
ksys.vmmon.rte 1.3.0.0 COMMITTED Base Server Runtime
Path: /etc/objrepos
ksys.vmmon.rte 1.3.0.0 COMMITTED Base Server Runtime
4. Ensure that the ksysvmmgr command that is shown in Example 3-13 and the binary file for
the VM Agent daemon shown Example 3-14 both completed successfully.
Example 3-13 Checking the ksysvmmgr

# /usr/sbin/ksysvmmgr
Usage: ksysvmmgr [<FLAGS>] <ACTION> [<CLASS>] [<NAME>] [<ATTRIBUTES>]
KSYSVMMGR-002(ERROR): <ACTION> parameter is mandatory.
For <ACTION> parameter, possible actions are :
"add,query,modify,delete,status,resume,suspend,start,stop,help,start,stop,status,s
ync".
Retry using one of the possible actions.
Or refer to complete output of "ksysvmmgr -h"
Example 3-14 Checking the binary file of the VM Agent daemon

# /usr/sbin/ksys_vmmd
0513-095 The request for subsystem refresh was completed successfully.
5. To verify whether the VM Agent daemon is enabled, run the lssrc -s ksys_vmm command.
The status of the ksys_vmm subsystem must be Active in the output of this command, as
Example 3-15 Checking the agent daemon

# lssrc -s ksys_vmm
Installing a VM Agent in a Linux VM

To install the VM Agent Red Hat Package Manager (RPM) packages in a Linux VM, complete
the following steps:
1. Ensure that the following RSCT packages are installed in the Linux VM, as shown
Example 3-16:
– rsct.core
– rsct.opt.storagerm
– rsct.core.utils
– rsct.basic
– DynamicRM
Example 3-16 Listing the RSCT packages

# rpm -qa | egrep
"rsct.core|rsct.opt.storagerm|rsct.core.utils|rsct.basic|DynamicRM"
rsct.basic-3.2.2.3-17144.ppc64le
rsct.core-3.2.2.3-17144.ppc64le
rsct.core.utils-3.2.2.3-17144.ppc64le
DynamicRM-2.0.5-1.ppc64le
rsct.opt.storagerm-3.2.2.3-17144.ppc64le

Note: You can download the packages from Service and productivity tools.
For more information about configuring the repository to easily install those packages, see
IBM Knowledge Center.
2. Install the VM Agent RPM packages based on the following Linux distributions in the VM:
– In RHEL (Little Endian) VMs, run the command that is shown in Example 3-17.
Example 3-17 Installing VM Agent in Red Hat Linux VM

# rpm -ivh vmagent-1.3.0-1.0.el7.ppc64le.rpm
Preparing... ################################# [100%]
Updating / installing...
1:vmagent-1.3.0-1.0.el7 ################################# [100%]
Starting ksys_vmm daemon.
Ensure that the Resource Monitoring and Control (RMC) connection between the VMs
and HMC exists. If the firewall is enabled on the RHEL VM, the RMC connection might
be broken. Modify the firewall on the VMs to allow the RMC connection with the HMC.
– In SUSE Linux Enterprise Server (Little Endian) VMs, run the command that is shown
in Example 3-18.
Example 3-18 Installing VM Agent in SUSE Linux Enterprise Server

# rpm -ivh vmagent-1.3.0-1.0.suse123.ppc64le.rpm
Preparing... ################################# [100%]
Updating / installing...
1:vmagent-1.3.0-1.0.suse123 ################################# [100%]
Starting ksys_vmm daemon.
ksys_vmm has been started.
3.5 Configuring VM Recovery Manager HA

After the VM Recovery Manager HA solution is installed, you must complete some mandatory
configuration steps before you use the HA feature of the VM Recovery Manager HA solution.
You can use the ksysmgr command or the VM Recovery Manager HA GUI (see Chapter 4,
“IBM VM Recovery Manager High Availability GUI deployment” on page 323) to interact with
the KSYS daemon to manage the entire environment for HA.
The VM Recovery Manager HA solution monitors the hosts and the VMs when you add
information about your environment to the KSYS configuration settings. Complete the
following steps to set up the KSYS subsystem:
1. Initialize the KSYS cluster.
2. Add HMCs.
3. Add hosts.
4. Create host groups.
5. Configure VMs.
6. Configure VIOS.
7. Set contacts for event notification.
8. Enable the HA monitoring.
9. Discover and verify the KSYS configuration.
10.Optional: Back. up the configuration data.
The ksysmgr command

In the following sections, the ksysmgr command is used to start adding resources to the KSYS
node configuration. You notice that the ksysmgr command accepts aliases, for example, using
the ksysmgr create cluster command also has the same effect as using the ksysmgr add
ksyscluster command:
ksysmgr add ksyscluster
[<ksysclustername>]
type=<HA>
ksysnodes=<ksysnode>
[sync=<yes|no>]
add => ad*, cr*, make, mk
ksyscluster => ksyscl*, cl*
To find the syntax and aliases of every action or parameter that is accepted by the ksysmgr
command, use the -h option, as shown in Example 3-19.
Example 3-19 Finding the syntax and aliases of every action or parameter that is accepted by ksysmgr
# ksysmgr -h
No command arguments found
Here is a list of available actions for ksysmgr:
Available action
add
delete
discover
help
manage
unmanage
modify
move
query
recover
restore
restart
resync
report
cleanup
sync
pair
verify
monitor
lpm

3.5.1 Initializing the KSYS cluster
The KSYS environment relies on RSCT to create its cluster on the KSYS LPAR. After you
create the KSYS cluster, various daemons of RSCT and KSYS are activated. The KSYS node
can then process the commands that you specify in the command line.
To create and initialize a KSYS cluster, complete the following steps in each of the KSYS
LPARs:
1. Configure a cluster and add the KSYS node to the cluster by running the command that is
Example 3-20 Creating a cluster and adding a KYSYS node to the cluster
# ksysmgr add ksyscluster ITSO_HA ksysnodes=ksys7005rbdr type=HA
Adding node to current cluster configuration
Ksyscluster has been created, please run: "ksysmgr verify ksyscluster
<ksysclustername>"
2. Verify the KSYS cluster configuration by running the command that is shown in
Example 3-21.
Example 3-21 Verifying the cluster configuration

# ksysmgr verify ksyscluster ITSO_HA
Verified, Please run: "ksysmgr sync ksyscluster <ksysclustername>"
3. Deploy the KSYS cluster by running the command that is shown in Example 3-22.
Example 3-22 Deploying the KSYS cluster

# ksysmgr sync ksyscluster ITSO_HA
4. You can perform steps 1 - 3 by running the command that is shown in Example 3-23.
Example 3-23 Creating and deploying a KSYS cluster

# ksysmgr add ksyscluster ITSO_HA ksysnodes=ksys7005rbdr sync=yes type=HA
Adding node to current cluster configuration
Ksyscluster has been created, running verify now
Ksyscluster has been verified, running sync now
5. Verify that the KSYS cluster is created successfully by running one of the commands that
are shown in Example 3-24.
Example 3-24 Verifying the KSYS cluster

# ksysmgr query ksyscluster
Name: ITSO_HA
State: Online
Type: HA
# lsrpdomain
ITSO_HA Online 3.2.4.0 No 12347 12348
# lssrc -s IBM.VMR
Important: The output message must display the state of the KSYS cluster as Online.
3.5.2 Adding HMCs

The KSYS interacts with the HMC for discovery, verification, monitoring, recovery, and
cleanup operations. HMCs provide details about the hosts and VIOS partitions that are
managed by the HMCs. The VM Recovery Manager HA solution cannot be implemented
without configuring the HMCs.
Note: The HMC user, whose user name and password details are provided to the KSYS,
must have at least hmcsuperadmin privileges and remote access. The KSYS subsystem
uses the Representational State Transfer (REST) API to communicate with the HMCs in
the environment. Therefore, ensure that your environment allows HTTPS communication
between the KSYS and HMC subsystems.
To add the HMCs to the KSYS configuration setting, complete the following steps in the KSYS
LPAR:
1. Add the HMC with user name hscroot and password xyz123 by running the command that
is shown in Example 3-25.
Example 3-25 Adding the HMC

# ksysmgr add hmc rthmc3 login=hscroot password=abc123 ip=9.3.18.159
HMC rthmc3 added successfully
# ksysmgr add hmc rthmc6 login=hscroot password=abc123 ip=9.3.207.78

HMC rthmc6 added successfully
2. Repeat step 1 to add multiple HMCs.

3. Verify the HMCs that you added by running the command that is shown in Example 3-26.
Example 3-26 Listing the HMC that is added to the KSYS

# ksysmgr query hmc
Name: rthmc6
Ip: 9.3.207.78
Login: hscroot
Managed Host List:
Host Name Uuid

========= ====
rt12-8286-42A-2100E5W 4715697c-a486-3cbb-8b5a-2637fa4db054
rt14-8286-42A-SN2100DEW 6e2f37f5-1824-347a-99fd-488f818086dd
rt19-8286-42A-2100D5W d2038c84-8484-3f08-a608-6436e0f0f928
e17m1 2b67b2b2-3eef-3047-8e9e-0c6d5f043201
mbu 664989e6-b6c2-3bd3-b351-d49b08e7da94
rose 3d26082d-7521-3d52-a27a-38aa78e4c314

e9m1 224a76ad-7087-33ae-ad28-0202a14834c9
============================================================================
Name: rthmc3
Ip: 9.3.18.159
Login: hscroot
Managed Host List:
Host Name Uuid

========= ====
e52m3-8408-E8D-21DD71T e17e3ddc-71cc-327f-884b-4fa6a411bbb9
e60m2-8408-E8D-21DD4DT 2d8b3d3a-ca43-3147-9d02-fd999bfc7ff4
e46m4-8247-22L-2123D7A 5f322ea0-00dd-3615-acbb-a958add020b2
rt13-8286-42A-21E0B2V 4462c37c-65c6-3614-b02a-aa09d752c2ee
e52m2-8408-E8D-21DD44T ac43f075-3a03-3509-864a-cfd979139cb0
rt11-8286-42A-0607585 d04d8a4a-99fa-3e4b-80bc-c9d9716bd8f8
e46m5-8247-22L-212A71A c13ac0ed-f458-3b2e-be92-bf61b0e88933
e60m3-8408-E8D-21DD4BT a71b1e42-4f08-360d-910a-7689b17eb09b
============================================================================
This output shows all managed hosts for each HMC.
3.5.3 Adding hosts

After the HMCs are added to the KSYS subsystem, you can review the list of hosts that are
managed by each HMC, as shown in Example 3-26 on page 284, and then identify the hosts
that you want to add to the KSYS for HA.
To add hosts to the KSYS configuration, complete the following steps in the KSYS LPAR:
1. Add the managed hosts rt11-8286-42A-0607585 and rt12-8286-42A-2100E5W to the KSYS
by running the command that is shown in Example 3-27.
Example 3-27 Adding hosts to the KSYS

# ksysmgr add host rt11-8286-42A-0607585
Host rt11-8286-42A-0607585 added successfully
# ksysmgr add host rt12-8286-42A-2100E5W

Host rt12-8286-42A-2100E5W added successfully
Tip: If the host is connected to more than one HMC, you must specify the universally
unique identifier (UUID) of the host.
2. Repeat step 1 for all hosts that you want to add to the KSYS subsystem.
3. Verify the hosts that you added by running the command that is shown in Example 3-28.
Example 3-28 Verifying hosts that are added to the KSYS

# ksysmgr query host
Name: rt12-8286-42A-2100E5W
UUID: 4715697c-a486-3cbb-8b5a-2637fa4db054
VIOS: rt12v2
rt12v1
HMCs: rthmc6
HA_monitor: enable
Name: rt11-8286-42A-0607585
UUID: d04d8a4a-99fa-3e4b-80bc-c9d9716bd8f8
VIOS: rt11v2
rt11v1
HMCs: rthmc3
HA_monitor: enable
3.5.4 Creating host groups

You can group a set of hosts depending on your business requirements. Each host in the
KSYS subsystem must be a part of a host group.
The KSYS subsystem creates a health monitoring Shared Storage Pool (SSP) cluster across
the VIOSs that are part of the host group. The health cluster monitors the health of all VIOSs
across the cluster and retains the health data that is available to the KSYS subsystem by
using a VIOS on the host group. The SSP cluster is used only by the KSYS. You must not use
this SSP cluster for any other purpose. You can continue to use virtual Small Computer
System Interface (vSCSI) or N_Port ID Virtualization (NPIV) modes of the cluster. However, if
an SSP cluster exists in your environment, the KSYS subsystem does not deploy any new
SSP clusters, but instead uses the existing SSP cluster for health management. However, if
an existing SSP cluster is used, the KSYS subsystem might not support VIOS management.
The KSYS subsystem requires two disks to create the health monitoring SSP cluster across
the VIOSs on the host group. A disk of at least 10 GB is required to monitor health of the
hosts, which called as repository disk, and another disk of at least 10 GB is required to track
health data, which is called a HA disk, for each host group. These disks must be accessible to
all the managed VIOSs on each of the hosts on the host group. You must specify the disk
details when you create the host group or before you run the first discovery operation. You
cannot modify the disks after the discovery operation is run successfully. If you want to modify
the disks, you must delete the host group and re-create the host group with the disk details.
To create host group in the KSYS subsystem, complete the following steps in the KSYS
LPAR:
1. Identify all VIOSs that are managed by KSYS, as shown in Example 3-29.
Example 3-29 Listing all VIOSs that are managed by KSYS

# ksysmgr query vios
Name: rt12v1
UUID: 29798A5A-F115-4017-973F-0CF05FE26A97
Host: rt12-8286-42A-2100E5W
Version: VIOS 3.1.0.00
State: MANAGED
HM_versions: Unknown

Name: rt12v2
UUID: 124F4082-8307-4BF4-8DA2-5CC1E3F0DA7A
Host: rt12-8286-42A-2100E5W
State: MANAGED
Name: rt11v1
UUID: 5F48ABC5-8188-4C70-8F9F-84403FB29DC3
Host: rt11-8286-42A-0607585
State: MANAGED
Name: rt11v2
UUID: 55F05794-AF34-45E2-83DF-4DE40A7D6B7E
Host: rt11-8286-42A-0607585
State: MANAGED
2. To identify all the available shared disks by VIOS so that you can designate the repository
disk and the HA disk for the SSP cluster, run the commands that are shown in
Example 3-30.
Example 3-30 Identifying all available shared disks by VIOS

# ksysmgr query viodisk vios=rt12v1,rt12v2,rt11v1,rt11v2
Looking for free shared disks in these VIOSs:

rt12v2
rt12v1
rt11v2
rt11v1
This can take a few minutes
These are the shared free disks which appear on all VIOS in the list provided:
DiskNames are as they appear on VIOS rt12v2
DiskName Size ViodiskID

-------------------------------------------------------------------------------------------------
hdisk0 20480 01M0lCTTIxNDUxMjQ2MDA1MDc2ODAyODEwNDI2ODgwMDAwMDAwMDAwMDExMA==
hdisk2 20480 01M0lCTTIxNDUxMjQ2MDA1MDc2ODAyODEwNDI2ODgwMDAwMDAwMDAwMDExMQ==
hdisk3 20480 01M0lCTTIxNDUxMjQ2MDA1MDc2ODAyODEwNDI2ODgwMDAwMDAwMDAwMDExMg==
hdisk4 20480 01M0lCTTIxNDUxMjQ2MDA1MDc2ODAyODEwNDI2ODgwMDAwMDAwMDAwMDExRA==
It is possible also identify all available shared disks by host, as shown in Example 3-31.
Example 3-31 Identifying all available shared disks by hosts

# ksysmgr query viodisk hosts=rt12-8286-42A-2100E5W,rt11-8286-42A-0607585
Looking for free shared disks in these VIOSs:

rt12v2
rt12v1
rt11v2
rt11v1
This can take a few minutes
These are the shared free disks which appear on all VIOS in the list provided:
DiskNames are as they appear on VIOS rt12v2
DiskName Size ViodiskID

-------------------------------------------------------------------------------------------------
hdisk0 20480 01M0lCTTIxNDUxMjQ2MDA1MDc2ODAyODEwNDI2ODgwMDAwMDAwMDAwMDExMA==
hdisk2 20480 01M0lCTTIxNDUxMjQ2MDA1MDc2ODAyODEwNDI2ODgwMDAwMDAwMDAwMDExMQ==
hdisk3 20480 01M0lCTTIxNDUxMjQ2MDA1MDc2ODAyODEwNDI2ODgwMDAwMDAwMDAwMDExMg==
hdisk4 20480 01M0lCTTIxNDUxMjQ2MDA1MDc2ODAyODEwNDI2ODgwMDAwMDAwMDAwMDExRA==
3. Create a host group and add the hosts and disks (by using ViodiskID information) that you
want in this host group by running the command that is shown in Example 3-32.
Example 3-32 Creating a host group

ksysmgr add host_group ITSO_HG hosts=rt11-8286-42A-0607585,rt12-8286-42A-2100E5W
repo_disk=01M0lCTTIxNDUxMjQ2MDA1MDc2ODAyODEwNDI2ODgwMDAwMDAwMDAwMDExMA==
ha_disk=01M0lCTTIxNDUxMjQ2MDA1MDc2ODAyODEwNDI2ODgwMDAwMDAwMDAwMDExMQ==
Host_group ITSO_HG added successfully
Run discover command to proceed further
4. Repeat step 1 on page 286 for all host groups that you want to create in the KSYS
subsystem.
5. Verify the host groups that you created by running the command shown in Example 3-33.
Example 3-33 Listing host groups

Name: ITSO_HG
Hosts: rt12-8286-42A-2100E5W
rt11-8286-42A-0607585
high:100
medium:100
low:100
high:100
medium:100
low:100
Skip_power_on: No
HA_monitor: enable
You must enable HA monitoring for the KSYS subsystem to start monitoring the environment.
To check the System-Wide Persistent Attributes, run the ksysmgr query system command, as
Example 3-34 Listing System-Wide Persistent Attributes

# ksysmgr query system
System-Wide Persistent Attributes
notification_level: low
dup_event_processing: yes
ha_monitor: disable
User Scripts for Host Group:

To enable HA monitoring, run the command that is shown in Example 3-35.
Example 3-35 Enabling HA monitoring

# ksysmgr modify system ha_monitor=enable
KSYS ha_monitor has been updated
It is possible to check whether the HA monitor is enabled, as shown in the Example 3-36.
Example 3-36 Listing System-Wide Persistent Attributes after the HA monitor is enabled
ha_monitor: enable
3.5.5 Discovering and verifying the KSYS configuration

After adding various resources (HMCs, hosts, and host groups) to the KSYS subsystem, you
must run the discovery operation. During the initial discovery operation, the KSYS subsystem
creates the required HA setup to monitor the VMs and hosts. The KSYS subsystem creates
an SSP cluster based on the information that is specified in the configuration steps. During
any subsequent discovery operations, the KSYS subsystem scans the environment for any
changes to the environment and adapts to the modified environment.
For example, when you add a host or when you run the LPM operation from one host to
another host that is outside of the current KSYS subsystem, the KSYS configuration settings
are updated in the next discovery operation.
By default, the KSYS subsystem automatically rediscovers sites once every 24 hours at
midnight. You can change this period by modifying the auto_discover_time system attribute.
After the KSYS subsystem discovers the resources, a verification is required to ensure that
the VMs can be restarted on another host without any errors during a failover operation. The
first discovery operation can take a few minutes because the SSP health cluster is deployed
during the first discovery operation.
To discover and verify the configuration for a specific host group, complete the following steps:
1. Discover the resources by running the following command:
ksysmgr discover host_group hg_name
2. Verify the resources by running the following command:
ksysmgr verify host_group hg_name
Important: You must run the discovery and verification commands each time you modify
the resources in the KSYS subsystem
To perform both the discovery and verification operations, run the command that is shown in
Example 3-37.
Example 3-37 Running discovery and verification of the KSYS configuration

# ksysmgr discover host_group ITSO_HG verify=yes
Running discovery on Host_group ITSO_HG, this may take few minutes...
Existing HA trunk adapter found for VIOS rt12v2
SSP creation started for Host_group ITSO_HG
SSP creation completed for Host_group ITSO_HG
Preparing VIOS in rt12-8286-42A-2100E5W for HA management
VIOS in rt12-8286-42A-2100E5W prepared for HA management
Preparing VIOS in rt11-8286-42A-0607585 for HA management
VIOS in rt11-8286-42A-0607585 prepared for HA management

Discovery has finished for ITSO_HG
Host_group verification started for ITSO_HG

ERROR: Verify has encountered an error for VM rt11007
ERROR: Verify has encountered an error for VM rt11006
Verification has finished for ITSO_HG
Unverified VMs:
rt11007
rt11006
Following error is returned for VM rt11007:

Errors:\nHSCL2957 Either there is currently no RMC connection between the
management console and the partition rt11007 (9*8286-42A*0607585) or the partition
does not support dynamic partitioning operations. Verify the network setup on the
management console and the partition and ensure that any firewall authentication
between the management console and the partition has occurred. Run the management
console diagrmc command to identify problems that might be causing no RMC
connection.\n\nDetails:\nHSCL2957 Either there is currently no RMC connection
between the management console and the partition rt11007 (9*8286-42A*0607585) or
the partition does not support dynamic partitioning operations. Verify the network
setup on the management console and the partition and ensure that any firewall
authentication between the management console and the partition has occurred. Run
the management console diagrmc command to identify problems that might be causing
no RMC connection.\n\nJob execution on HMC failed with status
<COMPLETED_WITH_ERROR>. Message: <Errors:\nHSCL2957 Either there is currently no
RMC connection between the management console and the partition rt11007
(9*8286-42A*0607585) or the partition does not support dynamic partitioning
operations. Verify the network setup on the management console and the partition
and ensure that any firewall authentication between the management console and the
partition has occurred. Run the management console diagrmc command to identify
problems that might be causing no RMC connection.\n\nDetails:\nHSCL2957 Either
there is currently no RMC connection between the management console and the
partition rt11007 (9*8286-42A*0607585) or the partition does not support dynamic
partitioning operations. Verify the network setup on the management console and
the partition and ensure that any firewall authentication between the management
console and the partition has occurred. Run the management console diagrmc command
to identify problems that might be causing no RMC connection.\n>.
Following error is returned for VM rt11006:

Errors:\nHSCL2957 Either there is currently no RMC connection between the
management console and the partition rt11006 (6*8286-42A*2100E5W) or the partition
does not support dynamic partitioning operations. Verify the network setup on the
management console and the partition and ensure that any firewall authentication
between the management console and the partition has occurred. Run the management
console diagrmc command to identify problems that might be causing no RMC
connection.\n\nDetails:\nHSCL2957 Either there is currently no RMC connection
between the management console and the partition rt11006 (6*8286-42A*2100E5W) or
the partition does not support dynamic partitioning operations. Verify the network
setup on the management console and the partition and ensure that any firewall
authentication between the management console and the partition has occurred. Run
the management console diagrmc command to identify problems that might be causing
no RMC connection.\n\nJob execution on HMC failed with status
<COMPLETED_WITH_ERROR>. Message: <Errors:\nHSCL2957 Either there is currently no
RMC connection between the management console and the partition rt11006
(6*8286-42A*2100E5W) or the partition does not support dynamic partitioning
operations. Verify the network setup on the management console and the partition

and ensure that any firewall authentication between the management console and the
partition has occurred. Run the management console diagrmc command to identify
problems that might be causing no RMC connection.\n\nDetails:\nHSCL2957 Either
there is currently no RMC connection between the management console and the
partition rt11006 (6*8286-42A*2100E5W) or the partition does not support dynamic
partitioning operations. Verify the network setup on the management console and
the partition and ensure that any firewall authentication between the management
console and the partition has occurred. Run the management console diagrmc command
to identify problems that might be causing no RMC connection.\n>.
Please review the error(s) and take any corrective actions
Example 3-37 on page 290 shows that the VMs rt11006 and rt11007 have no RMC
connection problems.
Configuring VIOS
When you add hosts to the KSYS subsystem, all the VIOSs in the hosts are also added to the
KSYS subsystem. The VM Recovery Manager HA solution monitors the hosts and VMs by
using VIOSs on the host.
The VM Recovery Manager HA solution requires at least two VIOSs per host. You can have a
maximum of 24 VIOSs across different hosts in a single host group. If a host has more than
two VIOSs, you can exclude specific VIOS partitions from the HA management.
To exclude specific VIOS partitions from HA management, run the following command:
ksysmgr unmanage vios viosname
Configuring virtual machines

When a host is added to the KSYS subsystem, all the VMs on the host are included by default
in the HA management. If you do not want HA for any of the VMs, you can exclude specific
VMs from the HA management by running the following command with the appropriate flags:
ksysmgr unmanage vm name=vmname host=hostname | uuid=lparuuid |
ALL host=hostname | ALL host_group=hg_name
In Example 3-38, the VMs rt11006, rt11007, rt11008, rt1009, and rt11010 were unmanaged.
Example 3-38 Unmanaging VMs

# ksysmgr unmanage vm name=rt11006,rt11007,rt11008,rt11009,rt11010
VM rt11006 was successfully unmanaged. Please run discovery to apply changes
To list all VMs and query all managed and unmanaged VMs, run the ksysmgr query vm
command, as shown in Example 3-39.
Example 3-39 Querying VMs

# ksysmgr query vm
Managed VMs:
rt11002
rt12005
rt11005
rt11001
rt12004
rt12003
rt12002
rt12001
rt11004
rt11003
Unmanaged VMs:
rt11009
rt11007
rt11010
rt11008
rt11006
After you change the host group ITSO_HG, perform discovery and verification, as shown in
Example 3-40.
Example 3-40 Performing discovery and verification after unmanaging VMs

.
.
.
Discovery has finished for ITSO_HG
.
.
.
Enabling HA monitoring at the VM and application level

At this moment, you have HA monitoring only at the host level. To enable HA monitoring at the
VM and application level, you must install the VM Agent for HA monitoring, which is described
in 3.4.3, “Installing VM Agents” on page 279.
After installing the VM Agent for HA monitoring, you start the daemon agent by running the
ksysvmmgr start command in each VM.
Example 3-41 on page 295 shows how to start VM monitor (VMM) on AIX VMs.

Example 3-41 Starting VMM on AIX VMs
# ksysvmmgr status
(0) root @ rt11001: /

# ksysvmmgr start
(0) root @ rt11001: /

# ksysvmmgr status
Example 3-42 shows how to start VMM on Red Hat VMs.
Example 3-42 Starting VMM on Red Hat VMs

# ksysvmmgr status
ksys_vmm daemon is currently inoperative.

# ksysvmmgr start

# ksysvmmgr status
ksys_vmm daemon is active.
Example 3-43 shows how to start VMM on SUSE VMs.
Example 3-43 Starting VMM on SUSE VMs

# ksysvmmgr status

# ksysvmmgr start

# ksysvmmgr status
Now that VMM is enabled on the VMs, you must enable HA monitoring on KSYS on the VM
level for each VM by running the following command:
ksysmgr modify vm vm1[,vm2,...] ha_monitor=enable
In Example 3-44, the VMs rt11001, rt11002, rt11003, rt11004, rt11005, rt12001, rt12002,
rt12003, rt12004, and rt12005 were changed to enable the ha_monitor.
Example 3-44 Enabling the HA VM monitoring level on KSYS

# ksysmgr modify vm
rt11001,rt11002,rt11003,rt11004,rt11005,rt12001,rt12002,rt12003,rt12004,rt12005
ha_monitor=enable
In Example 3-45, discovery and verification run after VMM is enabled.
Example 3-45 Performing discovery and verification after enabling VMM

.
.
.
Creating first HA client adapter for VM rt12001
Finished creating first HA client adapter for VM rt12001
Creating second HA trunk client for VM rt12001
Finished creating second HA client adapter for VM rt12001
.
.
.
Starting HA monitoring for VM rt12001
HA monitoring for VM rt12001 started successfully
.
.
.
VM monitor state has moved to 'STARTED' for VM rt11002

.
.
.
3.6 Setting up HA policies

After you set up the KSYS subsystem successfully, set up recovery policies to customize the
default configuration settings to suit your HA preferences.
This session describes the VM Recovery Manager HA solution options that you can
customize.
Note: You must run the discovery and verification command after you set any policy.
3.6.1 HA monitoring policies

An HA monitoring policy turns on or off HA monitoring for the associated entity. The specified
policy at the lowest resource level is considered first for HA monitoring. If you do not specify
this policy for a resource, the policy of the parent resource is applied to the resource. For
example, if you enable HA monitoring for the host group, HA monitoring is enabled for all the
hosts within the host group unless you disable HA monitoring for specific hosts.
You can enable HA monitoring for VMs only after you install the VM Agent on each VM and
start the VM Agent successfully. If you do not set up the VM Agent, the KSYS subsystem
might return error messages for HA monitoring at the VM level.
To set the HA monitoring, run the following command:

ksysmgr modify system|host_group|host|vm name ha_monitor=enable|disable
Example 3-46 shows the ksysmgr query system command that checks the HA monitoring for
the KSYS.
Example 3-46 Checking HA monitoring for the KSYS

ha_monitor: enable
User Scripts for Host Group:
To check HA monitoring for the host group, run the ksysmgr query host_group command that
is shown in Example 3-47.
Example 3-47 Checking HA monitoring for the host group

Name: ITSO_HG
Hosts: rt12-8286-42A-2100E5W
rt11-8286-42A-0607585
high:100
medium:100
low:100
high:100
medium:100
low:100
Skip_power_on: No
HA_monitor: enable
To check the HA monitoring for the VM rt11001, run the ksysmgr query vm command that is
Example 3-48 Checking the HA monitoring for the VM rt11001

# ksysmgr query vm rt11001
Name: rt11001
UUID: 3BF44186-0679-4032-BF4C-95B24901822A
Host: rt12-8286-42A-2100E5W
Priority: Medium
HA_monitor: enable
Homehost: rt12-8286-42A-2100E5W
3.6.2 Environment-level policies

This section describes the environment-level policies.
Restart policy
The restart policy notifies the KSYS subsystem to restart the VMs automatically during a
failure. This attribute can have the following values:
auto If you set this attribute to auto, the KSYS subsystem automatically
restarts the VMs on the destination hosts. The KSYS subsystem
identifies the most suitable host based on free CPUs, memory, and
other specified policies. In this case, the KSYS subsystem also notifies
the registered contacts about the host or VM failure and the restart
operations. This is the default value of the restart_policy attribute.

advisory_mode If you set this attribute to advisory_mode, the VMs are not restarted
automatically after host or VM failures. In this case, the KSYS
subsystem notifies the registered contacts about the host or VM
failures. The administrator must review the failure and manually restart
the VMs on other hosts by using the ksysmgr commands.
To set the restart policy, run the following command syntax:

ksysmgr modify host_group name restart_policy=auto|advisory_mode
For example, to check the restart policy of the host group ITSO_HG, we run the command
ksysmgr query host_group, as shown in Example 3-49.
Example 3-49 Checking the restart_policy of the host group ITSO_HG

Name: ITSO_HG
Hosts: rt12-8286-42A-2100E5W
rt11-8286-42A-0607585
high:100
medium:100
low:100
high:100
medium:100
low:100
Skip_power_on: No
HA_monitor: enable
Host failure detection time

The host failure detection time policy indicates the time that the KSYS waits on a
non-responsive host before the KSYS declares the host to be in an inactive state. This value
is measured in seconds. The KSYS subsystem uses the specified time to ensure the health of
the host and attempts to connect to the host before the KSYS declares the failure. After this
duration, the VMs are restarted on another host that is located within the host group. The
value of this attribute can be 90 – 600 seconds. The default value is 90 seconds.
To set the host failure detection time, run the following command:
ksysmgr modify system|host_group name host_failure_detection_time=time_in_seconds
To check the KSYS host failure detection time, we run the command ksysmgr query system,
Example 3-50 Checking the KSYS host failure detection time

ha_monitor: enable
To check the host group failure detection time, we use the ksysmgr query host_group
command, as shown in Example 3-51.
Example 3-51 Checking the host group host failure detection time
Name: ITSO_HG
Hosts: rt12-8286-42A-2100E5W
rt11-8286-42A-0607585
high:100
medium:100
low:100
high:100
medium:100
low:100
Skip_power_on: No
HA_monitor: enable
Flexible capacity policy

The flexible capacity policy modifies the allocation of memory and CPU resources of a VM
when a VM is moved from its home host to another host on the host group. You can set
flexible capacity values based on the priority of a VM. You can set different flexible capacity
values for various priorities of VMs: high, medium, and low. You must specify the flexible
capacity values in percentage.
For example, you can define the following flexible capacity values at the host group level:
100% CPU and 100% memory for high priority VMs; 70% CPU and 80% memory for medium
priority VMs; and 60% CPU and 75% memory for low priority VMs. When a medium priority
VM is migrated from its home host to another host on the host group, its capacity is adjusted
to 70% CPU and 80% memory. If the VM is restored back to its home host, the VM is restored
with 100% resources.
The flexible capacity policy does not consider I/O slots, adapters, and resources that are
available in the hosts. You must ensure that all the I/O virtualization requirements of the VMs
are met within the host group environment. Also, the flexible capacity policy is applicable only
to VM relocation that is based on restart operations. LPM operations do not follow the flexible
capacity policy.
To set the flexible capacity policy, run the following command:

ksysmgr modify host_group
[memory_capacity=(1-100) | minimum | current_desired | none]
[priority=low|medium|high]
[cpu_capacity=(1-100) | minimum | current_desired | none]
[priority=low|medium|high]
Example 3-52 on page 301 shows the default settings of the flexible capacity policy for the
host group ITSO_HG.

Example 3-52 Listing the flexible capacity policies of the host group ITSO_HG
Name: ITSO_HG
Hosts: rt12-8286-42A-2100E5W
rt11-8286-42A-0607585
high:100
medium:100
low:100
high:100
medium:100
low:100
In Example 3-53, the flexible capacity policies were set as follows:

򐂰 Memory capacity to 60% for medium-priority VMs
򐂰 CPU capacity to 70% for medium-priority VMs
򐂰 Memory capacity to 50% for low-priority VMs
򐂰 CPU capacity to 50% for low-priority VMs
Example 3-53 Changing the flexible capacity policies

# ksysmgr modify host_group ITSO_HG options memory_capacity=60 priority=medium
For Host_group ITSO_HG attribute(s) 'memory_capacity', 'priority' was successfully
modified
# ksysmgr modify host_group ITSO_HG options cpu_capacity=70 priority=medium

For Host_group ITSO_HG attribute(s) 'cpu_capacity', 'priority' was successfully
modified
# ksysmgr modify host_group ITSO_HG options memory_capacity=50 priority=low

For Host_group ITSO_HG attribute(s) 'memory_capacity', 'priority' was successfully
modified.
# ksysmgr modify host_group ITSO_HG options cpu_capacity=50 priority=low

For Host_group ITSO_HG attribute(s) 'cpu_capacity', 'priority' was successfully
modified
Example 3-54 shows the host group ITSO_HG after changing its flexible capacity policies.
Example 3-54 Checking the flexible capacity policies after changing them for ITSO_HG
Name: ITSO_HG
Hosts: rt12-8286-42A-2100E5W
rt11-8286-42A-0607585
high:100
medium:60
low:50
high:100
medium:70
low:50
Affinity policies
An affinity policy specifies affinity rules for a set of VMs that defines how the VMs must be
placed within a host group during a relocation. The following affinity policies are supported if
all VMs in an affinity group have the same priority.
Collocation
A collocation policy indicates that the set of VMs must always be placed on the same host
after relocation, as shown in Figure 3-5.
Figure 3-5 Collocation policy
To set this policy, run the following commands:

ksysmgr add collocation name vm=vmname1[,...]>
ksysmgr modify collocation name policy=add|delete vm=vm1[,...]
Example 3-55 shows the collocation policy policy_1 for the VMs rt12001, r12002, and 12003.
Example 3-55 Creating a collocation policy

# ksysmgr add collocation policy_1 vm=rt12001,rt12002,rt12003
Collocation group policy_1 was created
To list the collocation policies that are available, run the command ksysmgr query
collocation, as shown in Example 3-56.
Example 3-56 Listing the collocation policies

# ksysmgr query collocation
Name: policy_1
Type: Collocation
VMs: rt12003
rt12002
rt12001
Example 3-57 shows how to delete a collocation policy.
Example 3-57 Deleting a collocation policy

# ksysmgr delete collocation policy_1
Collocation group policy_1 was removed

Anticollocation
An anticollocation policy indicates that the set of VMs must never be placed on the same
host after relocation, as shown in Figure 3-6.
Figure 3-6 Anticollocation policy
To set this policy, run the following commands:

ksysmgr add collocation name vm=vmname1[,...]>
ksysmgr modify collocation name policy=add|delete vm=vm1[,...]
Example 3-58 shows the anticollocation policy policy_2 with the VMs rt12001 and rt12002.
Example 3-58 Creating an anticollocation policy

# ksysmgr add anticollocation policy_2 vm=rt12001,rt11001
Anticollocation group policy_2 was created
To list the anticollocation policies that are available, run the command ksysmgr query
anticollocation, as shown in Example 3-59.
Example 3-59 Listing the anticollocation policies

# ksysmgr query anticollocation
Name: policy_2
Type: anticollocation
VMs: rt11001
rt12001
Example 3-60 shows how to delete an anticollocation policy.
Example 3-60 Deleting an anticollocation policy

# ksysmgr delete anticollocation policy_2
Anticollocation group policy_2 was removed
Workgroup
A workgroup policy indicates that the set of VMs must be prioritized based on the assigned
priority.
To set this option, run the following commands:

ksysmgr add workgroup name vm=vmname1[,...]
ksysmgr modify workgroup name policy=add|delete vm=vm1[,...]
Example 3-61shows the creation of a workgroup policy with the VMs rt11001, rt11002, and
rt11003.
Example 3-61 Creating a workgroup policy

# ksysmgr add workgroup dev_itso vm=rt11001,rt11002,rt11003
Workgroup group dev_itso was created
To list the workgroup policies that are available, run the ksysmgr query workgroup command,
Example 3-62 Listing the workgroup policies

# ksysmgr query workgroup
Name: dev_itso
Type: workgroup
VMs: rt11002
rt11001
rt11003
Example 3-63 shows how to delete the workgroup policy.
Example 3-63 Deleting the workgroup policy

# ksysmgr delete workgroup dev_itso
Workgroup group dev_itso was removed
Note: When you set affinity policies, ensure that the host group has sufficient capacity for
the policies to be implemented during host failure, VM failure, or application failure. For
example, if the host group contains only two hosts, you cannot set an anticollocation policy
on VMs of a specific host because the host group does not contain multiple target hosts to
restart the VMs.

3.6.3 VM-level policies
This section describes the VM-level policies for relocating VMs during failover operations.
Host blacklist
The host blacklist policy specifies the list of hosts that must not be used for relocating a
specific VM during a failover operation. For a VM, you can add hosts within the host group to
the blacklist based on performance and licensing preferences, as shown in the Figure 3-7.
Figure 3-7 Host blacklist policy
To set this option, run the following command:

ksysmgr modify vm vmname
blacklist_hosts=hostname[,...] policy=add|delete
Example 3-64 shows that VM rt11004 do not have the blacklist policy.
Example 3-64 Checking the blacklist policy

Name: rt11004
UUID: 4FCD724B-8407-4945-A973-C25C4CA69A94
Host: rt11-8286-42A-0607585
Priority: Medium
HA_monitor: enable
Homehost: rt11-8286-42A-0607585
Example 3-65 shows that host rt12-8286-42A-2100E5W in VM rt1004 was set as the host
blacklist.
Example 3-65 Setting the blacklist policy on VM rt11004

# ksysmgr modify vm rt11004 blacklist_hosts=rt12-8286-42A-2100E5W
WARNING: No backup host will be left for this VM rt11004.
For VM rt11004 attribute(s) 'blacklist_hosts' was successfully modified.
Note: The warning appeared only because this test environment is composed only of two
hosts on the host group.
To list the blacklist host that is configured on VM rt11004, run the command ksysmgr query
vm rt11004, as shown in Example 3-66.
Example 3-66 Checking the blacklist policy on VM rt11004

Name: rt11004
UUID: 4FCD724B-8407-4945-A973-C25C4CA69A94
State: INIT
Host: rt11-8286-42A-0607585
Priority: Medium
HA_monitor: enable
Homehost: rt11-8286-42A-0607585
Blacklist_hosts: rt12-8286-42A-2100E5W
Example 3-67 shows how to delete the blacklist policy from VM rt11004.
Example 3-67 Deleting the blacklist host from VM rt11004

# ksysmgr modify vm rt11004 blacklist_hosts=rt12-8286-42A-2100E5W policy=delete
For VM rt11004 attribute(s) 'blacklist_hosts', 'policy' was successfully modified.
Failover priority
The failover priority policy specifies the order of processing multiple VMs restart operations.
For example, if a host fails and all the VMs must be relocated to other hosts on the host group,
the priority of the VM determines which VM is processed first. The supported values for this
attribute are High, Medium, or Low. You can set this attribute at the VM level only. You must
specify the UUID of the VM if you have two or more VMs with the same name. By default, all
VMs on the host group have a priority of Medium.
To set the failover priority, run the following command:

ksysmgr modify vm name1[,name2,...] | filepath=filepath priority=high|medium|low
In Example 3-68, the VM rt11001 is set to High priority.
Example 3-68 Setting the VM rt11001 to High priority

# ksysmgr modify vm rt11001 priority=High
For VM rt11001 attribute(s) 'priority' was successfully modified.

To check the priority of the VM rt11001, run the command ksysmgr query vm rt11001, as
shown n Example 3-69.
Example 3-69 Checking the priority policy of the VM rt11001

Name: rt11001
UUID: 3BF44186-0679-4032-BF4C-95B24901822A
Host: rt11-8286-42A-0607585
Priority: High
HA_monitor: enable
Example 3-70 shows that all VMs on the host rt12-8286-42A-2100E5W were set to High
priority.
Example 3-70 Setting all VMs on the host rt12-8286-42A-2100E5W to High priority
# ksysmgr modify vm ALL host=rt12-8286-42A-2100E5W priority=High
For VM rt12005 attribute(s) 'host', 'priority' was successfully modified.
Home host
The home host policy specifies the home host of the VM. By default, the KSYS subsystem
sets this value initially to the host where the VM was first discovered. You can change the
home host value of a VM even when the VM is running on another host. In such case, the
specified home host is used for all future operations. This attribute is useful when you get a
host repaired after failure and you want to restart the VMs in its home host.
To set the home host value, run the following command:

ksysmgr modify vm name1[,name2...] homehost=hostname
To check that the VM rt11001 is with the home host rt12-8286-42A-2100E5W, run the ksysmgr
query vm command, as shown in Example 3-71.
Example 3-71 Checking the home host of the VM rt11001

Name: rt11001
UUID: 3BF44186-0679-4032-BF4C-95B24901822A
Host: rt11-8286-42A-0607585
Priority: High
HA_monitor: enable
In Example 3-72, there is a request to change the home host of the VM rt11001 to
rt11-8286-42A-0607585.
Example 3-72 Changing the home host of the VM rt11001

# ksysmgr modify vm rt11001 homehost=rt11-8286-42A-0607585
In Example 3-73, you check that VM rt11001 has the home host rt11-8286-42A-0607585.
Example 3-73 Checking the home host of the VM 11001

Name: rt11001
UUID: 3BF44186-0679-4032-BF4C-95B24901822A
Host: rt11-8286-42A-0607585
Priority: High
HA_monitor: enable
Homehost: rt11-8286-42A-0607585
Relocation mapper flow diagram

Figure 3-8 on page 309 shows the relocation mapper flow diagram for the policies for VMs in
the IBM Recovery Manager HA for IBM Power Systems.
The steps in the diagram are as follows:

1. Create an empty map table.
2. Check the VM level priority: High, Medium, or Low.
3. Check the collocation priority.
4. Check the anticollocation priority.
5. Check for Best Fit: free CPU and memory.
6. Check the Priority List: Flex capacity.
7. Check for auto mode: Yes starts a recovery restart, and No sends a notification.

Figure 3-8 Relocation Mapper Flow
3.7 Setting up the VM Agent

If you configure the HA function at the VM level or application level, you must set up the VMM
and the application monitoring framework (AppMon) in each VM for which VM failure
detection is required.
You can install the VM Agent to monitor the VM and applications on the VMs that run only the
following operating systems:
򐂰 AIX Version 6.1 or later
򐂰 PowerLinux:
– RHEL (Little Endian) Version 7.4 or later
– SUSE Linux Enterprise Server (Little Endian) Version 12.3 or later
Currently, the HA feature at the VM level or application level is not supported for the IBM i
operating system or other Linux distributions. However, you can enable these VMs for
host-level health management. In addition, you can perform manual restart and LPM
operations on these VMs.
Note: You can configure the VM Agent only by using the ksysvmmgr command. You cannot
configure the VM Agent by using the VM Recovery Manager HA GUI.
3.7.1 The ksysvmmgr command and utility

The ksysvmmgr command (Figure 3-9) provides a consistent interface to manage the VMM
and Applications. The ksysvmmgr command interacts with the VMM daemon over a UNIX
socket.
Figure 3-9 The ksysvmmgr command
Example 3-74 shows the ksysvmmgr command syntax.
Example 3-74 The ksysvmmgr command syntax

# ksysvmmgr
Usage: ksysvmmgr [<FLAGS>] <ACTION> [<CLASS>] [<NAME>] [<ATTRIBUTES>]
KSYSVMMGR-002(ERROR): <ACTION> parameter is mandatory.
For <ACTION> parameter, possible actions are :
"add,query,modify,delete,status,resume,suspend,start,stop,help,start,stop,status,s
ync".
Retry using one of the possible actions.
Or refer to complete output of "ksysvmmgr -h"
After you install the VM Agent filesets successfully, complete the procedures in the following
sections to set up the VMM in each guest VM.

VMM daemon
The VMM daemon must be started in each VM in which that agent is installed. To start
monitoring the VMs and applications, run the command that is shown in Example 3-75.
Example 3-75 Starting the VMM daemon

# ksysvmmgr start
# ksysvmmgr status
Virtual machine monitor

The ksysvmmgr command configures the following classes and attributes, as shown in
Example 3-76.
Example 3-76 Listing the VMM details

# ksysvmmgr query vmm
VmMonitor
log=2
period=1
version=1.0
comment=2018-10-26 16:02:11
rebootappstarttype=VMM
VmUUID=3bf44186-0679-4032-bf4c-95b24901822
When you start the VMM, the VMM daemon sends heartbeats to the HM when requested by
the HM so that the KSYS subsystem can monitor the VMs.
The VMM can have the following attributes:

version Specifies the version of XML. This mandatory attribute is set to 1.0 for
the current version of VMM and cannot be modified.
log Specifies the log level of the VMM daemon. This attribute can have the
following values:
0 Only errors are logged. It is the default value.
1 Warnings are also logged.
2 Informational messages are also logged.
3 Details of the operation are logged. This information is used for
debugging.
period Specifies the time duration in seconds between two consecutive
occurrences of checks that are performed by the Application
Management Engine (AME). By default, the value of this attribute is
1 second. The value of this attribute must be in the range 0 - 6. For
best monitoring performance, do not modify the default value.
rebootappstarttype Specifies the method in which the applications must be started when
the VM is restarted on a new host. This attribute can have the following
values:
VMM Directs the VM Agent to start the applications immediately after the
VM is restarted on the new host.
OS Specifies that the applications must be started by the operating
system or by a user after the VM restarts on a new host. If the
application is not started by the operation system or a user, the VM
Agent starts the applications.
To modify the VMM parameters, run the following commands:

򐂰 # ksysvmmgr modify log=3
򐂰 # ksysvmmgr modify period=2
򐂰 # ksysvmmgr modify rebootappstarttype=OS
3.7.2 Application Management Engine

The AME validates the application monitoring configuration, starts and stops applications,
monitors application health, restarts applications in case of failure, and informs the
Application Reporting (AR) thread about the application configuration and status.
To register applications in the VMM daemon so that they are monitored for HA, run the
command that is shown in Example 3-77.
Example 3-77 Adding an application to be monitored by AME

# ksysvmmgr -s add app app1 monitor_script=/applic/mon1.sh
start_script=/applic/start_app1.sh stop_script=/applic/stop_app1.sh
To check the attributes of all applications, run the command ksysvmmgr query app, as shown
in Example 3-78.
Example 3-78 Checking the attributes of all applications

# ksysvmmgr query app
Application name=app1
version=
uuid=1542231184972777000
desc=
monitored=1
monitor_script=/applic/mon1.sh
monitor_period=30
monitor_timeout=15
monitor_failure_threshold=5
stop_script=/applic/stop_app1.sh
stop_stabilization_time=25
stop_max_failures=3
start_script=/applic/start_app1.sh
start_stabilization_time=25
start_max_failures=3
max_restart=3
nooftimes_restarted=0
critical=no
type=
instancename=
database=
vendor_id=

After you add an application, if the application fails or stops working correctly, the VM Agent
attempts to restart the application in the same VM for the number of times that are specified in
the max_restart attribute for the VM, which is set to 3 by default. If the application is still not
working correctly, the KSYS subsystem notifies you about the issue. You can manually review
the problem and restart the application.
Mark important applications as critical by running the command that is shown in

Example 3-79.
Example 3-79 Marking important applications as critical

# ksysvmmgr -s modify app app1 critical=yes
Modifying application "app1" into configuration successfully performed.
When you mark an application as critical, if the application fails or stops working correctly, the
VM Agent attempts to restart the application for the number of times that are specified in the
max_restart attribute for the VM. If the application is still not working correctly, the KSYS
subsystem notifies you about the issue and attempts to restart the VM on the same host. If
the application is still not working correctly, that is, if the application status is displayed as RED
when you run the ksysmgr query app command, the KSYS restarts the VM on another host
within the host group based on the specified policies.
Application class
The application class contains the following mandatory attributes:
򐂰 monitor_script: A mandatory script that is used by the VM Agent to verify application
health. This script is run regularly (based on the monitor_period attribute value) and the
result is checked for the following values:
– 0: Application is working correctly.
– Any value other than 0: Application is not working correctly or failed.
After several successive failures (based on the monitor_failure_threshold attribute
value), the application is declared as failed. Based on the specified policies, the KSYS
subsystem determines whether to restart the VM.
򐂰 stop_script: A mandatory script that is used by the VM Agent to stop the application if the
application must be restarted.
򐂰 start_script: A mandatory script that is used by the VM Agent to start the application if
the application must be restarted.
The application class contains the following optional attributes:

򐂰 monitored: Specifies whether the application is monitored by the KSYS subsystem. This
attribute can have the following values:
– 1 (default): The application monitoring is active.
– 0: The application monitoring is suspended.
򐂰 monitor_period: Specifies the time in seconds after which the application monitoring must
occur. The default value of 30 seconds specifies that the monitor_script script is run by
the VM Agent every 30 seconds.
򐂰 monitor_timeout: Specifies the waiting time in seconds to receive a response from the
monitor_script script. The default value is 10 seconds, which means that the VMM waits
for 10 seconds to receive a response from the monitor_script script, after which the script
is considered as failed.
򐂰 monitor_failure_threshold: Specifies the number of successive failures of the
monitor_script script that is necessary before the VMM restarts the application. A restart
operation is performed by successively calling the stop_script and start_script scripts.
򐂰 stop_stabilization_time: Specifies the waiting time in seconds to receive a response
from the stop_script script. The default value is 25 seconds, which means that the VMM
waits for 25 seconds to receive a response from the stop_script script, after which the
script is considered as failed.
򐂰 stop_max_failures: Specifies the number of successive failures of the stop_script script
that is necessary before the VMM considers that it cannot stop the application. The default
value is set to 3.
򐂰 start_stabilization_time: Specifies the waiting time in seconds to receive a response
from the start_script script. The default value is 25 seconds, which means the VMM
waits for 25 seconds to receive a response from the start_script script, after which the
script is considered as failed.
򐂰 start_max_failures: Specifies the number of successive failures of the start_script
script that is necessary before the VMM considers that it cannot start the application. The
default value is set to 3.
򐂰 max_restart: Specifies the number of cycles of successive VM restart operations that
result in a monitoring failure before the daemon pronounces that restarting at VM level is
insufficient. By default, this attribute is set to 3.
򐂰 status: Specifies the dynamic status of application that is returned by AME. This attribute
cannot be modified.
򐂰 version: Specifies the application version. This attribute does not have a default value.
򐂰 critical: Marks the application as critical. The valid values are Yes and No (default). If you
mark an application as critical, the failure of the application is sent to the KSYS subsystem
for further action.
򐂰 type: Specifies the type of application. By default, the type attribute does not have any
value that indicates general applications. Other supported values are ORACLE, IBM DB2, and
SAPHANA. This attribute is case-sensitive and you must use uppercase characters. For
these types of applications, if you do not specify start, stop, and monitor scripts, internal
scripts of the VMM are used.
򐂰 instancename: Specifies the instance name for applications. This attribute is applicable
only for applications whose scripts need an instance name as an argument. For example:
– If the application type is ORACLE, the instancename attribute must be specified with the
Oracle instance name.
– If the application type is DB2, the instancename attribute must be specified with the
IBM DB2® instance owner.
– If the application type is SAPHANA, the instancename attribute must be specified with the
SAP HANA instance name.
򐂰 database: Specifies the database that the applications must use. This attribute is
applicable only for applications whose scripts require database as an argument. For
example:
– If the application type is ORACLE, the database attribute must be specified with the
Oracle system identifier (SID).
– If the application type is DB2, the database attribute is not required.
– If the application type is SAPHANA, the database attribute must be specified with the SAP
HANA database.

VM Recovery Manager HA in-built scripts
Apart from any general application that you want the VM Agent to monitor, the VM Recovery
Manager HA solution supports the type of applications that can be monitored by using the
in-built scripts in the corresponding versions of operating systems, as shown in the Table 3-2.
Table 3-2 Version support matrix for application types and operating systems
Application type AIX operating system Linux operating system
(RHEL and SUSE Linux
Enterprise Server)
Oracle Oracle Database 12.1 or later Not supported
DB2 IBM DB2 11.3 or later IBM DB2 10.5 or later
SAPHANA Not supported SAP HANA 2.0 or later
Specific parameters for built-in supported application agents are shown in Table 3-3.
Table 3-3 Application agent-specific parameters

Attribute Oracle DB2 SAP HANA
type Oracle DB2 SAPHANA
version Taken from application Taken from application Taken from application
database Oracle SID DB2 database SAP HANA database
start_script /usr/sbin/agents/ora /usr/sbin/agents/db2 /usr/sbin/agents/sap

cle/startoracle /stopdb2 /startsaphana
stop_script /usr/sbin/agents/ora /usr/sbin/agents/db2 /usr/sbin/agents/sap

clestartoracle /stoporacle /stopsaphan
Application dependencies
If some applications have dependencies, for example, if you must specify a sequence of
applications to start or stop, specify the dependencies as shown in Figure 3-10.
Figure 3-10 Application dependencies
The start_sequence application dependency

This policy defines the order in which the applications start.
The stop_sequence application dependency

This policy defines the order in which the applications must be stopped.
In Example 3-80, the dependencies type start_sequence and stop_sequence were added to
applications app1, app2, and app3.
Example 3-80 Adding start_sequence dependencies

# ksysvmmgr -s add dependency dependency_list=app1,app2,app3
dependency_type=start_sequence
Adding dependency "dependency_list=app1,app2,app3" into configuration successfully
performed.
# ksysvmmgr -s add dependency dependency_list=app1,app2,app3

dependency_type=stop_sequence
Adding dependency "dependency_list=app1,app2,app3" into configuration successfully
performed.
Example 3-81 uses the command ksysvmmgr q dependency to list the dependencies.
Example 3-81 Listing dependencies after the start_sequence and stop_sequence dependencies
# ksysvmmgr q dependency
Dependency depuuid=1542239347575681000
dependency_list=app1,app2,app3
strict=YES

strict=YES
The parent_child application dependency

This policy defines the parent_child dependency between the applications.
Only one level of parent_child dependency is allowed:

򐂰 In this policy, if App1 is the parent and App2 is the child, App1 starts first.
򐂰 If App1 cannot start, App2 does not start.
򐂰 If App1 and App2 start in order and App1 stops because of any reason, App2 stops. App1
must be restarted, and then App2 is restarted.
In Example 3-82, the dependency parent_child was added between the applications app4
and app5.
Example 3-82 Adding parent_child dependence

# ksysvmmgr -s add dependency dependency_list=app4,app5
dependency_type=parent_child
Adding dependency "dependency_list=app4,app5" into configuration successfully
performed.
To list the dependencies after adding the parent_child dependency, run the command
ksysvmmgr q dependency, as shown in Example 3-83.
Example 3-83 Listing dependencies after adding the parent_child dependency

# ksysvmmgr q dependency
strict=YES
strict=YES
dependency_list=app4,app5
strict=YES
# ksysvmmgr q dependency 1542239846084192000

dependency_list=app4,app5
strict=YES
Application dependency class
The application dependency class contains the following mandatory attributes:
򐂰 dependency_type: Specifies the type of dependency between applications. This attribute
can have the following values:
– start_sequence: Specifies the order in which the applications must be started as
mentioned in the dependency_list attribute. The dependency_list attribute must have
more than one application for this dependency type.
– stop_sequence: Specifies the order in which the applications must be stopped as
mentioned in the dependency_list attribute. The dependency_list attribute must have
more than one application for this dependency type.
– parent_child: Specifies the parent-child relationship of the two specified applications
in which one application is the parent and the other is the child. The parent application
must start first and then the child application must start. You must stop the child
application first and then stop the parent application. If the parent application fails, the
child application stops automatically.
򐂰 dependency_list: Specifies the list of applications that have a dependency among them.
The dependency class also contains the following optional attributes:
– strict: Specifies whether to continue the script or command if the dependency policy
cannot be followed. If the strict attribute is set to Yes, the next application is not started
until the previous application starts and is in the normal state. If the strict attribute is set
to No, the next application is started immediately after the first application is started
regardless of the state of the first application. This attribute is applicable only for the
start_sequence dependency.
3.8 Setting contacts for event notification

The KSYS subsystem tracks various events that occur in the environment, analyzes the
situation, and notifies the registered contacts about any issues. You must add the contact
information to the KSYS subsystem so that you can receive notifications about any situation
that might need your action.
You can add the following contact information for a specific user:
򐂰 Email address
򐂰 Phone number with the phone carrier email address
򐂰 Pager email address
Note: The KSYS node must have a public IP address to send the event notifications
successfully.
To register contact details so that you can receive notification from KSYS, run the following
commands in the KSYS LPAR:
򐂰 To add the email address of a specific user to receive notifications, run the following
command:
ksysmgr add notify user=username contact=email_address

Example 3-84 shows how to add and list an email address for a specific user.
Example 3-84 Adding and listing an email address for a specific user
# ksysmgr add notify user=Francisco contact=francisco.almeida@testamail.com
successfully added user info
# ksysmgr query notify contact

User: Francisco
Contact: francisco.almeida@testamail.com
You can add multiple email addresses for a specific user. However, you cannot add
multiple email addresses simultaneously. You must run the command multiple times to add
multiple email addresses, as shown in Example 3-85.
Example 3-85 Adding multiple email addresses

# ksysmgr add notify user=Francisco contact=francisco.almeida@mailtes.com

User: Francisco
Contact: francisco.almeida@testamail.com
User: Francisco
Contact: francisco.almeida@mailtes.com
򐂰 To add a specific user to receive a System Management Services (SMS) notification, run
the following command:
ksysmgr add notify user=username
contact=10_digit_phone_number@phone_carrier_email_address
Example 3-86 shows how to add a specific user to receive an SMS notification.
Example 3-86 Adding a specific user to receive an SMS notification
# ksysmgr add notify user=Beatriz contact=1234567890@tomymail.com


User: Beatriz
Contact: 1234567890@tomymail.com
You must specify the phone number along with the email address of the phone carrier to
receive an SMS notification. To determine the email address of your phone carrier, contact
the phone service provider.
򐂰 To add a specific user to receive a pager notification, run the following command:
ksysmgr add notify user=username contact=pager_email_address
Example 3-87 shows how to add a specific user to receive a pager notification.
Example 3-87 Adding a specific user to receive a pager notification

# ksysmgr add notify user=Dayana contact=1234567890@skytel.com
User: Beatriz
User: Dayana
Contact: 1234567890@skytel.com
򐂰 To remove a specific user, run the following command:

ksysmgr delete notify user=username
Example 3-88 shows how to remove a specific user notification.
Example 3-88 Removing a user notification

#ksysmgr query notify contact
User: Beatriz
# ksysmgr delete notify user=Beatriz

successfully deleted user info
#ksysmgr query notify contact
3.9 Uninstalling VM Recovery Manager HA

To uninstall the VM Recovery Manager HA solution, you can use the command-line interface
(CLI) to uninstall all the installed VM Recovery Manager HA filesets by running the installp
-u command.
Complete the following procedures to uninstall the VM Recovery Manager HA solution:

1. Uninstalling the KSYS software.
2. Uninstalling VM Agents.
Note: You must have root authority to perform any uninstallation tasks.
3.9.1 Uninstalling the KSYS software

If for some reason you must uninstall the KSYS filesets from the KSYS node, remove the
cluster before uninstalling the filesets. Otherwise, the RSCT Peer Domain remains in the
node. Example 3-89 shows the command that is used to perform the cluster removal.
Example 3-89 Removing the KSYS cluster before removing the KSYS filesets
# ksysmgr remove ksyscluster ITSO_HA
WARNING: This action will remove all configuration and destroy the KSYS setup, its
recommended to create a backup "ksysmgr add snapshot -h"
Do you want to a backup to be created now ? [y|n]
y
Created: /var/ksys/snapshots/oldclust_BASIC_2018-11-27_15:23:46.xml.tar.gz
Successfully created a configuration snapshot:
/var/ksys/snapshots/oldclust_BASIC_2018-11-27_15:23:46.xml.tar.gz
y

This may take a few minutes to safely remove the ksyscluster
Virtual client adapters have been deleted
Host_groups have been deleted
Trunk adapters have been deleted
IBM.VMR process stopped successfully
Peer domain stopped successfully
Peer domain was removed successfully
Tip: If you reinstall the filesets and re-create the environment later, consider creating a
snapshot before removing the cluster so that you can restore the snapshot after reinstalling
the filesets.
Example 3-90 shows how to uninstall the KSYS filesets.
Example 3-90 Unistalling the KYS filesets

# installp -ug ksys.*
.
.
.
--------------------
-------------------------------------------------------------------------------
ksys.ha.license 1.3.0.0 USR DEINSTALL SUCCESS
ksys.hautils.rte 1.3.0.0 ROOT DEINSTALL SUCCESS
ksys.hautils.rte 1.3.0.0 USR DEINSTALL SUCCESS
ksys.main.msg.en_US.cmds 1.3.0.0 USR DEINSTALL SUCCESS
ksys.ui.agent 1.3.0.0 ROOT DEINSTALL SUCCESS
ksys.ui.agent 1.3.0.0 USR DEINSTALL SUCCESS
ksys.ui.common 1.3.0.0 USR DEINSTALL SUCCESS
ksys.main.cmds 1.3.0.0 ROOT DEINSTALL SUCCESS
ksys.main.cmds 1.3.0.0 USR DEINSTALL SUCCESS
ksys.main.rte 1.3.0.0 ROOT DEINSTALL SUCCESS
ksys.main.rte 1.3.0.0 USR DEINSTALL SUCCESS
3.9.2 Uninstalling VM Agents

This section shows how to uninstall the VM Agents.
Uninstalling a VM Agent in an AIX VM

To uninstall a VM Agent in AIX, complete the following steps:
1. Stop the VM Agent module in the AIX VM, as shown in Example 3-91.
Example 3-91 Stopping the AIX VM Agent

# ksysvmmgr status
(0) root @ rt11001: /

# ksysvmmgr stop
(0) root @ rt11001: /
# ksysvmmgr status
2. Uninstall the VM Agent filesets from the AIX VM by running the command that is shown in
Example 3-92.
Example 3-92 Unistalling the VM Agent filesets from AIX

# installp -u ksys*
.
.
.
--------------------
-------------------------------------------------------------------------------
ksys.vmmon.rte 1.3.0.0 ROOT DEINSTALL SUCCESS
ksys.vmmon.rte 1.3.0.0 USR DEINSTALL SUCCESS
Uninstalling a VM Agent in a Linux VM

To uninstall a VM Agent in a Linux VM, complete the following steps:
1. Stop the VM Agent module in the Linux VM, as shown in Example 3-93.
Example 3-93 Stopping the VM Agent module in the Linux VM

# ksysvmmgr status
# ksysvmmgr stop
ksys_vmm has been requested to stop.
# ksysvmmgr status
2. Uninstall the VM Agent package from the Linux VM, as shown in Example 3-94.
Example 3-94 Uninstalling the VM Agent package from the Linux VM

# rpm -e vmagent
Stopping ksys_vmm daemon.

4
Chapter 4. IBM VM Recovery Manager High

Availability GUI deployment
This chapter introduces the IBM VM Recovery Manager High Availability (HA) GUI for Power
Systems deployment.

򐂰 Installing the GUI filesets
򐂰 Configuring and discovering the VM Recovery Manager HA infrastructure by using the
GUI only
򐂰 VM Recovery Manager HA dashboard
򐂰 Relevant move tests

4.1 Installing the GUI filesets
To use the VM Recovery Manager HA by using the GUI, you must install the GUI server
fileset on a system to manage KSYS nodes by using the GUI. The logical partition (LPAR) in
which you want to install the GUI filesets must be running IBM AIX 7.2 with Technology Level
2 Service Pack 1 (7200-02-01) or later. You can choose to install the GUI server fileset on one
of the KSYS nodes. You must install the following GUI server filesets before you start the GUI.
The GUI agent filesets are automatically installed along with the KSYS filesets. To install the
GUI server filesets, complete the following steps:
1. Install the GUI server fileset based on the following scenarios:
If you are installing the GUI server filesets on one of the KSYS nodes, run the following
command:
installp -acFXYd fileset_location -V2 [-e filename.log] ksys.ui.server
If you are installing the GUI server filesets on a separate system that manages all the
KSYS nodes, run the following command:
installp -acFXYd fileset_location -V2 [-e filename.log] ksys.ha.license
ksys.ui.server ksys.ui.common
Example 4-1 shows how to install the GUI server filesets on the KSYS node.
Example 4-1 Installing the GUI server filesets on the KSYS node
# installp -acFXYd . ksys.ui.server
.
.
.
#######################################################################
#######################################################################
##
## The IBM VMRestart for AIX graphical user interface
## (GUI) server installation is starting. To complete the process,
## you must install additional files when the installation completes.
## These additional files were not included in the server fileset
## because they are licensed under the General Public License (GPL).
## However, you can automatically download the required files by
## running the following script:
##
## /opt/IBM/ksys/ui/server/dist/server/bin/vmruiinst.ksh
##
#######################################################################
#######################################################################
.
.
.
--------------------
-------------------------------------------------------------------------------
ksys.ui.server 1.3.0.1 USR APPLY SUCCESS
ksys.ui.server 1.3.0.1 ROOT APPLY SUCCESS

2. Install the open source software packages, which are not included in the installed filesets,
based on the following scenarios.
If the GUI server LPAR is connected to the internet, run the command in the GUI server
LPAR as shown in Example 4-2. The command downloads and installs the remaining files
that are not included in the filesets because these files are licensed under the General
Public License (GPL).
Example 4-2 Installing open source software packages on LPAR to connect to the internet
# /opt/IBM/ksys/ui/server/dist/server/bin/vmruiinst.ksh
Warning: "/tmp/vmruiinst.downloads" does not exist. Creating...

"/tmp/vmruiinst.downloads" has been created.
Checking if the requisites have already been downloaded...

** "info-4.13-3.aix5.3.ppc.rpm" needs to be retrieved.
** "cpio-2.11-2.aix6.1.ppc.rpm" needs to be retrieved.
** "readline-6.2-2.aix5.3.ppc.rpm" needs to be retrieved.
** "libiconv-1.13.1-2.aix5.3.ppc.rpm" needs to be retrieved.
** "bash-4.2-5.aix5.3.ppc.rpm" needs to be retrieved.
** "gettext-0.17-6.aix5.3.ppc.rpm" needs to be retrieved.
** "libgcc-4.9.2-1.aix6.1.ppc.rpm" needs to be retrieved.
** "libgcc-4.9.2-1.aix7.1.ppc.rpm" needs to be retrieved.
** "libstdc++-4.9.2-1.aix6.1.ppc.rpm" needs to be retrieved.
** "libstdc++-4.9.2-1.aix7.1.ppc.rpm" needs to be retrieved
Total number of requisite downloads needed: 10

.
.
.
To use the VM Restart for AIX graphical user interface (GUI), you
must install third-party files. These third-party files were not included in
the server filesets because they are licensed under the General Public License
(GPL). This script downloads and installs the files that are required by
SQLite and Node.js.
SQLite dependencies: | Node.js dependencies:

========================================= | =======================
bash info | libgcc
libiconv readline | libstdc++
gettext cpio |
The files that are required for Node.js are used on the server, and are also
installed on each cluster node automatically during the cluster discovery
operation.
IBM does not offer support for these files if the files are used outside the
context of the VM Restart GUI.
Do you want to download and install these files? (y/n)

Y
.
.
.
Attempting to start the server...
The server was successfully started.
Chapter 4. IBM VM Recovery Manager High Availability GUI deployment 325

The installation completed successfully. To use the VM Restart GUI,
open a web browser and enter the following URL:
https://9.x.x.x:3000/login
After you log in, you can add existing clusters in your environment to the
VM Restart GUI.
If the GUI server LPAR is configured to use an HTTP/HTTPS proxy to access the internet,
run the following command in the GUI server LPAR to specify the proxy information:
/opt/IBM/ksys/ui/server/dist/server/bin/vmruiinst.ksh [-p <HTTP_PROXY>] [-P
<HTTPS_PROXY>]
Tip: You can also specify the proxy information by using the http_proxy environment
variable.
If the GUI server LPAR is not connected to the internet, complete the following steps:
a. Copy the vmruiinst.ksh file from the GUI server LPAR to a system that is running the
AIX operating system and that has internet access.
b. Run the vmruiinst.ksh -d /directory command where /directory is the location
where you want to download the remaining files. For example, /vmruiinst.ksh -d
/tmp/vmrui_rpms.
c. Download the following prerequisite packages for the GUI server:
• info-4.13-3.aix5.3.ppc.rpm
• cpio-2.11-2.aix6.1.ppc.rpm
• readline-6.2-2.aix5.3.ppc.rpm
• libiconv-1.13.1-2.aix5.3.ppc.rpm
• bash-4.2-5.aix5.3.ppc.rpm
• gettext-0.17-6.aix5.3.ppc.rpm
• libgcc-4.9.2-1.aix6.1.ppc.rpm
• libgcc-4.9.2-1.aix7.1.ppc.rpm
• libstdc++-4.9.2-1.aix6.1.ppc.rpm
• libstdc++-4.9.2-1.aix7.1.ppc.rpm
d. Copy the downloaded files to a directory in the GUI server LPAR.
e. In the GUI server LPAR, run the vmruiinst.ksh -i /directory command where
/directory is the location where you copied the downloaded files.
f. After the vmruiinst.ksh command completes, a message displays a URL for the VM
Recovery Manager HA GUI server. Enter the specified URL in a web browser in the
GUI server LPAR and start using the VM Recovery Manager HA GUI.

4.2 Configuring and discovering the VM Recovery Manager HA
infrastructure by using the GUI only
This chapter demonstrates how to configure and discover the infrastructure of VM Recovery
Manager HA from the start by using only the GUI.
Complete the following steps:

1. After installing the VM Recovery Manager HA GUI, use the URL that is presented at the
end of the installation to log in to the GUI, as shown in the Figure 4-1.
Figure 4-1 VM Recovery Manager HA GUI login window
After logging in to the VM Recovery Manager HA GUI, the window that is shown in
Figure 4-2 opens.
Figure 4-2 Welcome window for the VM Recovery Manager HA GUI

The AIX LPAR the VM Recovery Manager HA GUI installed can manage one KSYS to
multiple KSYS subsystems.
2. After clicking Add KYS, as shown in Figure 4-3, add the following KSYS login information:
– KSYS IP address
– Root user
– Password
– Cluster type
Figure 4-3 Adding KSYS Login information
3. In the Host Group Details window, add the host group name, as shown in Figure 4-4.
Figure 4-4 Host Group Details window

4. In the HCM Selection window, enter the Hardware Management Console (HMC)
information, as shown in Figure 4-5.
Figure 4-5 HMC Selection window
5. In the Host Selection window, there is a request to select the host that will be part of this
host group. In this case, select host rt12, as shown in Figure 4-6.
Figure 4-6 Host Selection window: Adding host rt12

Figure 4-7 shows the selection of host rt11.
Figure 4-7 Host Selection window: Adding host rt11
6. In the Virtual I/O Server (VIOS) Selection window, select a VIOS from the hosts, as shown
in Figure 4-8.
Figure 4-8 VIOS Selection window
7. In the Disk Section window, enter the disks to use as the Repository Disk and HA Pool
Disk, as shown in Figure 4-9 on page 331.

Figure 4-9 Disk Selection window
8. In the VM Selection window, you are requested to select the virtual machine (VM) that will
be managed on this host group. Select the VM and click Policies to change the
VM-specific policies, as shown in Figure 4-10.
Figure 4-10 Changing the VM policies

Figure 4-11 shows the VM that is selected from host rt11.
Figure 4-11 Selecting a VM from host rt11
Figure 4-12 shows the VM that is selected from host rt12.
Figure 4-12 Selecting a VM from host rt12
The summary shows details of the host group that is configured, as shown in Figure 4-13
on page 333.

Figure 4-13 Summary Host Group configuration
9. Click Submit & Deploy. The deployment of the host group configuration starts, as shown
in Figure 4-14.
Figure 4-14 Deploying the host group configuration

When the deployment finishes, the message that is shown in Figure 4-15 appears.
Figure 4-15 Finishing the deployment of the host group
10.Click Go in the dashboard to access the management options, as shown in Figure 4-16.
Figure 4-16 VM Recovery Manager HA dashboard

4.3 VM Recovery Manager HA dashboard
This section shows the different features in the dashboard.
4.3.1 KSYS cluster features

To check and change the KSYS cluster policies, click ksyscluster, as shown in Figure 4-17.
Figure 4-17 The ksyscluster policies
To unregister the KSYS and set the Notifications Preferences, go to the Settings window and
click Notification Preferences, as shown in Figure 4-18.
Figure 4-18 Setting the Notifications Preferences

To add or remove an HMC, click Edit, as shown in Figure 4-19.
Figure 4-19 Adding and removing an HMC
To check the ITSO_HG host group components and policies, click the ITSO_HG host group, as
Figure 4-20 Host group policies
To add or remove a host from the host group, click Edit, as shown in Figure 4-21.
Figure 4-21 Adding or removing a host from the host group

To configure the collocation policies for the VMs, click Other Policies for VM, as shown in
Figure 4-22.
Figure 4-22 Configuring collocation policies
Figure 4-23 shows the configuration of the Anti-Collocation policy for the VMs rt11005 and
rt12005.
Figure 4-23 Configuring the Anti-Collocation policy
Clicking Discovery & Verify shows three discovery and verify options: Discovery Host Group,
Verify Host Group, and Discovery and Verify Host Group, as shown in Figure 4-24.
Figure 4-24 Discover & Verify menu options

When you click Discovery and Verify Host Group, a confirmation window opens, as shown
in Figure 4-25.
Figure 4-25 Confirming to run Discovery and Verify
Click Discovery and Verify and all running activities are displayed, as shown in Figure 4-26.
Figure 4-26 Running Discovery and Verify
To filter the event logs by Critical, Warning, Maintenance, and Informational event, click
Events, as shown in Figure 4-27.
Figure 4-27 Filtering the Events log

To show detailed information, click the event log, as shown in Figure 4-28.
Figure 4-28 Information Event error detail
As shown in Figure 4-20 on page 336 (left side), you can also check the host group, HMC,
and hosts, as shown in Figure 4-29.
Figure 4-29 Checking the host group components

When you select the host rt11, an information summary opens, as shown in Figure 4-30.
Figure 4-30 Host rt11 information summary
Click Policies, and the policies information from host rt11 appear, as shown in Figure 4-31 on
page 340.
Figure 4-31 Policies information from host rt11

You can check on all VMs that are related to host rt11, as shown in Figure 4-32.
Figure 4-32 Checking the VMs that are related to host rt11

When you select the host rt12 (Figure 4-29 on page 339), an information summary window
opens, as shown in Figure 4-33.
Figure 4-33 Host rt12 information summary
Click Policies, and the policies information from host rt12 appear, as shown in Figure 4-34.
Figure 4-34 Policies information about host rt12
You can check all VMs that are related to host rt12, as shown in Figure 4-35.
Figure 4-35 Checking VMs that are related to host rt12

When you select the VIOS rt11v1, an informational summary appears, as shown in
Figure 4-36.
Figure 4-36 Checking the VIOS summary
When you click SSP Cluster for the rt11v1 VIOS, the cluster name and status appear, as
Figure 4-37 Checking the SSP Cluster information from the VIO server
Click Policies on the rt11v1 VIOS, and the information policies appear, as shown in
Figure 4-38.
Figure 4-38 Checking the policy information for VIOS

Select the VM rt11001, as shown in Figure 4-32 on page 341. A summary opens, as shown in
Figure 4-39.
Figure 4-39 Checking the VM summary
To check the application that is managed by the VM monitor (VMM) daemon, click
Application, as shown in Figure 4-40.
Figure 4-40 Checking the application monitor

Click Policies for the rt1001 VM, and the policy information appears, as shown in
Figure 4-41.
Figure 4-41 Checking the VM policies
4.4 Relevant move tests

This section shows the relevant VM Recovery Manager HA move tests.
4.4.1 VM Live Partition Mobility move test

This test performs a Live Partition Mobility (LPM) move of VM rt11001.
Select VM rt11001 and click Migrate → Migrate to New Host Using LPM, as shown in
Figure 4-42.
Figure 4-42 Migration to a new host by using LPM

The available hosts and the option for KSYS to decide the best host are shown. In this case,
we select the host that is shown in Figure 4-43.
Figure 4-43 Selecting the host to migrate by LPM
The LPM activity message appears, as shown in Figure 4-44.
Figure 4-44 LPM activity

During the LPM activity, a purple signal appears, as shown in Figure 4-45.
Figure 4-45 Signal that is shown during LPM activity
After the LPM operation finishes, the dashboard shows VM rt11001 at host rt12, as shown in
Figure 4-46.
Figure 4-46 Dashboard after LPM migration

Now, we test the option to migrate back to the home host, as shown in Figure 4-47.
Figure 4-47 Migrating back to the home host
The same purple signal appears, as shown in Figure 4-48.
Figure 4-48 Signal that is shown while migrating back to the home host

After the LPM operation finishes, the dashboard shows VM rt11001 at host rt11, as shown in
Figure 4-49.
Figure 4-49 Dashboard after the LPM migration
4.4.2 VM restart move test

This test performs a VM restart to move VM rt12001.
Select VM rt12001, and then select Restart → Restart & Bring to New Host, as shown in
Figure 4-50.
Figure 4-50 Migration to a new host by using restart

The available hosts appear, as shown in Figure 4-51.
Figure 4-51 Choosing the host to migrate due to a restart
During the restart activity, a red signal appears, as shown in Figure 4-52.
Figure 4-52 Signal shown during the VM restart activity

After the restart operation finishes, the dashboard shows VM rt12001 at host rt11, as shown
in Figure 4-53.
Figure 4-53 Dashboard after the restart migration
Now, we test the option to Restart at Home Host, as shown in Figure 4-54.
Figure 4-54 Migrating back to the home host

The same red signal is shown in Figure 4-55.
Figure 4-55 Signal that is shown while migrating back to the home host
After the restart operation finishes, the dashboard shows VM rt12001 at host rt12, as shown
in Figure 4-56.
Figure 4-56 Dashboard after the restart migration

4.4.3 Host LPM move test
In this test, the Host LPM moves all managed VMs from host rt12 to rt11.
Select host rt12, and then click Migrate → Migrate to New host using LPM, as shown in
Figure 4-57.
Figure 4-57 Migrating all VMs for one host by using LPM
The available hosts appear, as shown in Figure 4-58.
Figure 4-58 Choosing the host to migrate due to LPM

All VMs are now managed from host rt12 and a purple signal appears, as shown in
Figure 4-59.
Figure 4-59 Signal shown during host LPM activity
After the LPM operation finishes, the dashboard shows that all VMs that are managed from
host rt12 are migrated to host rt11, as shown in Figure 4-60.
Figure 4-60 Dashboard after the Host LPM migration

Now, we click Migrate Back to Home Host on all VMs from host rt12, as shown in
Figure 4-61.
Figure 4-61 Migrating back all VMs for one host by using LPM
The message restore has started for vm appears, as shown in Figure 4-62.
Figure 4-62 Restore messages for VMs
A purple signal appears, as shown in Figure 4-63.
Figure 4-63 Signal shown during host LPM activity

After the LPM operation finishes, the dashboard shows that all VMs that are managed from
host rt12 are migrated back from host rt11, as shown in Figure 4-64.
Figure 4-64 Dashboard after the Host LPM migration

5
Chapter 5. Test scenarios

This chapter provides test scenarios that are performed in an example non-production
environment.

򐂰 Test environment
򐂰 Linux virtual machine failures
򐂰 AIX VM failures
򐂰 Host failures

5.1 Test environment
Figure 5-1 shows the environment that is used for the test scenarios in this chapter.
Figure 5-1 Environment that is used for the test scenarios
The environment is composed of the following items:

򐂰 One KSYS
򐂰 Two IBM Power Systems S822 servers: S822-1 and S882-2
򐂰 Two Hardware Management Consoles (HMCs): HMC1 and HMC2
򐂰 One IBM Storwize V7000 that is used for shared storage
Example 5-1 shows the KSYS cluster that is created.
Example 5-1 Listing the KSYS cluster

p18ksys:/ # ksysmgr query cluster
Name: ksyscluster
State: Online
Type: HA
The HMCs that are used in this environment are listed in Example 5-2.
Example 5-2 Listing the HMCs

p18ksys:/ # ksysmgr query hmc
Name: hmc2
Ip: 129.40.180.25
Login: hscroot
Managed Host List:
Host Name Uuid

========= ====

Server-8284-22A-SN10EE85P fc68b423-a590-3db7-86e9-a7b7d26a07f0
============================================================================
Name: hmc1
Ip: 129.40.180.24
Login: hscroot
Managed Host List:
Host Name Uuid

========= ====
Server-8284-22A-SN101AFDR 07e502a4-5051-3867-bd64-11adc2fe8e68
============================================================================
The host group that is used in this scenario is shown in Example 5-3.
Example 5-3 Listing the host group

p18ksys:/ # ksysmgr query host_group
Name: HG_TCHU
Hosts: Server-8284-22A-SN101AFDR
Server-8284-22A-SN10EE85P
low:100
medium:100
high:100
low:100
medium:100
high:100
Skip_power_on: No
HA_monitor: enable
SSP Cluster Attributes

Sspname: KSYS_ksyscluster_1
Sspstate: UP
Ssp_version: VIOS 3.1.0.11
VIOS: p18v02
p18v01
p18v04
p18v03
Chapter 5. Test scenarios 359

5.2 Linux virtual machine failures
This scenario simulates a Linux virtual machine (VM) failure that is managed by IBM VM
Recovery Manager High Availability (HA).

1. Check where the VM that is used in the scenario is allocated. Example 5-4 shows VM
p18lnx02 is on host Server-8284-22A-SN10EE85P before you start the Linux VM failure.
Example 5-4 Listing the location of VM p818nx02

p18ksys:/ # ksysmgr query vm p18lnx02
Name: p18lnx02
UUID: 495778C3-80DB-4FE3-989D-E38C905450E8
State: VERIFY
Host: Server-8284-22A-SN101AFDR
Priority: High
VM_failure_detection_speed: fast
HA_monitor: enable
Homehost: Server-8284-22A-SN10EE85P
2. List all VMs that are allocated to host Server-8284-22A-SN101AFDR, as shown in

Example 5-5.
Example 5-5 Listing VMs that are allocated on host Server-8284-22A-SN101AFDR

hscroot@p18vhmc1:~> lsrefcode -m Server-8284-22A-SN101AFDR -r lpar -F lpar_name
p18v03
p18ibmi01
p18v04
p18aix02
p18aix03
p18aix04
p18aix05
p18lnx02
3. List all VMs that are allocated to host Server-8284-22A-SN10EE85P, as shown in

Example 5-6.
Example 5-6 Listing VMs that are allocated to host Server-8284-22A-SN10EE85P

hscroot@p18vhmc2:~> lsrefcode -m Server-8284-22A-SN10EE85P -r lpar -F lpar_name
p18v01
p18v02
p18lnx01
4. Check the version of Linux in VM p18lnx02, as shown in Example 5-7.
Example 5-7 Checking the Linux version on VM p18lnx02

p18lnx02:/etc # cd ..
p18lnx02:/ # cat /etc/os-release
NAME="SLES"
VERSION="15"
VERSION_ID="15"

PRETTY_NAME="SUSE Linux Enterprise Server 15"
ID="sles"
ID_LIKE="suse"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:suse:sles:15"
5. Simulate a Linux kernel crash, as shown in Example 5-8.
Example 5-8 Simulating a Linux crash

p18lnx02:/ # sh -x system_crash.sh
kernel.panic=0
+ echo c
sysrq: SysRq : Trigger a crash
Oops: Kernel access of bad area, sig: 11 [#1]
SMP NR_CPUS=2048
NUMA
pSeries
.
.
.
Kernel panic - not syncing: Fatal exception
---[ end Kernel panic - not syncing: Fatal exception
On HMC p18vhmc1, VM p18lnx02 changed the reference code, as shown in Example 5-9.
Example 5-9 VM p18lnx02 reference code after a Linux kernel crash
p18lnx02:B200A101 LP=00009
Example 5-10 shows that in HMC p18vhmc2 VM p18lnx02 restarted.
Example 5-10 VM p18lnx02 initializing on Server-8284-22A-SN10EE85P

p18lnx02:CA00E140
6. Monitor the KSYS to see the VM restart of p18lnx0, as shown in Example 5-11.
Example 5-11 Monitoring the VM restart

p18ksys:/ # ksysmgr query system status monitor=yes
Host_group HG_TCHU is currently in Ready state
Press Q to quit monitoring for activity
Restart in progress for Host_group HG_TCHU
Stopping HA monitoring for VM p18lnx02
HA monitoring for VM p18lnx02 stopped
Shutdown has started for VM p18lnx02
Shutdown has completed for VM p18lnx02
Restart has started for VM p18lnx02
Starting HA monitoring for VM p18lnx02
HA monitoring for VM p18lnx02 started
Restart on Target host Server-8284-22A-SN10EE85P has completed for VM
p18lnx02

7. Check the status of VM p18lnx02 on host Server-8284-22A-SN10EE85P, as shown in
Example 5-12.
Example 5-12 Listing VM s on host Server-8284-22A-SN10EE85P

p18v01
p18v02
p18lnx01
p18lnx02
After the final restart of VM p18lnx02, the reference on host

Server-8284-22A-SN101ADFDR is cleared, as shown in Example 5-13.
Example 5-13 Listing VMs on host Server-8284-22A-SN101AFDR

scroot@p18vhmc1:~> lsrefcode -m Server-8284-22A-SN101AFDR -r lpar -F lpar_name
p18v03
p18ibmi01
p18v04
p18aix02
p18aix03
p18aix04
p18aix05
8. Check the status of VM p18lnx02 by running the ksysmgr command, as shown in

Example 5-14.
Example 5-14 Listing VM p18lnx02

p18ksys:/ # ksysmgr query vm p18lnx02
Name: p18lnx02
UUID: 495778C3-80DB-4FE3-989D-E38C905450E8
State: READY
Host: Server-8284-22A-SN10EE85P
Priority: High
VM_failure_detection_speed: fast
HA_monitor: enable


5.3 AIX VM failures
This scenario simulates an AIX VM failure when managed by VM Recovery Manager HA.

1. Check where the VM that is used for this scenario is allocated. Example 5-15 shows that
VM p18aix02 is on host Server-8284-22A-SN10EE85P before you start the VM failure.
Example 5-15 Listing VM p18aix02

p18ksys:/ # ksysmgr query vm p18aix02
Name: p18aix02
UUID: 52138D9D-034C-4114-AA45-4E2DA66D8761
State: VERIFY
Host: Server-8284-22A-SN101AFDR
Priority: Medium
HA_monitor: enable
Version_conflict:
2. List all VMs that are allocated on host Server-8284-22A-SN101AFDR, as shown in

Example 5-16.
Example 5-16 Listing VMs that are allocated on host Server-8284-22A-SN101AFDR

hscroot@p18vhmc1:~> lsrefcode -m Server-8284-22A-SN101AFDR -r lpar -F lpar_name
p18v03
p18ibmi01
p18v04
p18aix02
p18aix03
p18aix04
p18aix05
3. List all VMs that are allocated on host Server-8284-22A-SN10EE85P, as shown in

Example 5-17.
Example 5-17 Listing VMs that are allocated on host Server-8284-22A-SN10EE85P

p18v01:
p18v02:
p18lnx01:Linux ppc64le
4. Check the version of AIX from VM p18aix02, as shown in Example 5-18.
Example 5-18 Checking the AIX version on VM p18aix02

p18aix02:/ # oslevel -s
7200-03-02-1846

5. Simulate the AIX kernel crash, as shown in Example 5-19.
Example 5-19 Simulating an AIX kernel crash

p18aix02:/vm_crash # sh -x aix_crash.sh
+ + hostname -s
shost=p18aix02
+ PS1=p18aix02:$PWD #
+ alias __A=
+ alias __B=
+ alias __C=
+ alias __D=
+ alias __H=
.
.
.
+ ./crash_vm -c
Crash ..
Uhh.. I am crashing
crash
Illegal Trap Instruction Interrupt in Kernel
.panic_trap+000000 tweq r14,r14 r14=0000000000000002
KDB(0)> PuTTY
On HMC p18vhmc1, VM p18lnx02 changed the reference code, as shown in

Example 5-20.
Example 5-20 VM p18aix02 reference code after a kernel crash

p18aix02:0c20
On HMC p18vhmc2, VM p18lnx02 restarted, as shown in Example 5-21.
Example 5-21 VM p18aix02 initializing on Server-8284-22A-SN10EE85P

p18aix02:CA00E140
6. Monitor the KSYS to see the restart of VM p18lnx02, as shown in Example 5-22.
Example 5-22 Monitoring the VM restart

Stopping HA monitoring for VM p18aix02
HA monitoring for VM p18aix02 stopped
Shutdown has started for VM p18aix02
Shutdown has completed for VM p18aix02
Restart has started for VM p18aix02
Starting HA monitoring for VM p18aix02
HA monitoring for VM p18aix02 started
Restart on Target host Server-8284-22A-SN10EE85P has completed for VM p18aix02
Configuration cleanup started for VM p18aix02
VM monitoring for VM p18aix02 started
Configuration cleanup completed for VM p18aix02
1 out of 1 VMs have been successfully restarted

After the final restart of VM p18aix02, the reference code on host
Server-8284-22A-SN101ADFDR is cleared, as shown in Example 5-23.
Example 5-23 Listing the VMs on host Server-8284-22A-SN101AFDR

scroot@p18vhmc1:~> lsrefcode -m Server-8284-22A-SN101AFDR -r lpar -F lpar_name
p18v03
p18ibmi01
p18v04
p18aix03
p18aix04
p18aix05
7. Check the home host of VM p18lnx02, as shown in Example 5-24.
Example 5-24 Listing VM s on host Server-8284-22A-SN10EE85P

scroot@p18vhmc2:~> lsrefcode -m Server-8284-22A-SN10EE85P -r lpar -F lpar_name
p18v01:
p18v02:
p18aix02:
8. Check the home host of VM p18lnx02 by running the ksysmgr command, as shown in
Example 5-25.
Example 5-25 Listing VM p18aix02

p18ksys:/ # ksysmgr query vm p18aix02
Name: p18aix02
UUID: 52138D9D-034C-4114-AA45-4E2DA66D8761
Host: Server-8284-22A-SN10EE85P
Priority: Medium
HA_monitor: enable


5.4 Host failures
This scenario simulates a host crash of host Server-8284-22A-SN101AFDR.

1. List the VMs that are available on host Server-8284-22A-SN101AFDR, as shown in
Example 5-26.
Example 5-26 Listing VMs that are available on host Server-8284-22A-SN101AFDR

hscroot@p18vhmc1:~> lsrefcode -m Server-8284-22A-SN101AFDR -r lpar -F
lpar_name:refcode
p18v03:
p18ibmi01:00000000
p18v04:
p18aix02:
p18aix03:
p18aix04:
p18aix05:
2. List the VMs that are available on host Server-8284-22A-SN10EE85P, as shown in

Example 5-27.
Example 5-27 Listing VMs that are available on host Server-8284-22A-SN10EE85P

hscroot@p18vhmc2:~> lsrefcode -m Server-8284-22A-SN10EE85P -r lpar -F
lpar_name:refcode
p18v01:
p18v02:
3. Simulate a host crash by shut downing immediately both Virtual I/O Servers (VIOSs) from
host Server-8284-22A-SN10EE85P, as shown in Example 5-28.
Example 5-28 Shutting down the VIOSs from host Server-8284-22A-SN10EE85P

chsyscfg chsyspwd chsysstate
hscroot@p18vhmc1:~> chsysstate -m Server-8284-22A-SN101AFDR -r lpar -n p18v03 -o
shutdown --immed
shutdown --immed
All VMs from host Server-8284-22A-SN10EE85P have crashed, as shown in

Example 5-29.
Example 5-29 VMs crashed from host Server-8284-22A-SN10EE85P

p18ibmi01:A6040266
p18v04:00000000
p18aix02:0c20
p18aix03:CA00E175
p18aix04:CA00E175
p18aix05:CA00E175
p18lnx02:00000000

4. Check the KSYS by running the ksysmgr command. All monitored VMs restarted in host
Server-8284-22A-SN10EE85P, as shown in Example 5-30.
Example 5-30 VM restarting in host Server-8284-22A-SN10EE85P

Stopping HA monitoring for VM p18lnx02
HA monitoring for VM p18lnx02 stopped
Shutdown has started for VM p18lnx02
Shutdown has completed for VM p18lnx02
Restart has started for VM p18lnx02
Starting HA monitoring for VM p18lnx02
HA monitoring for VM p18lnx02 started
p18lnx02
Shutdown has started for VM p18ibmi01
Starting VM monitoring for VM p18lnx02
Shutdown has completed for VM p18ibmi01
Restart has started for VM p18ibmi01
p18ibmi01
Configuration cleanup started for VM p18ibmi01
ERROR: Restart has encountered an error for VM p18ibmi01 during
Configuration Cleanup

p18aix02
p18aix04
p18aix05
p18aix03
Configuration cleanup started for VM p18lnx02
ERROR: Restart has encountered an error for VM p18lnx02 during
ERROR: Restart has encountered an error for VM p18aix02 during
6 out of 6 VMs have been successfully restarted
5. List the VMs on host Server-8284-22A-SN10EE85P. You see that all partitions that are
monitored from the KSYS from the crashed host Server-8284-22A-SN101AFDR restarted
on host Server-8284-22A-SN10EE85P, as shown in Example 5-31.
Example 5-31 Listing VMs on host Server-8284-22A-SN10EE85P

hscroot@p18vhmc2:~> lsrefcode -m Server-8284-22A-SN10EE85P -r lpar -F
lpar_name:refcode
p18v01:
p18ibmi01:00000000
p18v02:
p18aix02:
p18aix03:
p18aix04:
p18aix05:
6. List the VMs on host Server-8284-22A-SN101AFDR. You see that the referenced VMs
restarted, as shown in Example 5-32.
Example 5-32 Listing the remaining referenced VMs

lpar_name:refcode
p18v03:00000000
p18ibmi01:00000000
p18v04:00000000

p18aix02:00000000
p18aix03:00000000
p18aix04:00000000
p18aix05:00000000
p18lnx02:00000000
7. Start the VIOS from host Server-8284-22A-SN101AFDR, as shown in Example 5-33.
Example 5-33 Starting the VIOS from host Server-8284-22A-SN101AFDR

on -f default_profile
on -f default_profile
8. Perform discovery and verification on host group HG_TCHU, as shown in Example 5-34.
Example 5-34 Performing discovery and verification on host group HG_TCHU

p18ksys:/ # ksysmgr discovery host_group HG_TCHU verify=yes
Running discovery on Host_group HG_TCHU, this may take few minutes...
Existing HA trunk adapter found for VIOS p18v01
Preparing VIOS in Server-8284-22A-SN101AFDR for HA management
VIOS in Server-8284-22A-SN101AFDR prepared for HA management
Preparing VIOS in Server-8284-22A-SN10EE85P for HA management
VIOS in Server-8284-22A-SN10EE85P prepared for HA management
Preparing VM p18aix03 in Host Server-8284-22A-SN10EE85P for HA management
VM p18aix03 in Host Server-8284-22A-SN10EE85P Prepared for HA management
Preparing VM p18lnx02 in Host Server-8284-22A-SN10EE85P for HA management
VM p18lnx02 in Host Server-8284-22A-SN10EE85P Prepared for HA management
Existing first HA client adapter found for VM p18lnx02
Existing first HA client adapter found for VM p18aix03
Existing second HA client adapter found for VM p18aix03
Existing second HA client adapter found for VM p18lnx02
Skipping VM p18lnx02 on Host Server-8284-22A-SN10EE85P due to some other
process in progress
Discovery has started for VM p18aix03
Configuration information retrieval started for VM p18aix03
Discovery has started for VM p18ibmi01
Configuration information retrieval started for VM p18ibmi01

Configuration information retrieval completed for VM p18aix03
Discovery for VM p18aix03 is complete
Configuration information retrieval completed for VM p18ibmi01
Discovery for VM p18ibmi01 is complete
VM monitor state has moved to 'STARTED' for VM p18aix03
Discovery has finished for HG_TCHU
1 managed VM has been skipped for discovery
Host_group verification started for HG_TCHU

p18aix03 verification has started
p18ibmi01 verification has started
p18lnx02 verification has started
p18ibmi01 verification has completed
p18lnx02 verification has completed
p18aix05 verification has completed
Verification has finished for HG_TCHU
After the discovery and verification of host group HG_TCHU, the VMs that are referenced on
host Server-8284-22A-SN101AFDR are cleared, as shown in Example 5-35.
Example 5-35 Listing VMs on host Server-8284-22A-SN101AFDR

lpar_name:refcode
p18v03:
p18v04:

A
Appendix A. Additional material

This book refers to additional material that can be downloaded from the internet, as described
in the following sections.
Locating the web material

The web material that is associated with this book is available in softcopy on the internet from
the IBM Redbooks web server:
ftp://www.redbooks.ibm.com/redbooks/SG248426
Alternatively, you can go to the IBM Redbooks website:

ibm.com/redbooks
Search for SG248426, select the title, and then click Additional materials to open the directory
that corresponds with the IBM Redbooks form number, SG248426.
Using the web material

The additional web material that accompanies this book includes the following files:
File name Description
IBM_VMR_DR_IP_change_scripts.tar IP change script
System requirements for downloading the web material

The web material requires the following system configuration:
Hard disk space: 100 MB minimum
Operating System: Windows, Linux, or macOS
Processor: i3 minimum
Memory: 1024 MB minimum

Downloading and extracting the web material
Create a subdirectory (folder) on your workstation, and extract the contents of the web
material compressed file into this folder.

Related publications
The publications that are listed in this section are considered suitable for a more detailed
description of the topics that are covered in this book.
IBM Redbooks
The following IBM Redbooks publications provide more information about the topics in this
document. Some publications that are referenced in this list might be available in softcopy
only.
򐂰 IBM PowerVM Enhancements What is New in 2013, SG24-8198
򐂰 A Practical Guide for Resource Monitoring and Control (RMC), SG24-6615
You can search for, view, download, or order these documents and other Redbooks,
Redpapers, web docs, drafts, and additional materials at the following website:
ibm.com/redbooks
Online resources
These websites are also relevant as further information sources:
򐂰 IBM Knowledge Center: Hardware Management Console (HMC) REST APIs
https://ibm.co/2Xol4Gf
򐂰 IBM Knowledge Center: Preparing for partition mobility
https://ibm.co/2KrXsdQ
򐂰 Service and Productivity Tools
https://ibm.co/2x3aPbt
Help from IBM

IBM Support and downloads
ibm.com/support
IBM Global Services

ibm.com/services

SG24-8426-00
Implementing IBM VM Recovery Manager for IBM Power SystemsISBN 0738458104
(0.5” spine)
0.475”<->0.873”
250 <-> 459 pages
Back cover
SG24-8426-00
ISBN 0738458104
Printed in U.S.A.
®
ibm.com/redbooks

SG 248426

Uploaded by

Copyright:

Available Formats

SG 248426

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

SG 248426

Uploaded by

Copyright:

Available Formats

Front cover

Implementing IBM VM Recovery Manager for IBM

First Edition (October 2019)

This edition applies to:

© Copyright International Business Machines Corporation 2019. All rights reserved.

Chapter 1. Introduction to high availability and disaster recovery . . . . . . . . . . . . . . . . 1

Chapter 2. IBM VM Recovery Manager High Availability components . . . . . . . . . . . . 39

© Copyright IBM Corp. 2019. All rights reserved. iii

Chapter 4. IBM VM Recovery Manager High Availability GUI deployment . . . . . . . . 323

Chapter 5. Test scenarios . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 357

Appendix A. Additional material . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 371

iv Implementing IBM VM Recovery Manager for IBM Power Systems

Related publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 373

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION “AS IS”

© Copyright IBM Corp. 2019. All rights reserved. vii

The following terms are trademarks of other companies:

viii Implementing IBM VM Recovery Manager for IBM Power Systems

This publication targets technical professionals (consultants, technical support staff, IT

Dino Quintero is an IT Management Consultant and an IBM Level 3 Senior Certified IT

© Copyright IBM Corp. 2019. All rights reserved. ix

Primitivo Cervantes is a Senior Certified IT Specialist at IBM US and provides HA and DR

Byron Martinez Martinez is an IT Specialist at IBM Colombia who provides services to

x Implementing IBM VM Recovery Manager for IBM Power Systems

Thanks to the following people for their contributions to this project:

Javier Bazan Lazcano

Ahmed (Mash) Mashhour

Now you can become a published author, too!

Stay connected to IBM Redbooks

xii Implementing IBM VM Recovery Manager for IBM Power Systems

Chapter 1. Introduction to high availability

The following topics are described in this chapter:

© Copyright IBM Corp. 2019. All rights reserved. 1

Figure 1-1 Continuous availability, HA, and continuous operation relationship

Continuous operations (CO) The ability of a system to continuously operate and

2 Implementing IBM VM Recovery Manager for IBM Power Systems

1.1.1 General concepts

From an operating system or system perspective, two concepts also exist:

Chapter 1. Introduction to high availability and disaster recovery 3

Figure 1-2 Internal- and external-managed HA and DR

Highlights of this environment include these items:

Highlights of this environment include these items:

1.1.2 Outage-related differences

4 Implementing IBM VM Recovery Manager for IBM Power Systems

In an external-managed environment, you need a longer downtime to do an update to the

Figure 1-3 Software and hardware maintenance-dependent action times

Chapter 1. Introduction to high availability and disaster recovery 5

In an external-managed environment, data corruption in the operating system or the

Figure 1-4 Failure-dependent action times

1.1.3 High availability and continuous availability

Table 1-1 Availability measuring

One nine 90 36.5 days 72 hours 16.8 hours

Two nines 99 3.65 days 7.2 hours 1.68 hours

Three nines 99.9 8.76 hours 43.2 min. 10.1 min.

Four nines 99.99 52.56 min. 4.32 min. 1.01 min.

Five nines 99.999 5.26 min. 25.9 sec. 6.05 sec.

Six nines 99.9999 31.55 sec.a 2.59 sec. 0.605 sec.

6 Implementing IBM VM Recovery Manager for IBM Power Systems

What does this mean in terms of the annual availability percentages?

Planing for availability