IBM EC12 sg248049

Front cover
Draft Document for Review August 29, 2012 10:29 am SG24-8049-00
IBM zEnterprise EC12 Technical Guide

Describes the zEnterprise System and related features and functions Discusses hardware and software capabilities Explains virtualizing and managing the infrastructure
Ivan Dobos Wolfgang Fries Parwez Hamid Octavian Lascu Gerard Laumay Swee Seng Ng Fernando Nogal Frank Packheiser Vicente Ranieri Jr Karan Singh Andr Spahni Esra Ufacik Hans Wijngaard Zhaoxu Zhang
ibm.com/redbooks
Draft Document for Review August 29, 2012 10:28 am
8049edno.fm
International Technical Support Organization IBM zEnterprise EC12 Technical Guide August 2012
SG24-8049-00
8049edno.fm
Note: Before using this information and the product it supports, read the information in Notices on page xv.
First Edition (August 2012) This edition applies to the IBM zEnterprise EC12 and the IBM zEnterprise BladeCenter Extension Model 003. This document was created or updated on August 29, 2012. Note: This book is based on a pre-GA version of a product and may not apply when the product becomes generally available. We recommend that you consult the product documentation or follow-on versions of this redbook for more current information.
Copyright International Business Machines Corporation 2010. All rights reserved. Note to U.S. Government Users Restricted Rights -- Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.
8049edno.fm
iii
8049edno.fm
iv
8049TOC.fm
Contents
Notices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xv Trademarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xvii Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xix The team who wrote this book . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xix Now you can become a published author, too! . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxii Comments welcome. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxii Stay connected to IBM Redbooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxii Chapter 1. Introducing the IBM zEnterprise EC12 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.1 IBM zEnterprise EC12 elements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 1.2 zEC12 highlights . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 1.2.1 Flash Express . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 1.3 zEC12 technical overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 1.3.1 Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 1.3.2 Model upgrade paths . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 1.3.3 Frames . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 1.3.4 Processor cage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 1.3.5 I/O connectivity, PCIe, and InfiniBand . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 1.3.6 I/O subsystems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 1.3.7 Cryptography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 1.3.8 Capacity on Demand (CoD) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 1.3.9 Parallel Sysplex support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 1.4 IBM zEnterprise BladeCenter Extension (zBX) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 1.4.1 Blades . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 1.4.2 IBM WebSphere DataPower Integration Appliance XI50 for zEnterprise . . . . . . . 18 1.5 Unified Resource Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 1.6 Hardware Management Consoles and Support Elements . . . . . . . . . . . . . . . . . . . . . . 19 1.7 Operating systems and software. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 1.8 Reliability, availability, and serviceability. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 1.9 Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 1.9.1 LSPR workload suite. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 1.9.2 Fundamental components of workload capacity performance . . . . . . . . . . . . . . . 22 1.9.3 Relative nest intensity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 1.9.4 LSPR workload categories based on relative nest intensity . . . . . . . . . . . . . . . . . 25 1.9.5 Relating production workloads to LSPR workloads . . . . . . . . . . . . . . . . . . . . . . . 26 1.9.6 Workload performance variation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 1.9.7 Main performance improvement drivers with zEC12 . . . . . . . . . . . . . . . . . . . . . . 28 Chapter 2. CPC Hardware Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1 Frames and cage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1.1 Frame A . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1.2 Frame Z . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1.3 I/O cages, I/O drawers, and PCIe I/O drawers . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1.4 Top exit I/O cabling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2 Book concept . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2.1 Book interconnect topology. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2.2 Oscillator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2.3 System control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Copyright IBM Corp. 2010. All rights reserved.
31 32 33 34 34 35 36 37 39 40 v
8049TOC.fm
2.2.4 Book power . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3 Multi-chip module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.4 Processor units and storage control chips . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.4.1 PU chip . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.4.2 Processor unit (core) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.4.3 PU characterization. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.4.4 Storage control (SC) chip . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.4.5 Cache level structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.5 Memory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.5.1 Memory subsystem topology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.5.2 Redundant array of independent memory (RAIM) . . . . . . . . . . . . . . . . . . . . . . . . 2.5.3 Memory configurations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.5.4 Memory upgrades . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.5.5 Book replacement and memory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.5.6 Flexible Memory Option . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.5.7 Preplanned Memory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.6 Reliability, availability, serviceability (RAS) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.6.1 RAS in the CPC memory subsystem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.6.2 General zEC12 RAS features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.7 Connectivity. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.7.1 Redundant I/O interconnect . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.7.2 Enhanced book availability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.7.3 Book upgrade . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.8 Model configurations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.8.1 Upgrades . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.8.2 Concurrent PU conversions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.8.3 Model capacity identifier . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.8.4 Model capacity identifier and MSU value . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.8.5 Capacity Backup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.8.6 On/Off Capacity on Demand and CPs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.9 Power and cooling. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.9.1 Power consumption . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.9.2 High voltage DC power . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.9.3 Internal Battery Feature (IBF) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.9.4 Power capping and power saving . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.9.5 Power estimation tool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.9.6 Cooling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.9.7 Radiator Unit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.9.8 Water Cooling Unit (WCU) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.9.9 Backup air cooling system . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.10 Summary of zEC12 structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Chapter 3. CPC System design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1 Design highlights . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Book design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.1 Cache levels and memory structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.2 Book interconnect topology. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3 Processor unit design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3.1 Out-of-order execution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3.2 Superscalar processor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3.3 Compression and cryptography accelerators on a chip . . . . . . . . . . . . . . . . . . . . 3.3.4 Decimal floating point accelerator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3.5 IEEE floating point . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
40 40 41 42 43 45 45 47 48 49 49 50 53 53 53 54 55 55 56 57 59 60 61 61 62 63 64 65 65 67 68 68 69 69 70 70 70 71 72 75 76 77 78 79 79 82 82 83 86 86 87 88
vi
8049TOC.fm
3.3.6 Processor error detection and recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88 3.3.7 Branch prediction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 3.3.8 Wild branch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 3.3.9 Translation look-aside buffer. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90 3.3.10 Instruction fetching, decode, and grouping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90 3.3.11 Extended translation facility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90 3.3.12 Instruction set extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 3.3.13 Transactional execution (TX) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 3.3.14 Run-time instrumentation (RI) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 3.4 Processor unit functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 3.4.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 3.4.2 Central processors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93 3.4.3 Integrated Facility for Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94 3.4.4 Internal Coupling Facility. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94 3.4.5 System z Application Assist Processors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95 3.4.6 System z Integrated Information Processor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98 3.4.7 zAAP on zIIP capability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100 3.4.8 System Assist Processors. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101 3.4.9 Reserved processors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101 3.4.10 Processor unit assignment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102 3.4.11 Sparing rules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102 3.4.12 Increased flexibility with z/VM-mode partitions . . . . . . . . . . . . . . . . . . . . . . . . . 103 3.5 Memory design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103 3.5.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103 3.5.2 Central storage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 3.5.3 Expanded storage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106 3.5.4 Hardware system area . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106 3.6 Logical partitioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 3.6.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 3.6.2 Storage operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112 3.6.3 Reserved storage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115 3.6.4 Logical partition storage granularity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116 3.6.5 LPAR dynamic storage reconfiguration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116 3.7 Intelligent resource director . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117 3.8 Clustering technology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118 3.8.1 Coupling facility control code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120 3.8.2 Dynamic CF dispatching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121 Chapter 4. CPC I/O System Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1 Introduction to InfiniBand and PCIe . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1.1 Infrastructure types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1.2 InfiniBand specification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1.3 Data, signalling, and link rates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1.4 PCIe . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2 I/O system overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2.1 Characteristics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2.2 Summary of supported I/O features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3 I/O cages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4 I/O drawers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.5 PCIe I/O drawers. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.6 I/O cage, I/O drawer and PCIe I/O drawer offerings . . . . . . . . . . . . . . . . . . . . . . . . . . 4.6.1 I//O cage and I/O drawer carry forward . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.7 Fanouts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123 124 124 125 125 125 126 126 126 127 130 132 134 135 136
Contents
vii
8049TOC.fm
4.7.1 HCA2-C fanout (FC0162) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.7.2 PCIe copper fanout (FC0169) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.7.3 HCA2-O (12xIFB) fanout (FC0163). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.7.4 HCA2-O LR (1xIFB) fanout (FC0168) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.7.5 HCA3-O (12xIFB) fanout (FC0171). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.7.6 HCA3-O LR (1xIFB) fanout (FC0170) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.7.7 Fanout considerations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.7.8 Fanout summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.8 I/O feature cards . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.8.1 I/O feature card types ordering information. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.8.2 PCHID report . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.9 Connectivity. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.9.1 I/O feature support and configuration rules. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.9.2 FICON channels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.9.3 OSA-Express4S features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.9.4 OSA-Express3 features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.9.5 OSA-Express for ensemble connectivity. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.9.6 HiperSockets. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.10 Parallel Sysplex connectivity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.10.1 Coupling links . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.10.2 External clock facility. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.11 Cryptographic functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.11.1 CPACF functions (FC 3863) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.11.2 Crypto Express4S feature (FC 0865) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.11.3 Crypto Express3 feature (FC 0864) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.12 Flash Express . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Chapter 5. CPC Channel Subsystem. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.1 Channel subsystem. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.1.1 Multiple CSSs concept . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.1.2 CSS elements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.1.3 Multiple subchannel sets. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.1.4 Parallel access volumes and extended address volumes. . . . . . . . . . . . . . . . . . 5.1.5 Logical partition name and identification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.1.6 Physical channel ID . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.1.7 Channel spanning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.1.8 Multiple CSS construct . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.1.9 Adapter ID . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.1.10 Channel subsystem enhancement for I/O resilience. . . . . . . . . . . . . . . . . . . . . 5.2 I/O configuration management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3 Channel subsystem summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.4 System-initiated CHPID reconfiguration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.5 Multipath initial program load . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Chapter 6. Cryptography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.1 Cryptographic synchronous functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2 Cryptographic asynchronous functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2.1 Secure key functions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2.2 Additional functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.3 CPACF protected key . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.4 PKCS #11 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.4.1 The PKCS #11 model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.4.2 z/OS PKCS #11 implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
138 138 138 139 140 140 141 143 144 144 145 146 147 149 153 155 158 159 161 161 166 167 167 167 167 167 169 170 170 171 171 174 174 175 175 176 177 177 178 178 180 181 183 184 184 184 185 187 189 189 190
viii
8049TOC.fm
6.4.3 Secure IBM Enterprise PKCS #11 (EP11) Coprocessor. . . . . . . . . . . . . . . . . . . 6.5 Cryptographic feature codes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.6 CP Assist for Cryptographic Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.7 Crypto Express4S . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.7.1 Configuration rules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.8 Crypto Express3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.8.1 Configuration rules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.9 Tasks performed by PCIe Crypto Express . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.9.1 PCIe Crypto Express as a CCA coprocessor . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.9.2 PCIe Crypto Express as an EP11 coprocessor . . . . . . . . . . . . . . . . . . . . . . . . . 6.9.3 PCIe Crypto Express as an Accelerator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.9.4 IBM Common Cryptographic Architecture (CCA) Enhancements. . . . . . . . . . . . 6.10 TKE workstation feature . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.10.1 Logical partition, TKE host, and TKE target . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.10.2 Optional smart card reader . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.11 Cryptographic functions comparison . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.12 Software support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Chapter 7. zBX zEnterprise BladeCenter Extension Model 003 . . . . . . . . . . . . . . . . . 7.1 zBX concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2 zBX hardware description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2.1 zBX racks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2.2 Top of rack (TOR) switches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2.3 zBX BladeCenter chassis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2.4 zBX blades . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2.5 Power distribution unit (PDU) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.3 zBX entitlements, firmware, and upgrades . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.3.1 zBX management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.3.2 zBX firmware. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.4 zBX connectivity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.4.1 Intranode management network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.4.2 Primary and alternate HMCs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.4.3 Intraensemble data network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.4.4 Network connectivity rules with zBX . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.4.5 Network security considerations with zBX . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.4.6 zBX storage connectivity. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.5 zBX connectivity examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.5.1 A single node ensemble with a zBX . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.5.2 A dual node ensemble with a single zBX . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.5.3 A dual node ensemble with two zBXs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.6 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Chapter 8. Software support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.1 Operating systems summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.2 Support by operating system . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.2.1 z/OS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.2.2 z/VM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.2.3 z/VSE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.2.4 z/TPF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.2.5 Linux on System z. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.2.6 zEC12 functions support summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3 Support by function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3.1 Single system image . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
192 193 194 194 196 197 198 199 200 201 202 202 203 207 207 208 210 211 212 212 213 214 216 218 223 223 225 225 226 227 229 230 233 233 235 237 238 239 240 241 243 244 244 245 245 245 245 245 246 256 257
Contents
ix
8049TOC.fm
8.3.2 zAAP support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3.3 zIIP support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3.4 zAAP on zIIP capability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3.5 Transactional Execution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3.6 Maximum main storage size . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3.7 Flash Express . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3.8 Large page support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3.9 Guest support for execute-extensions facility . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3.10 Hardware decimal floating point . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3.11 Up to 60 logical partitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3.12 Separate LPAR management of PUs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3.13 Dynamic LPAR memory upgrade . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3.14 Capacity Provisioning Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3.15 Dynamic PU add . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3.16 HiperDispatch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3.17 The 63.75 K Subchannels. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3.18 Multiple Subchannel Sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3.19 Third Subchannel Set . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3.20 IPL from an alternate subchannel set . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3.21 MIDAW facility. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3.22 HiperSockets Completion Queue . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3.23 HiperSockets integration with the intraensemble data network (IEDN) . . . . . . 8.3.24 HiperSockets Virtual Switch Bridge. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3.25 HiperSockets Multiple Write Facility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3.26 HiperSockets IPv6 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3.27 HiperSockets Layer 2 support. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3.28 HiperSockets network traffic analyzer for Linux on System z . . . . . . . . . . . . . . 8.3.29 FICON Express8S . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3.30 FICON Express8 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3.31 z/OS discovery and autoconfiguration (zDAC) . . . . . . . . . . . . . . . . . . . . . . . . . 8.3.32 High performance FICON (zHPF) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3.33 Request node identification data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3.34 Extended distance FICON . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3.35 Platform and name server registration in FICON channel . . . . . . . . . . . . . . . . 8.3.36 FICON link incident reporting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3.37 FCP provides increased performance. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3.38 N_Port ID virtualization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3.39 OSA-Express4S 1000BASE-T Ethernet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3.40 OSA-Express4S 10 Gigabit Ethernet LR and SR . . . . . . . . . . . . . . . . . . . . . . . 8.3.41 OSA-Express4S Gigabit Ethernet LX and SX. . . . . . . . . . . . . . . . . . . . . . . . . . 8.3.42 OSA-Express3 10 Gigabit Ethernet LR and SR . . . . . . . . . . . . . . . . . . . . . . . . 8.3.43 OSA-Express3 Gigabit Ethernet LX and SX . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3.44 OSA-Express3 1000BASE-T Ethernet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3.45 Open Systems Adapter for IBM zAware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3.46 Open Systems Adapter for Ensemble. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3.47 Intranode management network (INMN). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3.48 Intraensemble data network (IEDN) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3.49 OSA-Express4S NCP support (OSN) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3.50 Integrated Console Controller . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3.51 VLAN management enhancements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3.52 GARP VLAN Registration Protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3.53 Inbound workload queueing (IWQ) for OSA-Express4S and OSA-Express3 . . 8.3.54 Inbound workload queueing (IWQ) for Enterprise Extender . . . . . . . . . . . . . . . x
258 259 259 260 260 261 262 262 262 263 263 264 264 264 265 265 265 266 266 266 266 267 267 268 268 268 269 269 270 270 271 273 273 273 274 274 274 274 275 276 276 277 278 279 279 280 280 280 281 281 281 282 283
8049TOC.fm
8.3.55 Query and display OSA configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3.56 Link aggregation support for z/VM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3.57 QDIO data connection isolation for z/VM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3.58 QDIO interface isolation for z/OS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3.59 QDIO optimized latency mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3.60 Large send for IPv6 packets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3.61 OSA-Express4S checksum offload . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3.62 Checksum offload for IPv4 packets when in QDIO mode . . . . . . . . . . . . . . . . . 8.3.63 Adapter interruptions for QDIO . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3.64 OSA Dynamic LAN idle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3.65 OSA Layer 3 Virtual MAC for z/OS environments. . . . . . . . . . . . . . . . . . . . . . . 8.3.66 QDIO Diagnostic Synchronization. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3.67 Network Traffic Analyzer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3.68 Program directed re-IPL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3.69 Coupling over InfiniBand . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3.70 Dynamic I/O support for InfiniBand CHPIDs . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.4 Cryptographic support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.4.1 CP Assist for Cryptographic Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.4.2 Crypto Express4S . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.4.3 Crypto Express3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.4.4 Web deliverables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.4.5 z/OS ICSF FMIDs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.4.6 ICSF migration considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.5 z/OS migration considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.5.1 General guidelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.5.2 HCD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.5.3 InfiniBand coupling links . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.5.4 Large page support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.5.5 HiperDispatch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.5.6 Capacity Provisioning Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.5.7 Decimal floating point and z/OS XL C/C++ considerations. . . . . . . . . . . . . . . . . 8.5.8 IBM System z Advanced Workload Analysis Reporter (IBM zAware). . . . . . . . . 8.6 Coupling facility and CFCC considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.7 MIDAW facility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.7.1 MIDAW technical description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.7.2 Extended format data sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.7.3 Performance benefits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.8 IOCP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.9 Worldwide portname (WWPN) tool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.10 ICKDSF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.11 zEnterprise BladeCenter Extension (zBX) Model 003 software support . . . . . . . . . . 8.11.1 IBM Blades . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.11.2 IBM WebSphere DataPower Integration Appliance XI50 for zEnterprise . . . . . 8.12 Software licensing considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.12.1 MLC pricing metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.12.2 Advanced Workload License Charges (AWLC) . . . . . . . . . . . . . . . . . . . . . . . . 8.12.3 System z New Application License Charges (zNALC) . . . . . . . . . . . . . . . . . . . 8.12.4 Select Application License Charges (SALC). . . . . . . . . . . . . . . . . . . . . . . . . . . 8.12.5 Midrange Workload Licence Charges (MWLC). . . . . . . . . . . . . . . . . . . . . . . . . 8.12.6 Parallel Sysplex Licence Charges (PSLC) . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.12.7 System z International Program License Agreement (IPLA). . . . . . . . . . . . . . . 8.13 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
283 283 283 284 284 284 284 285 285 286 286 286 286 287 287 287 288 288 289 289 289 290 294 295 295 295 295 295 296 296 296 297 297 299 299 301 302 302 303 303 304 304 304 305 306 307 307 308 308 308 309 309
Contents
xi
8049TOC.fm
Chapter 9. System upgrades . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.1 Upgrade types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.1.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.1.2 Permanent upgrades . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.1.3 Temporary upgrades. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.2 Concurrent upgrades . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.2.1 Model upgrades . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.2.2 Customer Initiated Upgrade facility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.2.3 Summary of concurrent upgrade functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.3 MES upgrades . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.3.1 MES upgrade for processors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.3.2 MES upgrades for memory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.3.3 MES upgrades for I/O . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.3.4 MES upgrades for the zBX . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.3.5 Summary of Plan-ahead features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.4 Permanent upgrade through the CIU facility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.4.1 Ordering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.4.2 Retrieval and activation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.5 On/Off Capacity on Demand . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.5.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.5.2 Ordering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.5.3 On/Off CoD testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.5.4 Activation and deactivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.5.5 Termination . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.5.6 z/OS capacity provisioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.6 Capacity for Planned Event. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.7 Capacity Backup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.7.1 Ordering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.7.2 CBU activation and deactivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.7.3 Automatic CBU enablement for GDPS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.8 Nondisruptive upgrades . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.8.1 Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.8.2 Concurrent upgrade considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.9 Summary of Capacity on Demand offerings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.10 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Chapter 10. RAS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.1 zEC12 availability characteristics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.2 zEC12 RAS functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.2.1 Scheduled outages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.2.2 Unscheduled outages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.3 zEC12 Enhanced book availability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.3.1 EBA planning considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.3.2 Enhanced book availability processing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.4 zEC12 Enhanced driver maintenance. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.5 RAS capability for the HMC and SE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.6 RAS capability for zBX . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.7 Considerations for PowerHA in zBX environment. . . . . . . . . . . . . . . . . . . . . . . . . . . 10.8 IBM System z Advanced Workload Analysis Reporter (IBM zAware). . . . . . . . . . . . 10.9 RAS capability for Flash Express . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
311 312 312 315 316 316 317 319 322 322 323 325 326 327 328 328 330 331 332 332 333 336 337 338 338 342 344 344 346 347 347 348 349 352 353 355 356 357 358 359 360 360 362 368 369 370 371 373 374
Chapter 11. Environmental requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 375 11.1 zEC12 power and cooling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 376 11.1.1 Power consumption . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 376 xii
8049TOC.fm
11.1.2 Internal Battery Feature . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.1.3 Emergency power-off . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.1.4 Cooling requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.2 IBM zEnterprise EC12 physical specifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.3 IBM zEnterprise EC12 physical planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.3.1 Raised floor or non-raised floor. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.3.2 Top Exit Power . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.3.3 Top Exit I/O Cabling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.3.4 Weight distribution plate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.3.5 3-in-1 bolt down kit for raised floor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.4 zBX environmental requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.4.1 zBX configurations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.4.2 zBX power components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.4.3 zBX cooling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.4.4 zBX physical specifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.5 Energy management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.5.1 Power estimation tool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.5.2 Query maximum potential power . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.5.3 System Activity Display and Monitors Dashboard. . . . . . . . . . . . . . . . . . . . . . . 11.5.4 IBM Systems Director Active Energy Manager . . . . . . . . . . . . . . . . . . . . . . . . . 11.5.5 Unified Resource Manager: Energy Management . . . . . . . . . . . . . . . . . . . . . . Chapter 12. Hardware Management Console and Support Element . . . . . . . . . . . . . 12.1 Introduction to HMC and SE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.1.1 HMC and SE enhancements and changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.1.2 HMC media support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.1.3 Tree Style User Interface and Classic Style User Interface . . . . . . . . . . . . . . . 12.2 HMC and SE connectivity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.2.1 Hardware prerequisites news . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.2.2 TCP/IP Version 6 on HMC and SE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.2.3 Assigning addresses to HMC and SE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.3 Remote Support Facility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.4 HMC and SE remote operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.5 HMC and SE key capabilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.5.1 CPC management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.5.2 LPAR management. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.5.3 Operating system communication. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.5.4 HMC and SE microcode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.5.5 Monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.5.6 Capacity on Demand support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.5.7 Server Time Protocol support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.5.8 NTP client/server support on HMC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.5.9 Security and user ID management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.5.10 System Input/Output Configuration Analyzer on the SE and HMC . . . . . . . . . 12.5.11 Automated operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.5.12 Cryptographic support. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.5.13 z/VM virtual machine management. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.5.14 Installation support for z/VM using the HMC. . . . . . . . . . . . . . . . . . . . . . . . . . 12.6 HMC in an ensemble. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.6.1 Unified Resource Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.6.2 Ensemble definition and management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.6.3 HMC availability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.6.4 Considerations for multiple HMCs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
377 378 378 381 382 382 383 384 385 385 385 385 386 387 389 390 391 391 392 392 393 397 398 399 401 401 401 403 404 404 405 406 406 406 407 407 407 409 412 413 414 417 418 419 419 421 421 422 422 424 426 426
Contents
xiii
8049TOC.fm
12.6.5 HMC browser session to a primary HMC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 426 12.6.6 HMC ensemble topology. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 427 Appendix A. IBM zAware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A.1 Troubleshooting in Complex IT Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A.1.1 Smarter computing needs smarter monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . A.2 Introducing the IBM zAware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A.2.1 Value of IBM zAware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A.2.2 IBM z/OS Solutions to Improve Problem Diagnostics. . . . . . . . . . . . . . . . . . . . . A.3 IBM zAware Technology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A.4 IBM zAware PreRequisites . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A.5 Configuring and using the IBM zAware virtual appliance . . . . . . . . . . . . . . . . . . . . . . 429 430 430 431 431 431 432 438 439
Appendix B. Channel options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 441 Appendix C. Flash Express . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . C.1 Flash Express overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . C.2 Using Flash Express. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . C.3 Security on Flash Express . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . C.3.1 Integrated Key Controller . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . C.3.2 Key Serving Topology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . C.3.3 Error recovery scenarios . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Related publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . IBM Redbooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Other publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Online resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Help from IBM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 445 446 448 451 451 453 454 455 455 455 455 455
Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 457
xiv
8049spec.fm
Notices
This information was developed for products and services offered in the U.S.A. IBM may not offer the products, services, or features discussed in this document in other countries. Consult your local IBM representative for information on the products and services currently available in your area. Any reference to an IBM product, program, or service is not intended to state or imply that only that IBM product, program, or service may be used. Any functionally equivalent product, program, or service that does not infringe any IBM intellectual property right may be used instead. However, it is the user's responsibility to evaluate and verify the operation of any non-IBM product, program, or service. IBM may have patents or pending patent applications covering subject matter described in this document. The furnishing of this document does not grant you any license to these patents. You can send license inquiries, in writing, to: IBM Director of Licensing, IBM Corporation, North Castle Drive, Armonk, NY 10504-1785 U.S.A. IBM's statements regarding its plans, directions, and intent are subject to change or withdrawal without notice at IBM's sole discretion. Information regarding potential future products is intended to outline our general product direction and it should not be relied on in making a purchasing decision. The information mentioned regarding potential future products is not a commitment, promise, or legal obligation to deliver any material, code, or functionality. The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law: INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Some states do not allow disclaimer of express or implied warranties in certain transactions, therefore, this statement may not apply to you. This information could include technical inaccuracies or typographical errors. Changes are periodically made to the information herein; these changes will be incorporated in new editions of the publication. IBM may make improvements and/or changes in the product(s) and/or the program(s) described in this publication at any time without notice. Any references in this information to non-IBM websites are provided for convenience only and do not in any manner serve as an endorsement of those websites. The materials at those websites are not part of the materials for this IBM product and use of those websites is at your own risk. IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you. Any performance data contained herein was determined in a controlled environment. Therefore, the results obtained in other operating environments may vary significantly. Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems. Furthermore, some measurements may have been estimated through extrapolation. Actual results may vary. Users of this document should verify the applicable data for their specific environment. Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products. This information contains examples of data and reports used in daily business operations. To illustrate them as completely as possible, the examples include the names of individuals, companies, brands, and products. All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental. COPYRIGHT LICENSE: This information contains sample application programs in source language, which illustrate programming techniques on various operating platforms. You may copy, modify, and distribute these sample programs in
xv
8049spec.fm
any form without payment to IBM, for the purposes of developing, using, marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written. These examples have not been thoroughly tested under all conditions. IBM, therefore, cannot guarantee or imply reliability, serviceability, or function of these programs.
xvi
8049spec.fm
Trademarks
IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. These and other IBM trademarked terms are marked on their first occurrence in this information with the appropriate symbol ( or ), indicating US registered or common law trademarks owned by IBM at the time this information was published. Such trademarks may also be registered or common law trademarks in other countries. A current list of IBM trademarks is available on the Web at http://www.ibm.com/legal/copytrade.shtml The following terms are trademarks of the International Business Machines Corporation in the United States, other countries, or both:
AIX BladeCenter CICS DataPower DB2 Connect DB2 Distributed Relational Database Architecture Domino DRDA DS8000 ECKD ESCON FICON FlashCopy GDPS Geographically Dispersed Parallel Sysplex HACMP HiperSockets IBM Systems Director Active Energy Manager IBM IMS Language Environment Lotus MQSeries OMEGAMON Parallel Sysplex Passport Advantage Power Systems POWER6 POWER7 PowerHA PowerPC PowerVM POWER PR/SM Processor Resource/Systems Manager RACF Redbooks Redbooks (logo) Resource Link Resource Measurement Facility RETAIN RMF Sysplex Timer System p System Storage System x System z10 System z9 System z Tivoli WebSphere z/Architecture z/OS z/VM z/VSE z10 z9 zEnterprise
The following terms are trademarks of other companies: Intel, Intel logo, Intel Inside logo, and Intel Centrino logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries. Linux is a trademark of Linus Torvalds in the United States, other countries, or both. Microsoft, Windows, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both. Java, and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle and/or its affiliates. UNIX is a registered trademark of The Open Group in the United States and other countries. Other company, product, or service names may be trademarks or service marks of others.
Notices
xvii
8049spec.fm
xviii
8049pref.fm
Preface
The popularity of the Internet and the affordability of IT hardware and software have resulted in an explosion of applications, architectures, and platforms. Workloads have changed. Many applications, including mission-critical ones, are deployed on a variety of platforms, and the System z design has adapted to this change. It takes into account a wide range of factors, including compatibility and investment protection, to match the IT requirements of an enterprise. This IBM Redbooks publication discusses the new IBM zEnterprise System, which consists of the IBM zEnterprise EC12 (zEC12), an updated IBM zEnterprise Unified Resource Manager, and the IBM zEnterprise BladeCenter Extension (zBX) Model 003. The zEC12 is designed with improved scalability, performance, security, resiliency, availability, and virtualization. The superscalar design allows the zEC12 to deliver a record level of capacity over the prior System z servers. It is powered by 120 of the worlds most powerful microprocessors running at 5.5 GHz and is capable of executing more than 75,000 millions of instructions per second (MIPS). The zEC12 Model HA1 is estimated to provide up to 50% more total system capacity than the z196 Model M80. The zBX Model 003 infrastructure works with the zEC12 to enhance System z virtualization and management through an integrated hardware platform that spans mainframe, POWER7, and System x technologies. Through the Unified Resource Manager, the zEnterprise System is managed as a single pool of resources, integrating system and workload management across the environment. This book provides information about the zEnterprise System and its functions, features, and associated software support. Greater detail is offered in areas relevant to technical planning. This book is intended for systems engineers, consultants, planners, and anyone wanting to understand the zEnterprise System functions and plan for their usage. It is not intended as an introduction to mainframes. Readers are expected to be generally familiar with existing IBM System z technology and terminology.
The team who wrote this book

This book was produced by a team of specialists from around the world working at the International Technical Support Organization, Poughkeepsie Center. Octavian Lascu is a Project Leader at International Technical Support Organization, Poughkeepsie Center specialized in Large Systems hardware and High Performance Clustering infrastructure. Ivan Dobos is a Project Leader at International Technical Support Organization, Poughkeepsie Center. Vicente Ranieri is an Executive IT Specialist at STG Advanced Technical Skills (ATS) team supporting System z in Latin America. He has more than 30 years of experience working for IBM. Ranieri is a member of the zChampions team, a worldwide IBM team participating in the creation of System z technical roadmap and value proposition materials. Besides co-authoring several Redbooks publications, he has been an ITSO guest speaker since 2001, teaching the System z security update workshops worldwide. Vicente also presents in several
xix
8049pref.fm
IBM internal and external conferences. His areas of expertise include System z security, Parallel Sysplex, System z hardware, and z/OS. Vicente Ranieri is certified as a Distinguished IT Specialist by the Open group. Vicente is a member of the Technology Leadership Council Brazil (TLC-BR), and he is also a member of the IBM Academy of Technology. Gerard Laumay is a System z IT Specialist in the IBM Products and Solutions Support Centre (PSSC) in Montpellier, France. Part of the System z New Technology Introduction (NTI) team managing System z Early Support Programs (ESP) in Europe IOT, he has more than 26 years of experience in the large systems field, as a consultant with IBMs customers. His areas of expertise include IBM System z hardware, operating systems (z/OS, z/VM and Linux for System z), IBM middleware, new workloads and solutions on System z platform. Member of the zChampion worldwide technical community, he teaches at numerous IBM external and internal conferences. Gerard participates to international projects, and he has co-authored several IBM Redbook publications. Parwez Hamid is an Executive IT Consultant with the IBM Server and Technology Group. Over the past 38 years he has worked in various IT roles within IBM. Since 1988 he has worked with a large number of IBM mainframe customers, mainly introducing new technology. Currently, he provides pre-sales technical support for the IBM System z product portfolio and is the lead System z technical specialist for the UK and Ireland. Parwez has co-authored a number of ITSO Redbooks publications and he prepares technical material for the worldwide announcement of System z servers. He is the technical lead for the IBM System z zChampions team in IBM Europe. This team is made up of key IBM System z client-facing technical professionals from the Server and Software groups. Parwez works closely with System z product development in Poughkeepsie, and provides input and feedback for future product plans. Parwez is a Technical Staff member of the IBM UK Technical Council composed of senior technical specialists representing all IBM Client, Consulting, Services. and Product groups. Parwez teaches and presents at numerous IBM user group and internal conferences. Swee Seng Ng is a senior IT specialist working as a Technical Sales Support for IBM Singapore. Fernando Nogal is an IBM Certified Consulting IT Specialist working as an STG Technical Consultant for the Spain, Portugal, Greece, and Israel IMT. He specializes in advanced infrastructures and architectures. In his 30+ years with IBM, he has held a variety of technical positions, mainly providing support for mainframe clients. Previously, he was on assignment to the Europe Middle East and Africa (EMEA) Ssytem z Technical Support group, working full-time on complex solutions for e-business. His job included, and still does, presenting and consulting in architectures and infrastructures, and providing strategic guidance to System z clients regarding the establishment and enablement of advanced technologies on System z, including the z/OS, z/VM, and Linux environments. He is a zChampion and a member of the System z Business Leaders Council. An accomplished writer, he has authored and co-authored over 28 IBM Redbooks publications and several technical papers. Other activities include serving as a University Ambassador. He travels extensively on direct client engagements and as a speaker at IBM and client events and trade shows. Frank Packheiser is a Senior zIT Specialist at the Field Technical Sales Support office in Germany. He has 20 years of experience in zEnterprise, System z, zSeries, and predecessor mainframe servers. He has worked for 10 years for the IBM education center in Germany, developing and providing professional training. He also provides professional services to System z and mainframe clients. In the years 2008 and 2009 he supported clients in Middle East / North Africa (MENA) as a zIT Architect. Besides co-authoring several Redbooks publications since 1999, he has been an ITSO guest speaker on ITSO workshops for the last two years. xx
8049pref.fm
Karan Singh is a Project Leader at International Technical Support Organization, Global Content Services, Poughkeepsie Center. Andr Spahni is a Senior System Service Representative working for IBM Global Technology Services in Switzerland. He has 10 years of experience working with and supporting System z customers. Andr has been working for the Technical Support Competence Center (TSCC) Hardware FE System z for Switzerland, Germany, and Austria since 2008. His areas of xpertise include System z hardware, Parallel Sysplex, and connectivity. Esra Ufacik is a System z Client Technical Specialist in Systems and Technology Group. She holds a B.Sc. degree in Electronics and Telecommunication Engineering from Istanbul Technical University. Her IT career started with a Turkish Bank as a z/OS systems programmer. Her responsibilities included maintaining a parallel sysplex environment with DB2 data sharing; planning and executing hardware migrations; installing new operating system releases, middleware releases, and system software maintenance; and generating performance reports for capacity planning. Esra joined IBM in 2007 as a Software Support Specialist in Integrated Technology Services, where she work with mainframe customers. Acting as their account advocate, Esra was assigned to a Software Support Team Leader position in ITS and became involved in several projects. Since 2010 she has been with STG, where her role covers pre-sales technical consultancy, conducting System z competitive assessment studies, presenting the value of System z to various audiences, and assuring technical feasibility of proposed solutions. Esra is also a guest lecturer for "System z and large scale computing" classes, which are given to undergraduate Computer Engineering students within the scope of Academic Initiative. Hans Wijngaard is a Client Technical Specialist, who has been with IBM for over 26 years. He started as a hardware Customer Engineer, but worked most of the time as a z/OS systems programmer at IBM client sites. For more than 20 years he gained a broad experience in z/OS performance and tuning, as well as installing, migrating, implementing, and customzing a variety of both IBM- and non-IBM software. The past four years Hans has been working in the System z hardware area, as a Client Technical Specialist, and he is currently supporting IBM Sales Representatives, IBM Business Partners, and clients in various System z business engagements. Wolfgang Fries is a Senior Consultant in the System z HW Support Center in Germany. He spent several years at the European Support Center in Montpellier, France, providing international support for System z servers. Wolfgang has 35 years of experience in supporting large System z customers. His area of expertise include System z servers and connectivity. Zhaoxu Zhang is a Senior System Service Representative at the IBM Global Technology Services in Beijing, China. He joined IBM in 1998 to support and maintain System z products for clients throughout China. Zhaoxu has been working in the Technical Support Group (TSG) providing second level support to System z clients since 2006. His areas of expertise include System z hardware, Parallel Sysplex, and FICON connectivity. Thanks to the following people for their contributions to this project: This project has been coordinated by Bill White, ITSO Global Content Services System z Portfolio Manager, Poughkeepsie Center.
Preface
xxi
8049pref.fm
Now you can become a published author, too!

Heres an opportunity to spotlight your skills, grow your career, and become a published authorall at the same time! Join an ITSO residency project and help write a book in your area of expertise, while honing your experience using leading-edge technologies. Your efforts will help to increase product acceptance and customer satisfaction, as you expand your network of technical contacts and relationships. Residencies run from two to six weeks in length, and you can participate either in person or as a remote resident working from your home base. Find out more about the residency program, browse the residency index, and apply online at: ibm.com/redbooks/residencies.html
Comments welcome
Your comments are important to us! We want our books to be as helpful as possible. Send us your comments about this book or other IBM Redbooks publications in one of the following ways: Use the online Contact us review Redbooks form found at: ibm.com/redbooks Send your comments in an email to: redbooks@us.ibm.com Mail your comments to: IBM Corporation, International Technical Support Organization Dept. HYTD Mail Station P099 2455 South Road Poughkeepsie, NY 12601-5400
Stay connected to IBM Redbooks

Find us on Facebook: http://www.facebook.com/IBMRedbooks Follow us on Twitter: http://twitter.com/ibmredbooks Look for us on LinkedIn: http://www.linkedin.com/groups?home=&gid=2130806 Explore new Redbooks publications, residencies, and workshops with the IBM Redbooks weekly newsletter: https://www.redbooks.ibm.com/Redbooks.nsf/subscribe?OpenForm Stay current on recent Redbooks publications with RSS Feeds: http://www.redbooks.ibm.com/rss.html
xxii
8049ch01.fm
Chapter 1.
Introducing the IBM zEnterprise EC12

The IBM zEnterprise EC12 (zEC12) builds upon the strengths of its predecessor, the IBM zEnterprise 196. It was designed to help overcome problems in today's IT infrastructures and provide a foundation for the future. The zEC12 continues the evolution of integrated hybrid systems, introducing the zEnterprise BladeCenter Extension (zBX) Model 003, and an updated zEnterprise Unified Resource Manager. The zEC12 has a redesigned zEnterprise chip. It is the first six-core chip in mainframe history and operates at an industry leading, high frequency, 5.5 GHz. The zEC12 is a scalable symmetric multiprocessor (SMP) that can be configured with up to 101 processors running concurrent production tasks with up to 3 TB of memory. It introduces several PCIe I/O features, such as exploitation of Storage Class Memory through the Flash Express feature, and technologies, such as the IBM System z Advanced Workload Analysis Reporter (IBM zAware), an appliance with cutting edge pattern recognition analytics that uses heuristic techniques and represents the next generation of system health monitoring. The zEC12 goes beyond previous designs while continuing to enhance the traditional mainframe qualities, delivering unprecedented performance and capacity growth. The zEC12 has a well-balanced general-purpose design that allows it to be equally at ease with compute-intensive and I/O-intensive workloads.
8049ch01.fm
1.1 IBM zEnterprise EC12 elements

zEC12 continues the integration with heterogeneous platforms, based on IBMs BladeCenter technology. The zEC12 introduces the IBM zEnterprise BladeCenter Extension (zBX) Model 003. Similarly to its predecessor, the zBX Model 002, the zBX Model 003 houses up to 112 general purpose POWER7 and System x blades as well as specialized solutions, such as the IBM WebSphere DataPower XI50 for zEnterprise. The other key element is the zEnterprise Unified Resource Manager firmware, which has updated hypervisor firmware. The zEC12, with or without a zBX attached, but managed by the Unified Resource Manager, constitutes a node in an ensemble. An ensemble is a collection of up to eight highly virtualized heterogeneous zEC12 or zEnterprise nodes, with dedicated networks for system management and data transfer across the virtualized system images, managed as a single logical entity by the Unified Resource Manager functions, where diverse workloads can be deployed. Figure 1-1 shows the elements of the IBM zEnterprise EC12.
zEnterprise EC12
zEnterprise BladeCenter Extension
Hypervisors
Energy
Operations
Performance
Networks
Virtual Servers
Figure 1-1 Elements of the zEC12 ensemble node
Workloads continue to change. Multi-tier application architectures and their deployment on heterogeneous infrastructures are common today. But what is uncommon is the infrastructure setup needed to provide the high qualities of service required by mission critical applications 2
8049ch01.fm
Creating and maintaining these high-level qualities of service while using a large collection of distributed components takes a great amount of knowledge and effort. It implies acquiring and installing extra equipment and software to ensure availability and security, monitoring, and managing. Additional manpower is required to configure, administer, troubleshoot, and tune such a complex set of separate and diverse environments. Due to platform functional differences, the resulting infrastructure will not be uniform, regarding those qualities of service or serviceability. The zEC12 lies on an evolutionary path that directly addresses those infrastructure problems. The zEC12 design can simultaneously support a large number of diverse workloads while providing the highest qualities of service. The IBM holistic approach to System z design includes hardware, software, and procedures. It takes into account a wide range of factors, including compatibility and investment protection, thus ensuring a tighter fit with the IT requirements of the entire enterprise.
1.2 zEC12 highlights

This section reviews some of the most important features and function of zEC12.
Processor and memory

IBM continues its technology leadership with the zEC12. The zEC12 is built using IBMs modular multi-book design that supports one to four books per central processor complex (CPC). Each book contains a Multi-Chip Module (MCM), which hosts the redesigned CMOS 13S1 processor units, storage control chips, and connectors for I/O. The superscalar processor has enhanced out-of-order instruction execution, redesigned caches, and an expanded instruction set which includes a Transactional Execution facility, for better performance. Depending on the model, the zEC12 can support from a minimum of 32 GB to a maximum of 3040 GB of useable memory, with up to 768 GB per book. In addition, a fixed amount of 32 GB is reserved for the Hardware System Area (HSA) and is not part of customer-purchased memory. Memory is implemented as a Redundant Array of Independent Memory (RAIM). To exploit the RAIM function, up to 960 GB are installed per book, for a system total of 3840 GB.
Capacity and performance

The zEC12 provides increased capacity over its predecessor, the z196 system. This capacity is achieved both by increasing the performance of the individual processor units and by increasing the number of processor units (PUs) per system. The increased performance and the total system capacity available, along with possible energy savings, offer the opportunity to consolidate diverse applications on a single platform, with real financial savings. The introduction of new technologies and features help to ensure that the zEC12 is an innovative, security-rich platform, designed to maximize resource exploitation and utilization, and with the ability to integrate applications and data across the enterprise IT infrastructure. zEC12 has five model offerings ranging from one to 101 configurable processor units (PUs). The first four models (H20, H43, H66, and H89) have 27 PUs per book, and the high capacity model (the HA1) has four 30 PU books. Model HA1 is estimated to provide up to 50% more total system capacity than the z196 Model M80, keeping the same memory and power requirements. This comparison is based on the Large Systems Performance Reference (LSPR) mixed workload analysis.
1
CMOS 13S is a 32 nanometer CMOS logic fabrication process.
Chapter 1. Introducing the IBM zEnterprise EC12
8049ch01.fm
The zEC12 expands the subcapacity settings offer with three subcapacity levels for the first 20 processors, giving a total of 161 distinct capacity settings in the system, and providing for a range of over 1:320 in processing power. The zEC12 delivers scalability and granularity to meet the needs of medium-sized enterprises, while also satisfying the requirements of large enterprises having very demanding, mission-critical transaction and data processing requirements. The zEC12 continues to offer all the specialty engines available on previous System z systems.
I/O subsystem and I/O features

The zEC12 supports a PCIe I/O and InfiniBand infrastructure. PCIe features are installed in PCIe I/O drawers. In addition, when carried forward on an upgrade from a z196 or z10 EC, up to two I/O drawers, which were introduced with the IBM z10 BC, and one traditional I/O cage are also supported. There are up to 48 high-performance fanouts for data communications between the server and the peripheral environment. The multiple channel subsystems (CSS) architecture allows up to four CSSs, each with 256 channels. For I/O constraint relief, three subchannel sets, are available per CSS, allowing access to a greater number of logical volumes. for improved device connectivity for Parallel Access Volumes (PAVs), PPRC secondaries, and FlashCopy devices; this third subchannel set allows the user to extend the amount of addressable external storage. The zEC12 allows to IPL from subchannel set 1 (SS1) or subchannel set 2 (SS2), in addition to subchannel set 0. In addition, the system I/O buses take advantage of the PCIe technology and of InfiniBand technology, which is also exploited in coupling links.
Virtualization
Processor Resource/Systems Manager (PR/SM) manages all the installed and enabled resources (processors and memory) as a single large SMP system. It enables the configuration and operation of up to 60 logical partitions, which have processors, memory, and I/O resources assigned from the installed books. zEC12 provides improvements to the PR/SM HiperDispatch function. HiperDispatch provides work alignment to logical processors, and alignment of logical processors to physical processors. This alignment optimizes cache utilization, minimizes inter-book communication, and optimizes z/OS work dispatching, with the end result of increasing throughput. zEC12 provides for the definition of up to 32 HiperSockets. HiperSockets provide for memory communication across logical partitions, without the need of any I/O adapters, with VLAN capability. HiperSockets have been extended to bridge to an ensemble internode data network.
Increased flexibility with z/VM-mode logical partition

The zEC12 provides for the definition of a z/VM-mode logical partition (LPAR) containing a mix of processor types including CPs and specialty processors such as IFLs, zIIPs, zAAPs, and ICFs. z/VM V5R4 and later support this capability that increases flexibility and simplifies systems management. In a single LPAR, z/VM can manage guests that exploit Linux on System z on IFLs, z/VSE, z/TPF, and z/OS on CPs, execute designated z/OS workloads, such as parts of DB2 DRDA processing and XML, on zIIPs, and provide an economical Java execution environment under z/OS on zAAPs.
8049ch01.fm
zAware-mode logical partition

The zEC12 introduces the zAware mode logical partition. Either CPs or IFLs can be configured to the partition. The partition can be used exclusively by the IBM System z Advanced Workload Analysis Reporter (IBM zAware) offering.
Reliability, availability, and serviceability

System reliability, availability, and serviceability (RAS) are areas of continuous IBM focus. The objective is to reduce, if possible eliminate, all sources of planned and unplanned outages, with the objective of keeping the system running. It is a design objective to provide higher availability with a focus on reducing outages. With a properly configured zEC12, further reduction of outages can be attained through improved nondisruptive replace, repair, and upgrade functions for memory, books, and I/O adapters, as well as extending nondisruptive capability to download Licensed Internal Code (LIC) updates. Enhancements include removing pre-planning requirements with the fixed 32 GB HSA. Customers will no longer need to worry about using their purchased memory when defining their I/O configurations with reserved capacity or new I/O features. Maximums can be configured and IPLed so that insertion at a later time can be dynamic and not require a power on reset of the server. This approach provides many high-availability and nondisruptive operations capabilities that differentiate it in the marketplace. The ability to cluster multiple systems in a Parallel Sysplex takes the commercial strengths of the z/OS platform to higher levels of system management, competitive price/performance, scalable growth, and continuous availability.
1.2.1 Flash Express

Flash Express is an innovative optional feature introduced with the zEC12, with the objective of driving performance improvements and better availability for critical business workloads that cannot afford any hits to service levels. Flash Express is easy to configure, requires no special skills and provides rapid time to value. Flash Express implements Storage Class Memory (SCM) in a PCIe card form factor. Each Flash Express card implements internal NAND Flash solid state disk (SSD) and has a capacity of 1.6 TB of usable storage. Cards are installed in pairs providing mirrored data to ensure a high level of availability and redundancy. A maximum of four pairs of cards can be installed on a zEC12, for a maximum capacity of 6.4 TB of storage. Flash Express The Flash Express feature is designed to allow each logical partition to be configured with its own SCM address space. It is designed to be used for paging - 1 MB large pages can become pageable. Hardware encryption is included to improve data security. Data security is insured through a unique key stored on the Support Element (SE) hard drive. It is mirrored for redundancy. Data on the Flash Express feature is protected with this key and is only usable on the system with the key that encrypted it. The Secure Key Store is implemented using a smart card installed in the Support Element. The smart card (one pair - one for each SE) contains both a unique key personalized for each machine, and a small cryptographic engine that can perform a limited set of security functions within the smart card. Flash Express is supported by z/OS 1.13 minimum for handling z/OS paging activity and SVC dumps. Additional functions of Flash Express is expected to be supported later, including 2 GB large page and dynamic reconfiguration for Flash Express. IBM System z Advanced Workload Analysis Reporter
8049ch01.fm
IBM System z Advanced Workload Analysis Reporter (IBM zAware) is a feature introduced with the zEC12 which embodies the next generation of system monitoring. IBM zAware is designed to offer a near real-time, continuous learning, diagnostics and monitoring capability, to help pinpoint and resolve potential problems quickly enough to minimize business impacts. The ability to tolerate service disruptions is diminishing. In a 24 x 7 environment any disruption can have grave consequences, special when it last days, even hours. But increased system complexity makes it more probable that errors occur, and those errors are, too, increasingly complex. Some incidents early symptoms go undetected for long periods of time and can grow to large problems. Systems often experience soft failures (sick but not dead) which are much more difficult or unusual to detect. IBM zAware is designed to help in those circumstances. For more information see Appendix A, IBM zAware on page 429.
1.3 zEC12 technical overview

This section briefly reviews the major elements of zEC12.
1.3.1 Models
The zEC12 has a machine type of 2827. Five models are offered: H20, H43, H66, H89, and HA1. The model name indicates the maximum number of PUs available for purchase (withA1 standing for 101). A PU is the generic term for the z/Architecture processor on the Multi-Chip Module (MCM) that can be characterized as any of the following items: Central Processor (CP). Internal Coupling Facility (ICF) to be used by the Coupling Facility Control Code (CFCC). Integrated Facility for Linux (IFL) Additional System Assist Processor (SAP) to be used by the channel subsystem. System z Application Assist Processor (zAAP). One CP must be installed with or prior to the installation of any zAAPs. System z Integrated Information Processor (zIIP). One CP must be installed with or prior to installation of any zIIPs. In the five-model structure, only one CP, ICF, or IFL must be purchased and activated for any model. PUs can be purchased in single PU increments and are orderable by feature code. The total number of PUs purchased cannot exceed the total number available for that model. The number of installed zAAPs cannot exceed the number of installed CPs. The number of installed zIIPs cannot exceed the number of installed CPs. The multi-book system design provides an opportunity to concurrently increase the capacity of the system in three ways: Add capacity by concurrently activating more CPs, IFLs, ICFs, zAAPs, or zIIPs on an existing book. Add a new book concurrently and activate more CPs, IFLs, ICFs, zAAPs, or zIIPs. Add a new book to provide additional memory or one or more adapters to support a greater number of I/O features.
1.3.2 Model upgrade paths

Any zEC12 can be upgraded to another zEC12 hardware model. Upgrading from models H20, H43, H66, and H89 to HA1 is disruptive (that is, the machine is unavailable during this 6
8049ch01.fm
upgrade). Any z196 or z10 EC model can be upgraded to any zEC12 model. Figure 1-2 presents a diagram of the upgrade path. Note: An air-cooled zEC12 cannot be converted to an water cooled zEC12 or vice-versa.
z196 Upgrade to zEC12

When a z196 is upgraded to a zEC12, the Driver level has to be at least 93. If a zBX is involved, it must be at Bundle 27 or higher. Also, when upgrading a z196 that controls a zBX Model 002 to a zEC12, the zBX will be upgraded to a Model 003. That upgrade is disruptive, and proper planning should be in place.
Not offered
Downgrades within the zEC12 models Upgrade from a z114 with zBX to zEC12 Removal of a zBX without a controlling system Upgrades from IBM System z9 or earlier systems zBX Model 002 attachment to zEC12
z196 zBX Model 002
z10 EC
zEC12
zBX z196 Model 003
HA1 H89 H66 H43 H20

z 10 EC
Con curren t Upgrade
Figure 1-2 zEC12 upgrades
1.3.3 Frames
The zEC12 has two frames, bolted together, and known as the A frame and the Z frame. The frames contain the CPC components, including these: The processor cage, with up to four books PCIe I/O drawers, I/O drawers, and I/O cage Power supplies An optional internal battery feature (IBF) Cooling units for either air or water cooling Support elements
8049ch01.fm
1.3.4 Processor cage

The processor cage houses up to four processor books. Each book houses a MCM, memory, and I/O interconnects.
MCM technology
The zEC12 is built on a proven superscalar microprocessor architecture of its predecessor and provides several enhancements over the z196. In each book there is one MCM. The MCM has six PU chips and two SC chips. The PU chip has six cores, with four, five or six active cores, which can be characterized as CPs, IFLs, ICFs, zIIPs, zAAPs, or SAPs. Two MCM sizes are offered, which are 27 or 30 cores. The MCM provides a significant increase in system scalability and an additional opportunity for server consolidation. All books are interconnected with very high-speed internal communication links, in a full star topology through the L4 cache, which allows the system to be operated and controlled by the PR/SM facility as a memory- and cache-coherent symmetric multiprocessor (SMP). The PU configuration is made up of two spare PUs per CPC and a variable number of system assist processors (SAPs), which scale with the number of books installed in the server, such as three SAPs with one book installed and up to 16 when four books are installed. In addition, one PU is reserved and is not available for customer use. The remaining PUs can be characterized as central processors (CPs), Integrated Facility for Linux (IFL) processors, System z Application Assist Processors (zAAPs), System z Integrated Information Processors (zIIPs), Internal Coupling Facility (ICF) processors, or additional SAPs. The zEC12 offers a water cooling option for increased system and data center energy efficiency. In a a water cooled system, the MCM is cooled by a cold plate connected to the internal water cooling loop. In a air-cooled system, radiator units (RU) exchange the heat from internal water loop with air, and air backup. Both cooling options are fully redundant.
Processor features
The processor chip has a six-core design, with either four, five or six active cores, and operates at 5.5 GHz. Depending on the MCM version (27 PU or 30 PU), from 27 to 120 PUs are available, on one to four books. Each core on the PU chip includes a dedicated coprocessor for data compression and cryptographic functions, such as the CP Assist for Cryptographic Function (CPACF). This is an improvement over z196, where two cores shared a coprocessor. Hardware data compression can play a significant role in improving performance and saving costs over doing compression in software. Having standard clear key cryptographic coprocessors integrated with the processor provides high-speed cryptography for protecting data. Each core has its own hardware decimal floating point unit designed according to a standardized, open algorithm. Much of today's commercial computing is decimal floating point, so on-core hardware decimal floating point meets the requirements of business and user applications, and provides improved performance, precision, and function. In the unlikely case of a permanent core failure, each core can be individually replaced by one of the available spares. Core sparing is transparent to the operating system and applications.
Transactional Execution facility

The z/Architecture has been expanded with Transactional Execution facility. This set of instructions allows defining groups of instructions which should be executed atomically, that is, either all the results are committed or none is. The facility provides for faster and more
8049ch01.fm
scalable multi-threaded execution, and is known in the academia as hardware transactional memory.
Out-of-order execution
The zEC12 has a superscalar microprocessor with out-of-order (OOO) execution to achieve faster throughput. With OOO, instructions might not execute in the original program order, although results are presented in the original order. For instance, OOO allows a few instructions to complete while another instruction is waiting. Per machine cycle, up to three instructions can be decoded and up to seven instructions can be in execution.
Concurrent processor unit conversions

The zEC12 supports concurrent conversion between various PU types, providing flexibility to meet changing business environments. CPs, IFLs, zAAPs, zIIPs, ICFs, or optional SAPs can be converted to CPs, IFLs, zAAPs, zIIPs, ICFs, or optional SAPs
Memory subsystem and topology

zEC12 uses the buffered DIMM design that has been developed for the z196. For this purpose, IBM has developed a chip that controls communication with the PU and drives address and control from DIMM to DIMM. The DIMM capacities are 4, 16, and 32 GB. Memory topology provides the following benefits: Redundant array of independent memory (RAIM) for protection at the DRAM, DIMM, and memory channel level Maximum of 3.0 TB of user configurable memory with a maximum of 3840 GB of physical memory (with a maximum of 1 TB configurable to a single logical partition) One memory port for each PU chip; up to three independent memory ports per book Increased bandwidth between memory and I/O Asymmetrical memory size and DRAM technology across books Large memory pages (1 MB and 2GB) Key storage Storage protection key array kept in physical memory Storage protection (memory) key is also kept in every L2 and L3 cache directory entry Large (32 GB) fixed-size HSA eliminates having to plan for HSA
PCIe fanout hot-plug

The PCIe fanout provides the path for data between memory and the PCIe I/O cards using the PCIe 8GBps bus. The PCIe fanout is hot-pluggable. In the event of an outage, a redundant I/O interconnect allows a PCIe fanout to be concurrently repaired without loss of access to its associated I/O domains. Up to eight PCIe fanouts are available per book.
Host channel adapter fanout hot-plug

The host channel adapter fanout provides the path for data between memory and the I/O cards using InfiniBand (IFB) cables. The HCA fanout is hot-pluggable. In the event of an outage, a redundant I/O interconnect allows an HCA fanout to be concurrently repaired without loss of access to its associated I/O cards. Up to eight HCA fanouts are available per book.
1.3.5 I/O connectivity, PCIe, and InfiniBand

The zEC12 offers various improved features and exploits technologies such as PCIe, InfiniBand, and Ethernet. This section briefly reviews the most relevant I/O capabilities.
8049ch01.fm
The zEC12 takes advantage of PCIe Generation 2 to implement the following features: An I/O bus, which implements the PCIe infrastructure. This is a preferred infrastructure and can be used alongside InfiniBand. PCIe fanouts, which provide 8 GBps connections to the PCIe I/O features. The zEC12 takes advantage of InfiniBand to implement the following features: An I/O bus, which includes the InfiniBand infrastructure. This replaces the self-timed interconnect bus found in System z systems prior to z9. Parallel Sysplex coupling links using InfiniBand (IFB): 12x InfiniBand coupling links for local connections and 1x InfiniBand coupling links for extended distance connections between any two of: zEnterprise CPCs, and z10 CPCs. The 12x IB link has a bandwidth of 6 GBps. Host Channel Adapters for InfiniBand (HCA3) can deliver up to 40% faster coupling link service times than HCA2.
1.3.6 I/O subsystems

The zEC12 I/O subsystem is similar to the one on z196 and includes a PCIe infrastructure. The I/O subsystem is supported by both a PCIe bus and an I/O bus similar to that of z196, and includes the InfiniBand infrastructure (which replaced the self-timed interconnect found in the prior System z systems). This infrastructure is designed to reduce overhead and latency, and provide increased throughput. The I/O expansion network uses the InfiniBand Link Layer (IB-2, Double Data Rate). zEC12 also offers three I/O infrastructure elements for holding the I/O features cards: PCIe I/O drawers, for PCIe cards; and I/O drawers and I/O cages, for non-PCIe cards.
PCIe I/O drawer

The PCIe I/O drawer, together with the PCIe I/O features, offers improved granularity and capacity over previous I/O infrastructures, and can be concurrently added and removed in the field, easing planning. Two PCIe I/O drawers occupy the same space as an I/O cage, yet each offers 32 I/O card slots, a 14% increase in capacity. Only PCIe cards (features) are supported, in any combination. Up to five PCIe I/O drawers per CPC are supported.
I/O drawer
On the zEC12 I/O drawers are only supported when carried forward on upgrades from z196 or z10. The zEC12 can have up to two I/O drawers in the Z frame. I/O drawers can accommodate up to eight I/O features in any combination. Based on the amount of legacy I/O features carried forward, the configurator will determine the number of required I/O drawers.
I/O cage
On the zEC12 a maximum of one I/O cage is supported and is only available on upgrades from z196 or z10 to zEC12. The I/O cage in housed in the A frame. The I/O cage can accommodate up to 28 I/O features in any combination. Based on the amount of legacy I/O features carried forward, the configurator will determine the required number of I/O drawers and I/O cages.
10
8049ch01.fm
I/O features
The zEC12 supports the following PCIe features, which can be installed only in the PCIe I/O drawers: FICON Express8S SX and 10 KM LX (Fibre Channel connection) OSA-Express4S 10 GbE LR and SR, GbE LX and SX, 1000BASE-T Crypto Express4S Flash Express When carried-forward on an upgrade, the zEC12 also supports up to one I/O cage and up to two I/O drawers on which the following I/O features can be installed: FICON Express8 FICON Express4 10 KM LX and SX (four port cards only) OSA-Express3 10 GbE LR and SR OSA-Express3 GbE LX and SX OSA-Express3 1000BASE-T Crypto Express3 ISC-3 coupling links (peer-mode only) In addition, InfiniBand coupling links, which attach directly to the processor books, are supported.
FICON channels
Up to 160 features with up to 320 FICON Express8S channels are supported. The FICON Express8S features support a link data rate of 2, 4, or 8 Gbps. Up to 44 features with up to 176 FICON Express8 or FICON Express4 channels are supported: The FICON Express8 features support a link data rate of 2, 4, or 8 Gbps. The FICON Express4 features support a link data rate of 1, 2, or 4 Gbps. The zEC12 FICON features support the following protocols: FICON (FC) and High Performance FICON for System z (zHPF). zHPF offers improved access to data, of special importance to OLTP applications Channel-to-channel (CTC) Fibre Channel Protocol (FCP). FICON also offers the following capabilities: Modified Indirect Data Address Word (MIDAW) facility: provides more capacity over native FICON channels for programs that process data sets, which exploit striping and compression (such as DB2, VSAM, PDSE, HFS, and zFS) by reducing channel, director, and control unit overhead. Enhanced problem determination, analysis, and manageability of the storage area network (SAN) by providing registration information to the fabric name server for both FICON and FCP
Open Systems Adapter

The zEC12 allows any mix of the supported Open Systems Adapter (OSA) Ethernet features, up to 48 OSA-Express4S features with a maximum of 96 ports, and up to 24 OSA Express3 features with a maximum of 96 ports. OSA Express3 features are plugged into an I/O drawer or I/O cage, and OSA Express4S features are plugged into the PCIe I/O drawer.
11
8049ch01.fm
The maximum number of combined OSA Express3 and OSA Expres4S features can not exceed 48.
OSM and OSX CHPID types

The zEC12 provides OSA-Express4S and OSA-Express3 CHPID types OSM and OSX for zBX connections: OSA-Express for Unified Resource Manager (OSM): Connectivity to the intranode management network (INMN). Connects the zEC12 to the zBX via the Bulk Power Hubs (BPHs) for use of the Unified Resource Manager functions in the HMC. Uses OSA-Express3 1000BASE-T or OSA Express4S 1000BASE-T Ethernet exclusively. OSA-Express for zBX (OSX): Connectivity to the intraensemble data network (IEDN). Provides a data connection from the zEC12 to the zBX by using the OSA-Express4S 10 GbE or OSA-Express3 10 GbE feature.
OSA-Express4S and OSA-Express3 feature highlights

The zEC12 supports five OSA-Express4S and five OSA-Express3 features. OSA-Express4S and OSA-Express3 features provide the important benefits for TCP/IP traffic, namely reduced latency and improved throughput for standard and jumbo frames Performance enhancements are the result of the data router function present in all OSA-Express4S and OSA-Express3 features. What previously was performed in firmware, the OSA-Express4S and OSA-Express3 perform in hardware. Additional logic in the IBM ASIC handles packet construction, inspection, and routing, thereby allowing packets to flow between host memory and the LAN at line speed without firmware intervention. With the data router, the store and forward technique in direct memory access (DMA) is no longer used. The data router enables a direct host memory-to-LAN flow. This avoids a hop and is designed to reduce latency and to increase throughput for standard frames (1492 byte) and jumbo frames (8992 byte). For more information about the OSA features, see 4.9, Connectivity on page 146.
HiperSockets
The HiperSockets function, also known as internal queued direct input/output (internal QDIO or iQDIO), is an integrated function of the zEC12 that provides users with attachments to up to 32 high-speed virtual LANs with minimal system and network overhead. HiperSockets can be customized to accommodate varying traffic sizes. Because HiperSockets does not use an external network, it can free up system and network resources, eliminating attachment costs while improving availability and performance. For communications between logical partitions in the same zEC12 server, HiperSockets eliminates the need to use I/O subsystem features and to traverse an external network. Connection to HiperSockets offers significant value in server consolidation by connecting many virtual servers, and can be used instead of certain coupling link configurations in a Parallel Sysplex.
1.3.7 Cryptography
Integrated cryptographic features provide leading cryptographic performance and functionality. Reliability, availability, and serviceability (RAS) support is unmatched in the industry and the cryptographic solution has received the highest standardized security certification (FIPS 140-2 Level 42). The crypto cards were enhanced with additional
2
Federal Information Processing Standards (FIPS)140-2 Security Requirements for Cryptographic Modules
12
8049ch01.fm
capabilities to dynamically add or move crypto coprocessors to logical partitions without pre-planning. The zEC12 implements the PKCS #11, one of the industry-accepted standards called Public Key Cryptographic Standards (PKCS) provided by RSA Laboratories of RSA Security Inc., and the IBM Common Cryptographic Architecture (CCA) in its cryptographic features.
CP Assist for Cryptographic Function

The CP Assist for Cryptographic Function (CPACF) offers the full complement of the Advanced Encryption Standard (AES) algorithm and Secure Hash Algorithm (SHA) along with the Data Encryption Standard (DES) algorithm. Support for CPACF is available through a group of instructions known as the Message-Security Assist (MSA). z/OS Integrated Cryptographic Service Facility (ICSF) callable services as well as in-kernel crypto APIs and libica3 cryptographic functions library running on Linux on System z also invoke CPACF functions. ICSF is a base element of z/OS, and can transparently use the available cryptographic functions, CPACF or PCIe cryptographic features, to balance the workload and help address the bandwidth requirements of your applications. CPACF must be explicitly enabled, using a no-charge enablement feature (FC 3863), except for the Secure Hash Algorithms (SHA), which are shipped enabled with each server. The enhancements to CPACF are exclusive to the zEnterprise CPCs and they are supported by z/OS, z/VM, z/VSE, z/TPF, and Linux on System z.
Configurable Crypto Express4S feature

The Crypto Express4S represents the newest generation of cryptographic feature designed to complement the cryptographic capabilities of the CP Assist for Cryptographic Function (CPACF). It is an optional feature exclusive to zEC12. The Crypto Express4S feature has been designed to provide port granularity for increased flexibility with one PCIe adapter per feature. For availability reasons minimum of two features are required. The Crypto Express4S is a state-of-the-art, tamper-sensing and tamper-responding, programmable cryptographic feature providing a secure cryptographic environment. Each adapter contains a tamper-resistant hardware security module (HSM) that can be configured as a Secure IBM CCA coprocessor, as a Secure IBM Enterprise PKCS #11 (EP11) coprocessor or as an accelerator: Secure IBM CCA coprocessor is for secure key encrypted transactions exploiting CCA callable services (default). Secure IBM Enterprise PKCS #11 (EP11) coprocessor implements industry standardized set of services that adhere to the PKCS #11 specification v2.20 and more recent amendments. This new cryptographic coprocessor mode introduced the PKCS #11 secure key function. Accelerator for public key and private key cryptographic operations used with Secure Sockets Layer / Transport Layer Security (SSL/TLS) acceleration. Federal Information Processing Standards (FIPS) 140-2 certification is supported only when Crypto Express4S is configured as a CCA or an EP11 coprocessor
Configurable Crypto Express3 feature

The Crypto Express3 is an optional feature available only in a carry forward basis in zEC12. Each feature has two PCIe adapters. Each adapter can be configured as a secure coprocessor or as an accelerator:
3
PKCS #11 implementation for Linux
13
8049ch01.fm
Crypto Express3 Coprocessor is for secure key encrypted transactions (default). Crypto Express3 Accelerator is for Secure Sockets Layer/Transport Layer Security (SSL/TLS) acceleration.
Common Cryptographic Architecture (CCA) enhancements

The following functions have been added to Crypto Express4S cryptographic feature as well as to the Crypto Express3 cryptographic feature when running on zEC12 through ICSF FMID HCR77A0: Secure Cipher Text Translate Improved wrapping key strength for security and standards compliance DUKPT for derivation of Message Authentication Code (MAC) and encryption keys Compliance with new Random Number Generator standards EMV enhancements for applications supporting American Express cards
TKE workstation, and support for Smart Card Reader

The Trusted Key Entry (TKE) workstation and the TKE 7.2 Licensed Internal Code (LIC) are optional features on the zEC12. The TKE workstation offers a security-rich solution for basic local and remote key management. It provides to authorized personnel a method for key identification, exchange, separation, update, backup, and a secure hardware based key loading for operational and master keys. TKE also provides a secure management of host cryptographic module and host capabilities. TKE 7.2 LIC functions include: Support for Crypto Express4S defined as a CCA coprocessor; Support for Crypto Express4S defined as an Enterprise PKCS #11 (EP11) coprocessor; Support for new DES operational keys; Support for a new AES CIPHER key attribute; Allow creation of corresponding keys; Support for the new smart card part 74Y0551; Support for 4 smart card readers; Support for stronger key wrapping standards; Elliptic Curve Cryptography (ECC) master key support; Grouping of domains across one or more host cryptographic coprocessors; Stronger cryptography encryption for TKE inbound/outbound authentication; Support for TKE workstation audit records to be sent to a System z host and saved on the host as z/OS System Management Facilities (SMF) records; Support for decimalization tables for each domain on a host cryptographic adapter, used in computing PINs; Support for AES importer, AES exporter KEK, and cipher operational keys; Ability for TKE smart card on a TKE workstation with 7.2 code to hold up to 50 key parts; Support to display the privileged access mode ID and the TKE local cryptographic adapter ID on the TKE console; Requirement for the TKE local cryptographic adapter profile to have access to each TKE application; Ability to generate multiple key parts of the same type at one time.
14
8049ch01.fm
Support for an optional Smart Card Reader attached to the TKE workstation allows for the use of smart cards that contain an embedded microprocessor and associated memory for data storage. Access to and the use of confidential data on the smart cards is protected by a user-defined personal identification number (PIN). When Crypto Express4S is configured as a Secure IBM Enterprise PKCS #11 (EP11) coprocessor the TKE workstation is required to manage the Crypto Express4S feature. If the smart card reader feature is installed in the TKE workstation, the new smart card part 74Y0551 is required for EP11 mode.
1.3.8 Capacity on Demand (CoD)

On demand enhancements enable customers to have more flexibility in managing and administering their temporary capacity requirements. The zEC12 supports the same architectural approach for temporary offerings as z196. Within the zEC12, one or more flexible configuration definitions can be available to solve multiple temporary situations and multiple capacity configurations can be active simultaneously. Up to 200 staged records can be created for many scenarios, and up to eight of them can be installed on the server at any given time. The activation of the records can be done manually or the z/OS Capacity Provisioning Manager can automatically invoke them when Workload Manager (WLM) policy thresholds are reached. Tokens are available that can be purchased for On/Off CoD either before or after execution.
1.3.9 Parallel Sysplex support

Support for Parallel Sysplex includes the Coupling Facility Control Code and coupling links.
Coupling links support

Coupling connectivity in support of Parallel Sysplex environments is provided in zEC12 by the following features: Internal Coupling Channels (ICs) operating at memory speed InterSystem Channel-34 (ISC-3) operating at 2 Gbps and supporting an unrepeated link data rate of 2 Gbps over 9 m single mode fiber optic cabling with an LC Duplex connector 12x InfiniBand coupling links offering up to 6 GBps of bandwidth between zEC12, z196, z114 and z10 systems, for a distance of up to 150 m (492 feet). With the introduction of InfiniBand coupling links (HCA3-O 12xIFB), improved service times can be obtained 1x InfiniBand up to 5 Gbps connection bandwidth between zEC12, z196, z114, and z10 systems, for a distance of up to 10 km (6.2 miles). The HCA3-O LR (1xIFB) type has twice the number of links per fanout card, compared to type HCA2-O LR (1xIFB) All coupling link types can be used to carry Server Time Protocol (STP) messages. The zEC12 does not support ICB4 connectivity.
Only available on zEC12 when carried forward during an upgrade.
15
8049ch01.fm
Statement Of Direction: Removal of ISC-3 support on System z: The IBM zEnterprise EC12 is planned to be the last high-end System z server to offer support of the InterSystem Channel-3 (ISC-3) for Parallel Sysplex environments at extended distances. ISC-3 will not be supported on future high-end System z servers as carry forward on an upgrade. Previously we announced that the IBM zEnterprise 196 (z196) and IBM zEnterprise 114 (z114) servers were the last to offer ordering of ISC-3. Enterprises should continue migrating from ISC-3 features (#0217, #0218, #0219), to 12x InfiniBand (#0171 - HCA3-O fanout) or 1x InfiniBand (#0170 - HCA3-O LR fanout) coupling links.
Coupling Facility Control Code Level 18

Coupling Facility Control Code (CFCC) Level 18 is available for the zEC12, with the following enhancements: Performance enhancements Dynamic structure size alter improvement DB2 GBP cache bypass Cache structure management Coupling channel reporting improvement, enabling RMF to differentiate amongst various IFB link types and detect if a CIB link is running degraded Serviceability enhancements Additional structure control info in CF dumps Enhanced CFCC tracing support Enhanced Triggers for CF non-disruptive dumping. CF storage requirements may increase when moving from CF Level 17 (or below) to CF Level 18. Use of the CF Sizer Tool is recommended. The tool can be obtained from http://www.ibm.com/systems/z/cfsizer/
Server Time Protocol facility

Server Time Protocol (STP) is a server-wide facility that is implemented in the Licensed Internal Code of System z systems and coupling facilities. STP presents a single view of time to PR/SM and provides the capability for multiple servers and coupling facilities to maintain time synchronization with each other. Any System z systems or CFs can be enabled for STP by installing the STP feature. Each server and CF that are planned to be configured in a coordinated timing network (CTN) must be STP-enabled. The STP feature is designed to be the supported method for maintaining time synchronization between System z systems and coupling facilities. The STP design uses the CTN concept, which is a collection of servers and coupling facilities that are time-synchronized to a time value called coordinated server time. Network Time Protocol (NTP) client support is available to the STP code on the zEC12, z196, z114, z10 and z9. With this functionality, the zEC12, z196, z114, z10 and z9 can be configured to use an NTP server as an external time source (ETS). This implementation answers the need for a single time source across the heterogeneous platforms in the enterprise, allowing an NTP server to become the single time source for the zEC12, z196, z114, z10 and z9, as well as other servers that have NTP clients (UNIX, NT,
16
8049ch01.fm
and so on). NTP can only be used as ETS for an STP-only CTN where no server can have an active connection to a Sysplex Timer. The time accuracy of an STP-only CTN is further improved by adding an NTP server with the pulse per second output signal (PPS) as the ETS device. This type of ETS is available from various vendors that offer network timing solutions. Improved security can be obtained by providing NTP server support on the Hardware Management Console ((HMC) for the Support Element (SE), as the HMC is normally attached to the private dedicated LAN for System z maintenance and support. For zEC12, authentication support is added to the HMCs NTP communication with NTP time servers. A zEC12 cannot be connected to a Sysplex Timer. It is preferable to migrate to an STP-only Coordinated Time Network (CTN) for existing environments. It is possible to have a zEC12 as a Stratum 2 or Stratum 3 server in a Mixed CTN, as long as there are at least two System z10s attached to the Sysplex Timer operating as Stratum 1 servers.
1.4 IBM zEnterprise BladeCenter Extension (zBX)

Statement of Direction: The IBM zEnterprise EC12 will be the last server to support connections to an STP Mixed CTN. This includes the Sysplex Timer (9037). After zEC12, servers that require time synchronization, such as to support a base or Parallel Sysplex, will require Server Time Protocol (STP), and all servers in that network must be configured in STP-only mode. The IBM zEnterprise BladeCenter Extension(zBX) is the infrastructure for extending tried and true System z qualities of service and management capabilities across a set of heterogeneous compute elements in an ensemble. The zBX is available as an optional machine to work along with the zEC12 server and consists of the following: Up to four IBM 42U Enterprise racks. Up to eight BladeCenter chassis with up to 14 blades each with up to two chassis per rack. Blades, up to 1125. Intranode management network (INMN) top-of-rack (TOR) switches. The INMN provides connectivity between the zEC12 Support Elements and the zBX, for management purposes. Intraensemble data network (IEDN) top-of-rack (TOR) switches. The IEDN is used for data paths between the zEC12 and the zBX, and the other ensemble members, and also for customer data access. 8 Gbps Fibre Channel switch modules for connectivity to a SAN. Advanced Management Modules (AMMs) for monitoring and management functions for all the components in the BladeCenter Power Distribution Units (PDUs) and cooling fans. Optional acoustic rear door or optional rear door heat exchanger. The zBX is configured with redundant hardware infrastructure to provide qualities of service similar to those of System z, such as the capability for concurrent upgrades and repairs.
5
The maximum number of blades varies according to the blade type and blade function.
17
8049ch01.fm
1.4.1 Blades
There are two types of blades that can be installed and operated in the IBM zEnterprise BladeCenter Extension (zBX): Optimizer Blades: IBM WebSphere DataPower Integration Appliance XI50 for zEnterprise blades. IBM Blades: A selected subset of IBM POWER7 blades A selected subset of IBM BladeCenter HX5 blades These blades have been thoroughly tested to ensure compatibility and manageability in the IBM zEnterprise System environment: IBM POWER7 blades are virtualized by PowerVM Enterprise Edition, and the virtual servers run the AIX operating system. IBM BladeCenter HX5 blades are virtualized using an integrated hypervisor for System x and the virtual servers run Linux on System x (Red Hat Enterprise Linux - RHEL and SUSE Linux Enterprise Server - SLES) operating systems. Enablement for the blades is specified with an entitlement feature code to be configured on the zEC12s.
1.4.2 IBM WebSphere DataPower Integration Appliance XI50 for zEnterprise

The IBM WebSphere DataPower Integration Appliance XI50 for zEnterprise (DataPower XI50z) is a multifunctional appliance that can help provide multiple levels of XML optimization, streamline and secure valuable service-oriented architecture (SOA) applications, and provide drop-in integration for heterogeneous environments by enabling core Enterprise Service Bus (ESB) functionality, including routing, bridging, transformation, and event handling. It can help to simplify, govern, and enhance the network security for XML and web services. When the DataPower XI50z is installed in the zBX, the Unified Resource Manager provides integrated management for the appliance, to simplify control and operations including change management, energy monitoring, problem detection, problem reporting, and dispatching of an IBM System z Service Representative as needed.
1.5 Unified Resource Manager

The zEnterprise Unified Resource Manager is the integrated management fabric that executes on the Hardware Management Console (HMC) and Support Element (SE). The Unified Resource Manager is comprised of six management areas (see Figure 1-1 on page 2): 1. Operational controls (Operations): Includes extensive operational controls for various management functions. 2. Virtual server lifecycle management (Virtual servers): Enables directed and dynamic virtual server provisioning across hypervisors from a single uniform point of control. 3. Hypervisor management (Hypervisors): Enables the management of hypervisors and support for application deployment. 4. Energy management (Energy): Provides energy monitoring and management capabilities that can be used to better understand the power and cooling demands of the zEnterprise System.
18
8049ch01.fm
5. Network management (Networks): Creates and manages virtual networks, including access control, which allows virtual servers to be connected together. 6. Workload Awareness and platform performance management (Performance): Management of CPU resource across virtual servers hosted in the same hypervisor instance to achieve workload performance policy objectives. The Unified Resource Manager provides energy monitoring and management, goal-oriented policy management, increased security, virtual networking, and storage configuration management for the physical and logical resources of a given ensemble.
1.6 Hardware Management Consoles and Support Elements

The Hardware Management Consoles (HMCs) and Support Elements (SEs) are appliances that together provide hardware platform management for ensemble nodes. The HMC is used to manage, monitor, and operate one or more zEnterprise CPCs and their associated logical partitions and zBXs. The HMC6 has a global (ensemble) management scope, whereas the SE has (local) node management responsibility. When tasks are performed on the HMC, the commands are sent to one or more SEs, which then issue commands to their zEnterprise CPCs and zBXs. In order to promote high availability, an ensemble configuration requires a pair of HMCs in primary and alternate roles.
1.7 Operating systems and software

The zEC12 is supported by a large set of software, including ISV applications. This section lists only the supported operating systems. Exploitation of various features might require the latest releases. Further information is contained in Chapter 8, Software support on page 243. Exploitation of some features might require the latest releases. The supported operating systems for the zEC12 are: z/OS Version 1 Release 12 or later releases z/OS Version 1 Release 11 with the IBM Lifecycle Extension with PTFs z/OS Version 1 Release 10 with the IBM Lifecycle Extension with PTFs7 z/VM Version 5 Release 4 or later z/VSE Version 4 Release 3 or later z/TPF Version 1 Release 1 Linux on System z distributions: SUSE Linux: SLES 10 and SLES 118 Red Hat: RHEL 59 and RHEL 6 Operating system support for the IBM blades on the zBX includes: For the POWER7 blades, AIX Version 5 Release 3 or later, with PowerVM Enterprise Edition Linux on System x (64-bit only): Red Hat RHEL 5.5, 5.6, 5.7, 6.0, and 6.1 SUSE SLES 10 (SP4), 11 SP1
6 7 8 9
From Version 2.11, see12.6, HMC in an ensemble on page 422. The z/OS V1 Release 10 will require IBM Lifecycle Extension after September 2011. SLES is the abbreviation for SUSE Linux Enterprise Server. RHEL is the abbreviation for Red Hat Enterprise Linux.
19
8049ch01.fm
Windows Server 2008 R2 and Windows Server 2008 SP2 (Datacenter Edition recommended), 64-bit only With support for IBM WebSphere software, full support for SOA, Web services, J2EE, Linux, and Open Standards, the zEC12 is intended to be a platform of choice for integration of a new generation of applications with existing applications and data.
IBM compilers
The latest versions of the compilers are required in order to exploit the new and enhanced instructions introduced with each system. Enterprise COBOL, Enterprise PL/I, and XL C/C++ are leading-edge, z/OS-based compilers that maximize middleware by providing access to IBM DB2, CICS, and IMS systems, and allow integration with web services, XML, and Java. Such interoperability enables you to capitalize on existing IT investments while smoothly incorporating new, web-based applications into your organizations infrastructure. z/OS XL C/C++ helps you to create and maintain critical business applications written in C or C++ to maximize application performance and improve developer productivity. z/OS XL C/C++ can transform C or C++ source code to fully exploit System z hardware (including the zEC12), through hardware-tailored optimizations, built-in functions, performance-tuned libraries, and language constructs that simplify system programming and boost application runtime performance.
1.8 Reliability, availability, and serviceability

RAS is the acronym for Reliability, Availability and Serviceability. The zEC12 system reliability, availability, and serviceability (RAS) strategy is a building-block approach developed to meet the client's stringent requirements of achieving continuous reliable operation. Those building blocks are error prevention, error detection, recovery, problem determination, service structure, change management, and measurement and analysis. The initial focus is on preventing failures from occurring in the first place. This is accomplished by using Hi-Rel (highest reliability) components; using screening, sorting, burn-in, and run-in; and by taking advantage of technology integration. For Licensed Internal Code and hardware design, failures are eliminated through rigorous design rules; design walk-through; peer reviews; element, subsystem, and system simulation; and extensive engineering and manufacturing testing. The RAS strategy is focused on a recovery design that is necessary to mask errors and make them transparent to customer operations. An extensive hardware recovery design has been implemented to detect and correct memory array faults. In cases where total transparency cannot be achieved, you can restart the server with the maximum possible capacity. Some of the RAS improvements of the zEC12 are: Improved error detection for the L3 / L4 memory cache IBM System z Advanced Workload Analysis Reporter in order to detect abnormal behavior of z/OS OSA firmware changes to increase concurrent MCL capability Digital Temperature Sensor (DTS) and On Chip Temperature Sensor on the Processor Unit (PU) chips Examples of reduced impact of planned and unplanned system outages include: 20
8049ch01.fm
Enhanced book availability Hot pluggable PCIe I/O drawers and I/O drawers Redundant I/O interconnect Concurrent PCIe fanout and Host Channel Adapter (HCA-O, HCA-C) fanout card hot-plug Enhanced driver maintenance For a more detailed description, see Chapter 10, RAS on page 355
1.9 Performance
The zEC12 Model HA1 is designed to offer approximately 1.5 times more capacity than the z196 Model M80 system. Uniprocessor performance has also increased significantly. A zEC12 Model 701 offers, on average, performance improvements of about 1.25 times over the z196 Model 701. Figure 1-3 shows the estimated capacity ratios for zEC12, z196, z10 EC, and z9 EC. Notice that LSPR numbers given for z9 EC, z10 EC and z196 systems were obtained with the z/OS 1.11 operating system, and with z/OS 1.13 for the zEC12 system.
z9 EC z10 EC z196 zEC12
1) 2)
up to 54-way: 193 to 18,505 PCIs 1 up to 64-way: 214 to 31,826 PCIs 1 up to 80-way: 240 to 52,286 PCIs 1 up to 101-way: 240 to 78,500 PCIs 2
LSPR data based on z/OS 1.11 LSPR data based on z/OS 1.13
zEC12
Figure 1-3 zEC12 to z196, z10 EC and z9 EC performance comparison
On average, the zEC12 can deliver up to 50% more performance in an 101-way configuration than an IBM System z196 80-way. However, variations on the observed performance increase are dependent upon the workload type. Consult the Large System Performance Reference (LSPR) when you consider performance on the zEC12. The range of performance ratings across the individual LSPR workloads is likely to have a large spread. More performance variation of individual logical partitions exists because the impact of fluctuating resource requirements of other partitions can be more pronounced with the increased numbers of partitions and additional PUs available. For more information, see 1.9.6, Workload performance variation on page 28.
21
8049ch01.fm
For detailed performance information, see the LSPR web site: https://www-304.ibm.com/servers/resourcelink/lib03060.nsf/pages/lsprindex The MSU ratings are available from the following web site: http://www-03.ibm.com/systems/z/resources/swprice/reference/exhibits/
1.9.1 LSPR workload suite

Historically, LSPR capacity tables, including pure workloads and mixes, have been identified with application names or a software characteristic. Examples are CICS, IMS, OLTP-T10, CB-L11, LoIO-mix12 and TI-mix13. However, capacity performance is more closely associated with how a workload uses and interacts with a particular processor hardware design. With the availability of CPU Measurement Facility (CPU MF) data introduced on the z10, the ability to gain insight into the interaction of workload and hardware design in production workloads has arrived. CPU MF data helps LSPR to adjust workload capacity curves based on the underlying hardware sensitivities, in particular, the processor access to caches and memory. This is known as nest activity intensity. With this, the LSPR introduces three new workload capacity categories that replace all prior primitives and mixes. The LSPR contains the internal throughput rate ratios (ITRRs) for the zEC12 and the previous generation processor families, based upon measurements and projections that use standard IBM benchmarks in a controlled environment. The actual throughput that any user experiences can vary depending on considerations, such as the amount of multiprogramming in the user's job stream, the I/O configuration, and the workload processed. Therefore, no assurance can be given that an individual user can achieve throughput improvements equivalent to the performance ratios stated.
1.9.2 Fundamental components of workload capacity performance

Workload capacity performance is sensitive to three major factors: instruction path length, instruction complexity, and memory hierarchy. Let us examine each of these three.
Instruction path length

A transaction or job will need to execute a set of instructions to complete its task. These instructions are composed of various paths through the operating system, subsystems and application. The total count of instructions executed across these software components is referred to as the transaction or job path length. Clearly, the path length will vary for each transaction or job depending on the complexity of the task(s) that must be performed. For a particular transaction or job, the application path length tends to stay the same presuming the transaction or job is asked to perform the same task each time. However, the path length associated with the operating system or subsystem might vary based on a number of factors such as these: Competition with other tasks in the system for shared resources as the total number of tasks grows, more instructions are needed to manage the resources The n-way (number of logical processors) of the image or LPAR as the number of logical processors grows, more instructions are needed to manage resources serialized by latches and locks.
10 11 12 13
Traditional online transaction processing workload (formerly known as IMS) Commercial batch with long-running jobs Low I/O Content Mix Workload Transaction Intensive Mix Workload
22
8049ch01.fm
Instruction complexity
The type of instructions and the sequence in which they are executed will interact with the design of a microprocessor to affect a performance component we can define as instruction complexity. There are many design alternatives that affect this component, such as these: Cycle time (GHz) Instruction architecture Pipeline Superscalar Out-of-order execution Branch prediction As workloads are moved between microprocessors with various designs, performance will likely vary. However, when on a processor, this component tends to be quite similar across all models of that processor.
Memory hierarchy and memory nest

The memory hierarchy of a processor generally refers to the caches, data buses, and memory arrays that stage the instructions and data needed to be executed on the microprocessor to complete a transaction or job. There are many design alternatives, such as these, that affect this component: Cache size Latencies (sensitive to distance from the microprocessor) Number of levels, MESI (management) protocol, controllers, switches, number and bandwidth of data buses and others. Certain caches are private to the microprocessor core which means that only that microprocessor core can access them. Other cache(s) are shared by multiple microprocessor cores. We define the term memory nest for a System z processor to refer to the shared caches and memory along with the data buses that interconnect them. Figure 1-4 below introduces the concept of memory nest in a zEC12 2-book system.
The Nest
Memory L4 Cache
L3 Cache
L2
L1
Memory L4 Cache
L3 Cache L3 Cache
L2
L1
...
L2
L1
...
L2
L1
L3 Cache
L2
L1
CPU1
....
L2
L1
CPU6
CPU1
....
L2
L1
CPU6
CPU1
....
CPU6
CPU1
....
L2
L1
CPU6
Book 1
Figure 1-4 Memory hierarchy on the zEC12 2-book system
Book 2
23
8049ch01.fm
Workload capacity performance will be quite sensitive to how deep into the memory hierarchy the processor must go to retrieve the workloads instructions and data for execution. Best performance occurs when the instructions and data are found in the cache(s) nearest the processor so that little time is spent waiting prior to execution; as instructions and data must be retrieved from farther out in the hierarchy, the processor spends more time waiting for their arrival. As workloads are moved between processors with various memory hierarchy designs, performance will vary as the average time to retrieve instructions and data from within the memory hierarchy will vary. Additionally, when on a processor, this component will continue to vary significantly, because the location of a workloads instructions and data within the memory hierarchy is affected by many factors including, but not limited to, these: Locality of reference I/O rate Competition from other applications and/or LPARs.
1.9.3 Relative nest intensity

The most performance sensitive area of the memory hierarchy is the activity to the memory nest, namely, the distribution of activity to the shared caches and memory. The term Relative Nest Intensity (RNI) indicates the level of activity to this part of the memory hierarchy. Using data from CPU MF, the RNI of the workload running in an LPAR can be calculated. The higher the RNI, the deeper into the memory hierarchy the processor must go to retrieve the instructions and data for that workload. Many factors influence the performance of a workload. However, for the most part what these factors are influencing is the RNI of the workload. It is the interaction of all these factors that result in a net RNI for the workload which in turn directly relates to the performance of the workload. We emphasize that these are simply tendencies and not absolutes. For example, a workload might have a low I/O rate, intensive CPU use, and a high locality of reference; all factors that suggest a low RNI. But, what if it is competing with many other applications within the same LPAR and many other LPARs on the processor which tend to push it toward a higher RNI? It is the net effect of the interaction of all these factors that determines the RNI of the workload which in turn greatly influences its performance. The traditional factors that have been used to categorize workloads in the past are listed along with their RNI tendency in Figure 1-5
24
8049ch01.fm
Low
Batch Low Single Intensive Low High locality Simple Extensive
Relative Nest Intensity

Application Type IO Rate Application Mix CPU Usage Dispatch Rate Data Reference Pattern LPAR Configuration Software Configuration Tuning
High
Transactional High Many Light High Diverse Complex Limited
Figure 1-5 The traditional factors that have been used to categorize workloads.
Note that one can do little to affect most of these factors. An application type is whatever is necessary to do the job. Data reference pattern and CPU usage tend to be inherent to the nature of the application. LPAR configuration and application mix are mostly a function of what needs to be supported on a system. I/O rate can be influenced somewhat through buffer pool tuning. However, one factor that can be affected, software configuration tuning, is often overlooked but can have a direct impact on RNI. Here we refer to the number of address spaces (such as CICS AORs or batch initiators) that are needed to support a workload. This factor has always existed but its sensitivity is higher with todays high frequency microprocessors. Spreading the same workload over a larger number of address spaces than necessary can raise a workloads RNI as the working set of instructions and data from each address space increases the competition for the processor caches. Tuning to reduce the number of simultaneously active address spaces to the proper number needed to support a workload can reduce RNI and improve performance. In the LSPR, the number of address spaces for each processor type and Nway configuration is tuned to be consistent with what is needed to support the workload. Thus, the LSPR workload capacity ratios reflect a presumed level of software configuration tuning. This suggests that re-tuning the software configuration of a production workload as it moves to a bigger or faster processor might be needed in order to achieve the published LSPR ratios.
1.9.4 LSPR workload categories based on relative nest intensity

A workloads relative nest intensity is the most influential factor in determining workload performance. Other more traditional factors such as application type or I/O rate have RNI tendencies, but it is the net RNI of the workload that is the underlying factor in determining the workloads capacity performance. With this in mind, the LSPR now runs various combinations of former workload primitives such as CICS, DB2, IMS, OSAM, VSAM, WebSphere, COBOL and utilities to produce capacity curves that span the typical range of RNI. Three new workload categories are represented in the LSPR tables:
LOW (relative nest intensity):

A workload category representing light use of the memory hierarchy. This is similar to past high scaling primitives.
AVERAGE (relative nest intensity):

25
8049ch01.fm
A workload category representing average use of the memory hierarchy. This is similar to the past LoIO-mix workload and is expected to represent the majority of production workloads.
HIGH (relative nest intensity):

A workload category representing heavy use of the memory hierarchy. This is similar to the past TI-mix workload. These categories are based on the relative nest intensity, which is influenced by many variables such as application type, I/O rate, application mix, CPU usage, data reference patterns, LPAR configuration, and software configuration running, among others. CPU MF data can be collected by z/OS System Measurement Facility on SMF 113 records. On zEC12, the number of extended counters is increased to 183. The structure of the SMF records does not change.
1.9.5 Relating production workloads to LSPR workloads

Historically, there have been a number of techniques used to match production workloads to LSPR workloads such as these: Application name (a customer running CICS can use the CICS LSPR workload) Application type (create a mix of the LSPR online and batch workloads) I/O rate (low I/O rates used a mix of the low I/O rate LSPR workloads). The previous LSPR workload suite was made up of the following workloads: Traditional online transaction processing workload OLTP-T (formerly known as IMS) Web-enabled online transaction processing workload OLTP-W (also known as Web/CICS/DB2) A heavy Java-based online stock trading application WASDB (previously referred to as Trade2-EJB) Batch processing, represented by the CB-L (commercial batch with long-running jobs or CBW2) A new ODE-B Java batch workload, replacing the CB-J workload. The traditional Commercial Batch Short Job Steps (CB-S) workload (formerly CB84) was dropped. You can see the traditional factors that have been used to categorize workloads in Figure 1-5 on page 25. The previous LSPR provided performance ratios for individual workloads and for the default mixed workload, which was composed of equal amounts of four of the workloads described above (OLTP-T, OLTP-W, WASDB, and CB-L). Guidance in converting LSPR previous categories to the new ones is given in Figure 1-6. The IBM zPCR tool14 has been changed to support the new z/OS workload categories.
14
IBM Processor Capacity Reference: A no-cost tool that reflects the latest IBM LSPR measurements, and is available at http://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/PRS1381
26
8049ch01.fm
Low
Relative NEST Intensity
High
LSPR Primitives
CB-L
WASDB
OLTP-T
OLTP-W
LSPR Mixes
LoIO-Mix
TM-Mix
TI-Mix DI-Mix
CB-Mix
TD-Mix
LSPR Categories
Low
LowAverage
Average
AverageHigh
High
Figure 1-6 New z/OS workload categories defined
However, as discussed in the LSPR Workload Categories section, the underlying performance sensitive factor is how a workload interacts with the processor hardware. These past techniques were simply trying to approximate the hardware characteristics that were not available through software performance reporting tools. Beginning with the z10 processor, the hardware characteristics can now be measured using CPU MF (SMF 113) counters data. To reflect the memory hierarchy changes in the new zEC12 system, the number of counters has been increased to 183. Thus, there is an opportunity to be able to match a production workload to an LSPR workload category through these hardware characteristics (see the LSPR Workload Categories section for a discussion about RNI Relative Nest Intensity). The AVERAGE RNI LSPR workload is intended to match the majority of customer workloads. When no other data is available, use it for a capacity analysis. DASD I/O rate has been used for many years to separate workloads into two categories: those whose DASD I/O per MSU (adjusted) is <30 (or DASD I/O per PCI <5) and those higher than these values. The majority of production workloads fell into the low I/O category and a LoIO-mix workload was used to represent them. Using the same I/O test, these workloads now use the AVERAGE RNI LSPR workload. Workloads with higher I/O rates can use the HIGH RNI workload or the AVG-HIGH RNI workload that is included with zPCR. Low-Average and Average-High categories allow better granularity for workload characterization. For z10 and newer processors, the CPU MF data can be used to provide an additional hint as to workload selection. When available, this data allows the RNI for a production workload to be calculated. Using the RNI and another factor from CPU MF, the L1MP (percentage of data and instruction references that miss the L1 cache), a workload can be classified as LOW, AVERAGE or HIGH RNI. This classification and resulting hint is automated in the zPCR tool. It is best to use zPCR for capacity sizing. The LSPR workloads, updated for EC12 are considered to reasonably reflect current and growth workloads of the customer. The set contains three generic workload categories based on z/OS 1.13 supporting up to 101 processors in a single image.
27
8049ch01.fm
1.9.6 Workload performance variation

Because of the nature of the zEC12 multi-book system and resource management across those books, performance variability from application to application, similar to that seen on the z9 EC, z10 EC, z196 is expected. This variability can be observed in certain ways. The range of performance ratings across the individual workloads is likely to have a spread, but not as large as with the z10 EC. The memory and cache designs affect various workloads in a number of ways. All workloads are improved, with cache-intensive loads benefiting the most. When comparing moving from z9 EC to z10 EC with moving from z10 EC to z196 or from z196 to zEC12, it is likely that the relative benefits per workload will vary. Those workloads that benefited more than the average when moving from z9 EC to z10 EC will benefit less than the average when moving from z10 EC to z196, and vice-versa. Nevertheless, the workload variability for moving from z196 to zEC12 is expected to be less than the last few migrations. The customer impact of this variability is seen as increased deviations of workloads from single-number metric-based factors such as MIPS, MSUs, and CPU time charge back algorithms. Experience demonstrates that System z servers can be run at up to 100% utilization levels, sustained, although most clients prefer to leave a bit of white space and run at 90% or slightly under. For any capacity comparison exercise; using one number like MIPS or MSU metric is not a valid method. That is why, while doing capacity planning, it is best to use zPCR and involve IBM technical support.
1.9.7 Main performance improvement drivers with zEC12

The zEC12 is designed to deliver new levels of performance and capacity for large scale consolidation and growth. The following attributes and design points of the zEC12 contribute to overall performance and throughput improvements as compared to the z196.
z/Architecture implementation enhancements:

Transactional Execution (TX) designed for z/OS, Java, DB2 and other exploiters Runtime Instrumentation (RI) provides dynamic and self-tuning online re-compilation capability for Java workloads Enhanced DAT-2 for supporting 2 GB large pages for DB2 buffer pools, Java heap size and other large structures Software directives implementation to improve hardware performance Decimal format conversions for COBOL programs.
zEC12 microprocessor design enhancements:

Six processor cores per chip Enhanced Out Of Order (OOO) execution design Improved pipeline balance Enhanced branch prediction latency and instruction fetch throughput Improvements on execution bandwidth and throughput New design for Level 2 private cache with separation of cache structures for instructions and L2 operands Reduced access latency for most of Level 1 cache misses Bigger Level 2 cache with shorter latency Third level on-chip shared cache is doubled Fourth level book-shared cache is doubled Hardware and software prefetcher handling improvements Increased execution/completion throughput 28
8049ch01.fm
Improve fetch and store conflict scheme Enhance branch prediction structure and sequential instruction fetching Millicode performance improvements Optimized floating-point performance Faster engine for fixed-point division New second level branch prediction array One cryptographic/compression co-processor per core Cryptography support of UTF8<>UTF16 conversions Higher clock frequency at 5.5 GHz IBM CMOS 13S 32nm SOI technology with IBM eDRAM technology.
zEC12 design enhancements:

Increased total number of PUs available on the system, from 96 to 120, and number of characterizable cores, from 80 to 101 Hardware System Area increased from 16 GB to 32 GB Increased default number of SAP processors per book New CFCC code available for improved performance Elapsed time improvements when dynamically altering the size of a cache structure DB2 conditional write to a group buffer pool (GBP) Performance improvements for coupling facility cache structures to avoid flooding the coupling facility cache with changed data and avoid excessive delays and backlogs for cast-out processing Performance throughput enhancements for parallel cache castout processing by extending the number of RCC cursors beyond 512 CF Storage class and castout class contention avoidance by breaking up individual storage class and castout class queues to reduce storage class and castout class latch contention.
New features available on the zEC12:

Crypto Express4S performance enhancements Flash Express PCIe cards to handle paging workload spikes and improve performance.
29
8049ch01.fm
30
8049ch02.fm
Chapter 2.
CPC Hardware Components

This chapter introduces IBM zEnterprise EC12 (zEC12) hardware components along with significant features and functions with their characteristics and options. The objective is to explain the zEC12 hardware building blocks and how these components interconnect from a physical point of view. This information is useful for planning purposes and can help in defining configurations that fit your requirements. This chapter discusses the following topics: Frames and cage on page 32 Book concept on page 36 Multi-chip module on page 40 Processor units and storage control chips on page 41 Memory on page 48 Reliability, availability, serviceability (RAS) on page 55 Connectivity on page 57 Model configurations on page 61 Power and cooling on page 68 Summary of zEC12 structure on page 76
31
8049ch02.fm
2.1 Frames and cage

System z frames are enclosures built to Electronic Industry Association (EIA) standards. The zEC12 has two 42U EIA frames, shown in Figure 2-1. The two frames, A and Z, are bolted together and have positions for one processor cage and a combination of PCIe I/O drawers, I/O drawers, and one I/O cage. All books, including the distributed converter assemblies (DCAs) on the books and the cooling components, are located in the processor cage in the A frame. Figure 2-1 shows the front view of frame A (with four books installed) and frame Z of an air cooled zEC12.
Overhead Power Cables (option)
Flexible Service P rocessor (FSP) controller cards
Internal Batteries (option) Power Supplies
Processor Books with Memory, HCAand PCIe-Fanout cards InfiniBand and PCIe I/O Interconnects
Support Elements
N+1 Radiators
PCIe I/O drawers (maximum 5)
O ptional FICON FQC not shown
Figure 2-1 CPC cage, I/O drawers, and I/O cage locations - air-cooled system - front view
32
8049ch02.fm
Figure 2-2 shows the front view of a water cooled zEC12 .
Internal Batteries (optional)
Flexible S ervice Processor (FSP) controller cards
Power Supplies Processor Books with Memory, HCAand PCIe-Fanout cards Support Elements InfiniBand and PCIe I/O Interconnects I/O cage Carried Forward PCIe I/O drawer N+1 Water Cooling Units
Figure 2-2 CPC cage, I/O drawers, and I/O cage locations - water cooled system - front view
2.1.1 Frame A
As shown in Figure 2-1 on page 32 and Figure 2-2, frame A has these main components: Two optional Internal Battery Features (IBFs), which provide the function of a local uninterrupted power source. The IBF further enhances the robustness of the power design, increasing power line disturbance immunity. It provides battery power to preserve processor data in case of a loss of power on all connected AC or DC feeds from the utility provider. The IBF provides battery power to preserve full system function despite the loss of power at all system power cords. It allows continuous operation through intermittent losses, brownouts, and power source switching, or can provide time for an orderly shutdown in case of a longer outage. The IBF provides up to 10 minutes of full power, depending on the I/O configuration. The batteries are installed in pairs. Two to six battery units can be installed. The number is based on the zEC12 model and configuration. Two, fully redundant radiator units, containing water pumps to feeding the internal closed water loops for MCM cooling, heat exchanger, manifold assembly and blowers to provide the cooling of the MCMs. Instead of the radiator units, the customer can specify two Water Conditioning Units (WCUs) connected to a customer chilled water supply. When the WCUs option is used for cooling the books an additional exhaust air heat exchanger is installed in the rear of the frame. Processor cage, which contains up to four books, connected to the internal water cooling system.
Chapter 2. CPC Hardware Components
33
8049ch02.fm
Depends on the configuration the following I/O assembly can be found (where any combination of up to two drawer or the I/O cage is possible): up to two PCI/e I/O drawer for installation of new form factor cards for FICON8S, OSA-Express4S, Crypto Express4S and Flash Express. The PCIe I/O drawer is used for new installation or can be carried forward from z196 MES and is equipped with a maximum of 32 features. one I/O drawer each containing up to eight I/O cards of any type of FICON Express8, FICON Express4, OSA- Express3, ISC-3 and Crypto Express3 features. The I/O drawer itself and all the current installed features can only be carry forwarded with an MES from a z10 or a z196. one I/O cage, which can house 28 I/O card slots for installation of FICON Express8, FICON Express4, OSA- Express3, ISC-3 and Crypto Express3 features. One I/O cages is supported, the I/O cage itself and all the current installed features can only carried forward with an MES from a z10 or a z196. Air-moving devices (AMD), which provide N+1 redundant cooling for the fanouts, memory, and DCAs.
2.1.2 Frame Z
As shown in Figure 2-1 on page 32, frame Z has these main components: Two optional Internal Battery Features (IBFs). Bulk Power Assemblies (BPAs). The number of BPAs vary and depend on the configuration of the zEC12. Detailed information about the required number of BPAs can be found in 2.9.1, Power consumption on page 68 The Support Element (SE) tray, located in front of I/O drawer slots, contains the two SEs. Up two 4 drawers, which can be all PCIe I/O drawer or a combination of up to two I/O drawer and PCIe drawer. I/O cage is not supported in Frame Z. When the WCUs option is used for cooling the books an additional exhaust air heat exchanger is installed in the rear of the frame. An optional overhead power cable feature (shown in Figure 2-1 on page 32) and an also optional top exit I/O cabling feature which is, when ordered present at frame A and frame Z.
2.1.3 I/O cages, I/O drawers, and PCIe I/O drawers

Each book has up to eight dual port fanouts to support two types of I/O infrastructures for data transfer: PCIe I/O infrastructure with bandwidth of 8 GBps InfiniBand I/O infrastructure with bandwidth of 6 GBps PCIe I/O infrastructure uses the PCIe fanout to connect to PCIe I/O drawer that can contain the following feature cards: FICON Express8S channels (two port cards) OSA-Express channels: OSA-Express4S 10 Gb Ethernet (one port cards, LR or SR, one CHPID) OSA-Express4S Gb Ethernet (two port cards, LX or SX, one CHPID) OSA-Express4S 1000BASE-T Ethernet (two port cards, one CHPID) 34
8049ch02.fm
Crypto Express4S. Each Crypto Express4S feature holds one PCI Express cryptographic adapter. Flash Express. Each Flash Express feature occupies two I/O slots but does not have a CHPID type. Logical partitions in all CSSs have access to the features. InfiniBand I/O infrastructure uses the HCA2-C fanout to connect to I/O drawer(s) or I/O cage(s) that can contain a variety of channel, Coupling Link, OSA-Express, and Cryptographic feature cards: FICON channels (FICON or FCP modes) FICON Express4 channels (four port cards) FICON Express8 channels (four port cards) ISC-3 links (up to four coupling links, two links per daughter card). Two daughter cards (ISC-D) plug into one mother card (ISC-M). OSA-Express channels: OSA-Express3 10 Gb Ethernet (two ports cards, LR or SR, two CHPIDs) OSA-Express3 Gb Ethernet (four port cards, LX and SX, two CHPIDs) OSA-Express3 1000BASE-T Ethernet (four port cards, two CHPIDs) OSA-Express2 Gb Ethernet (two port cards, SX, LX, two CHPIDs) OSA-Express2 1000BASE-T Ethernet (two port card, two CHPIDs) Crypto Express3 feature (FC 0864) has two PCI Express adapters per feature. A PCI Express adapter can be configured as a cryptographic coprocessor for secure key operations or as an accelerator for clear key operations. InfiniBand coupling to a coupling facility is achieved directly from the HCA2-O (12xIFB) fanout and HCA3-O (12xIFB) fanout to the coupling facility with a bandwidth of 6 GBps. The HCA2-O LR (1xIFB) fanout and HCA3-O LR (1xIFB) fanout support long distance coupling links for up to 10 km or 100 km when extended by using System z qualified DWDM1 equipment. Supported bandwidths are 5 Gbps and 2.5 Gbps, depending on the DWDM equipment used.
2.1.4 Top exit I/O cabling

On zEC12 you now have the option of ordering the infrastructure to support top exit for the fiber optic cables (ESCON, FICON, OSA, 12x InfiniBand, 1x InfiniBand, and ISC-3) as well as for the copper cables for the 1000BASE-T Ethernet features. Top exit I/O cabling is designed to provide you with an additional option. Instead of all of your cables exiting under the CPC and/or under the raised floor, you now have the flexibility to choose the option that best meets the requirements of your data center. Top exit I/O cabling can also help to increase the air flow. This option is offered on new build as well as MES orders.
Dense Wave Division Multiplexing
35
8049ch02.fm
2.2 Book concept

The central processor complex (CPC) uses a packaging design for its processors based on books. A book contains a multi-chip module (MCM), memory, and connectors to I/O drawers, or an I/O cage and to other CPCs. Books are located in the processor cage in frame A. The zEC12 has from one book to four books installed. A book and its components are shown in Figure 2-3.
Rear
16 D I MM s 100m m H ig h M C M @ 1800W inte rn al system wa ter cool ed
F ront
M MC
Memo ry
FS P and I/O F anou t Cards
M emory
3 DC A Po wer Sup pl ies
14 D IM Ms 10 0m m H i gh
I nterna l water l oop C o nnector s
Figure 2-3 Book structure and components
Each book contains the following components: One multi-chip module (MCM) with six hex-core microprocessor chips, having either 27 or 30 processor units PUs), depending on the model, and two storage control chips with 384 MB of Level 4 cache. Memory DIMMs plugged into 30 available slots, providing from 60 GB to 960 GB of physical memory installed in a book. A combination of up to eight (host channel adapter (HCA) or PCIe) fanout cards. HCA2-Copper connections are for 6GBps links to the I/O cage or I/O drawers in the CPC. PCIe fanouts are used for 8GBps links to the PCIe I/O drawers, and the HCA-Optical fanouts connect to external CPCs (coupling links). Three distributed converter assemblies (DCAs) that provide power to the book. Loss of a DCA leaves enough book power to satisfy the books power requirements (n+1 redundancy). The DCAs can be concurrently maintained. Two flexible service processor (FSP) cards for system control. Figure 2-4 depicts the book logical structure, showing its component connections, including the PUs on MCM.
36
8049ch02.fm
Mem2
Mem1
Mem0
MCU
MCU
MCU
PU
PU
PU
FBC
FBC
SC
SC
PSI
PU
PU
PU
GX
Logical Node
Figure 2-4 Book logical structure
Memory is connected to MCM through three memory control units (MCUs). GX0 to GX7 are the I/O buses interfaces to HCAs, with full store buffering, maximum of 10 GB/s per bus direction, and support for Infiniband and PCIe. Processor support interfaces (PSIs) are used to communicate with FSP cards for system control. Fabric book connectivity (FBC) provides the point-to-point connectivity between books.
2.2.1 Book interconnect topology

Figure 2-5 is showing the point-to-point topology for book communication. Each book communicates directly to all other books in the CPC.
Book
FBC FBC
FBC
Book
FBC FBC
Book
FBC
Book
Figure 2-5 Book-to-book communication
Up to four books can reside in the processor cage. Books slide into a mid-plane card that supports up to four books and is located in the top of frame A. The mid-plane card is also the location of two oscillator cards.
37
8049ch02.fm
The location of books is as follows: In a one-book model, the first book slides in the second slot from the left (processor cage slot location LG06). In a two-book model, the second book slides in the right-most slot (processor cage slot location LG15). In a three-book model, the third book slides in the third slot from the left (processor cage slot location LG10). In a four-book model, the fourth book slides into the left-most slot (processor cage slot location LG01). Table 2-1 indicates the order of book installation and position in the processor cage.
Table 2-1 Book installation order and position in the processor cage Book Installation order Position in cage (LG) Book0 Fourth 01 Book1 First 06 Book2 Third 10 Book3 Second 15
Book installation is concurrent, except for the upgrade to the model HA1. Concurrent book replacement requires a minimum of two books. Tip: The processor cage slot locations are important in the sense that in the physical channel ID (PCHID) report, resulting from the IBM configurator tool, locations 01, 06, 10, and 15 are used to indicate whether book features such as fanouts and AID assignments relate to the first, second, third, or fourth book in the processor cage.
Dual external clock facility: Two external clock facility (ECF) cards are already installed and shipped with the CPC and provide a dual-path interface for pulse per second (PPS). This redundancy allows continued operation even if a single ECF card fails.
This redundant design also allows concurrent maintenance. The two connectors that connect to the PPS output of an NTP server are located above the books and are connected on the mid-plane to which the books are connected. The support element provides a simple network time protocol (SNTP) client. When server time protocol (STP) is used, the time of an STP-only coordinated timing network (CTN) can be synchronized with the time provided by a network time protocol (NTP) server, allowing a heterogeneous platform environment to have the same time source. The time accuracy of an STP-only CTN is improved by adding an NTP server with the pulse per second output signal (PPS) as the external time source (ETS) device. NTP server devices with pulse per second output (PPS) are available from several vendors that offer network timing solutions. A cable connection from the PPS port on the ECF card to the PPS output of the NTP server is required when the zEC12 is using STP and configured in an STP-only CTN using NTP with pulse per second as the external time source. STP tracks the highly stable and accurate PPS signal from the NTP server and maintains an accuracy of 10 s to the ETS, as measured at the PPS input of the IBM zEnterprise EC12. If STP uses an NTP server without PPS, a time accuracy of 100 ms to the ETS is maintained. Figure 2-6 shows the location of the two ECF cards on the CPC, above book 0 and book 3 locations.
38
8049ch02.fm
ECF
OSC
OSC
ECF
Book 0
Book 1
Book 2
Book 3
LG01
LG06
LG10
LG15
Figure 2-6 ECF and OSC cards
STP: Server time protocol (STP) is available as FC 1021. STP is implemented in the Licensed internal code (LIC) and is designed for multiple servers to maintain time synchronization with each other and synchronization to an External Time Source. For more information, see the following publications: Server Time Protocol Planning Guide, SG24-7280 Server Time Protocol Implementation Guide, SG24-7281 Server Time Protocol Recovery Guide, SG24-7380
2.2.2 Oscillator
The zEC12 has two oscillator cards (OSC), a primary and a backup. Although not part of the book design, they are found above the books, connected to the same mid-plane to which the books are connected. If the primary fails, the secondary detects the failure, takes over transparently, and continues to provide the clock signal to the CPC. Figure 2-6 on page 39 shows the location of the two OSC cards on the CPC, above book 1 and book 2 locations.
39
8049ch02.fm
2.2.3 System control

Various system elements use flexible service processors (FSPs). An FSP is based on the IBM Power PC microprocessor technology. It connects to an internal Ethernet LAN to communicate with the support elements (SEs) and provides a subsystem interface (SSI) for controlling components. Figure 2-7 is a conceptual overview of the system control design.
FSP
Bulk Power
FSP
Bulk Power
Primary SE
Ethernet 'hub' 1
Ethernet 'hub' 2
Alternate SE
HMC
FSP External LAN FSP FSP FSP FSP FSP FSP FSP
Book
Book
.....
I/O cage
I/O cage
Figure 2-7 Conceptual overview of system control elements
A typical FSP operation is to control a power supply. An SE sends a command to the FSP to bring up the power supply. The FSP (using SSI connections) cycles the various components of the power supply, monitors the success of each step and the resulting voltages, and reports this status to the SE. Most system elements are duplexed (n+1), and each element has an FSP. Two internal Ethernet LANs and two SEs, for redundancy, and crossover capability between the LANs are available so that both SEs can operate on both LANs. The SEs, in turn, are connected to one or two (external) LANs (Ethernet only), and the Hardware Management Consoles (HMCs) are connected to these external LANs. One or more HMCs can be used, but two (a primary and an alternate2) are mandatory with an ensemble. Additional HMCs can operate a zEC12 when it is not a member of an ensemble.
2.2.4 Book power

Each book gets its power from three distributed converter assemblies (DCAs) that reside in the book. The DCAs provide the required power for the book in an n+1 design. Loss of one DCA leaves enough book power to satisfy its power requirements. The DCAs can be concurrently maintained and are accessed from the rear of the frame.
2.3 Multi-chip module

The multi-chip module (MCM) is a 103-layer glass ceramic substrate (size is 96 x 96 mm) containing eight chip sites and 7356 land grid array (LGA) connections. There are six
2
See 12.6, HMC in an ensemble on page 422 for more information.
40
8049ch02.fm
processor unit (PU) chips and two storage control (SC) chips. Figure 2-8 illustrates the chip locations. The total number of transistors on all chips on the MCM is more than 20 billion.
92 mm PU2 PU1 PU0
SC1
SC0
PU3
PU4
PU5
92 mm
Figure 2-8 zEC12 multi-chip module
The MCM plugs into a card that is part of the book packaging. The book itself is plugged into the mid-plane board to provide interconnectivity between the books, so that a multibook system appears as a symmetric multiprocessor (SMP) system.
2.4 Processor units and storage control chips

Both processor unit (PU) and storage control (SC) chips on the MCM use CMOS 13S chip technology. CMOS 13S is state-of-the-art microprocessor technology based on 15-layer copper interconnections and silicon-on insulator (SOI) technologies. The chip lithography line width is 0.032 m (32 nm). On the MCM, four serial electrically erasable programmable ROM (SEEPROM) chips are rewritable memory chips that hold data without power, use the same technology, and are used for retaining product data for the MCM and engineering information. Two of them are active, whereby the other two are used for redundandancy. Figure 2-9 is the MCM structure diagram, showing the PUs and SCs and their connections, which we detail in the following sections.
41
8049ch02.fm
Memory 2 GX 2 & 6
6 PU c or es 6 L1 c ac he s 6 L2 c ac he s 1 L3 c ache 6 COP MC, GX
Mem ory 1 GX 7 & 1

6 PU c or es 6 L1 cac he s 6 L2 cac he s 1 L3 c ache 6 COP MC, GX
Memory 0 PSI GX 0 P SI GX 5
6 PU core s 6 L1 c aches 6 L2 c aches 1 L3 ca che 6 C OP MC, GX
GX 4
6 PU c or es 6 L1 cac he s 6 L2 cac he s 1 L3 c ache 6 COP MC, GX
GX 3
192MB L4 SC
192MB L4 SC
Off- Book Interconnect
Figure 2-9 PU MCM structure
2.4.1 PU chip
The zEC12 PU chip is an evolution of the z196 core design, using CMOS 13S technology, out-of-order instruction processing, higher clock frequency, and redesigned and larger caches. Compute intensive workloads can achieve additional performance improvements through compiler enhancements, and larger caches can improve system performance on many production workloads. Each PU chip has up to six cores running at 5.5 GHz, which means the cycle time is slightly shorter then 0,18 ns. There are six PU chips on each MCM. The PU chips come in three versions, having four, five, or six active cores. For models H20, H43, H66, and H89, the processor units in the MCM in each book are implemented with 27 active cores per MCM. This means that model H20 has 27, model H43 has 54, model H66 has 81, and model H89 has 108 active cores. The model HA1 has six 30 active cores per MCM. This means that there are 120 active cores on model HA1. A schematic representation of the PU chip is shown in Figure 2-10.
42
8049ch02.fm
G X
i/o
CORE
CORE
CORE
M C
i/o
SC i/o
SC i/o
G X G X
i/o
L3 Cache
SC i/o
L3 Control
L3 Cache
SC i/o
M C
CORE
CORE
CORE
M C
i/o
Figure 2-10 PU chip diagram
Each PU chip has 2.75 billion transistors. Each one of the six cores has its own L1 cache with 64 KB for instructions and 96 KB for data. Next to each core resides its private L2 cache, with 1 MB for instructions and 1 MB for data respectively. There is one L3 cache, with 48 MB. This 48 MB L3 cache is a store-in shared cache across all cores in the PU chip. It has 192 x 512Kb eDRAM macros, dual address-sliced and dual store pipe support, an integrated on-chip coherency manager, cache and cross-bar switch. The L3 directory filters queries from local L4. Both L3 slices can deliver up to 160 GB/s bandwidth to each core simultaneously. The L3 cache interconnects the six cores, GX I/O buses, and memory controllers (MCs) with storage control (SC) chips. The memory controller (MC) function controls access to memory. The GX I/O bus controls the interface to the host channel adapters (HCAs) accessing the I/O. The chip controls traffic between the cores, memory, I/O, and the L4 cache on the SC chips. There are also one dedicated co-processors (CoP) for data compression and encryption functions for each core. The compression unit is integrated with the CP assist for cryptographic function (CPACF), benefiting from combining (or sharing) the use of buffers and interfaces. The assist provides high-performance hardware encrypting and decrypting support for clear key operations. For more information, see 3.3.3, Compression and cryptography accelerators on a chip on page 86.
2.4.2 Processor unit (core)

Each processor unit, or core, is a superscalar, out of order processor, having six execution units, as follows: Two fixed point (integer) Two load/store One binary floating point One decimal floating point Up to three instructions can be decoded per cycle and up to seven instructions/operations can be initiated to execute per clock cycle (<0.18 ns). The instructions execution can occur out of program order, as well as memory address generation and memory accesses can also occur out of program order. Each core has special circuitry to make execution and memory
43
8049ch02.fm
accesses appear in order to software. Not all instructions are directly executed by the hardware. This is the case of several complex instructions: some are execute by millicode and some are cracked into multiple operation, which are then executed by the hardware. The following functional areas are on each core, as shown in Figure 2-11: Instruction sequence unit (ISU): This unit enables the out-of-order (OOO) pipeline. It keeps track of register names, OOO instruction dependency, and handling of instruction resource dispatch. This unit is also central to performance measurement through a function called instrumentation. Instruction fetching unit (IFU) (prediction): These unit contain the instruction cache, branch prediction logic, instruction fetching controls, and buffers. Its relative size is the result of the elaborate branch prediction design, which is further described in 3.3.2, Superscalar processor on page 86. Instruction decode unit (IDU): The IDU is fed from the IFU buffers and is responsible for parsing and decoding of all z/Architecture operation codes. Load-store unit (LSU): The LSU contains the data cache and is responsible for handling all types of operand accesses of all lengths, modes and formats as defined in the z/Architecture. Translation unit (XU): The XU has a large translation look-aside buffer (TLB) and the Dynamic Address Translation (DAT) function that handles the dynamic translation of logical to physical addresses. Fixed-point unit (FXU): The FXU handles fixed point arithmetic. Binary floating-point unit (BFU): The BFU handles all binary and hexadecimal floating-point, and fixed-point multiplication and division operations. Decimal unit (DU): The DU executes both floating- point and fixed-point decimal operations. Recovery unit (RU): The RU keeps a copy of the complete state of the system, including all registers, collects hardware fault signals, and manages the hardware recovery actions. Dedicated Co-Processor (COP): The dedicated coprocessor is responsible for data compression and encryption functions for each core.
44
8049ch02.fm
IFU
RU
I D U
ISU
BFU FXU DF U
I-cache XU
LSU
Instr. L2
L2 Control
Data-L2 COP
Figure 2-11 Core layout
2.4.3 PU characterization
In each MCM, some PUs can be characterized for customer use. The characterized PUs can be used for general purpose to run supported operating systems (as z/OS, z/VM, Linux on System z), or specialized to run specific workloads (as Java, XML services, IPSec, some DB2 workloads) or functions (as Coupling Facility Control Code). For more information about PU characterization, see 3.4, Processor unit functions on page 91. The maximum number of characterized PUs depends on the zEC12 model. Some PUs are characterized by the system as standard system assist processors (SAPs), to run the I/O processing. Also as standard, there are at least two spare PUs per system, which are used to assume the function of a failed PU. The remaining installed PUs can be characterized for customer use. A zEC12 model nomenclature includes a number which represents this maximum number of PUs that can be characterized for customer use, as shown in Table 2-2.
Table 2-2 Number of PUs per zEC12 model Model H20 H43 H66 H89 HA1 Books 1 2 3 4 4 Installed PUs 27 (1 x 27) 54 (2 x 27) 81 (3 x 27) 108 (4 x 27) 120 (4 x 30) Standard SAPs 4 8 12 16 16 Min Spare PUs 2 2 2 2 2 Max characterized PUs 20 43 66 89 101
2.4.4 Storage control (SC) chip

The storage control (SC) chip uses the same CMOS 13S 32nm SOI technology, with 15 layers of metal. It measures 28.4 x 23.9 mm, has 3.3 billion transistors and 2.1 billion cells for eDRAM. Each MCM has two SC chips. The L4 cache on each SC chip has 192 MB, resulting in 384 MB of L4 cache shared per book.
45
8049ch02.fm
Figure 2-12 shows a schematic representation of the SC chip.
L4 Cache (48 MB)
L4 Cache (48 MB)
Fabric
IOs
T OD
Clk
Repower
Data BitStack
L4 Controller
Perv PLL Data BitPerv Stack Perv
F abric
IOs
L4 Cache (48 MB)
L4 Cache (48 MB)
Figure 2-12 SC chip diagram
Most of the SC chip space is taken by the L4 controller and the 192 MB L4 cache, which consists of four 48 MB quadrants with 256 (1.5 Mb) eDRAM macros per quadrant. The L4 cache is logically organized as 16 address sliced banks, with 24-way set associatively. The L4 cache controller is a single pipeline with multiple individual controllers, sufficient to handle 125 simultaneous cache transactions per chip. The L3 caches on PU chips communicate with the L4 caches via the attached SC chip using uni-diretional busses. L3 is divided into two logical slices, each slice is 24 MB consisting of two 12 MB banks. L3 is 12-way set associative, each bank has 4K sets, and the cache line size is 256B. The bus/clock ratio (2:1) between the L4 cache and the PU is controlled by the storage controller on the SC chip. The SC chip also acts as an L4 cache cross-point switch for L4-to-L4 traffic to up to three remote books by three bidirectional data buses. The integrated SMP fabric transport and system coherency manager use the L4 directory to filter snoop traffic from remote books, with a enhanced synchronous fabric protocol for improved latency and cache management. There are two clock domains and the clock function is distributed among both SC chips.
46
8049ch02.fm
2.4.5 Cache level structure

The zEC12 implements a four level cache structure, as shown in Figure 2-13.
PU Chip 6 Cores
L1 + + + + L1 2MB + + + + 2MB L2 L2 48 MB eDRAM Inclusive L3
PU Chip 6 Cores
PU Chip 6 Cores
PU Chip 6 Cores
PU Chip 6 Cores
PU Chip 6 Cores
+ Cach e for cores 1 to 6 LRU Cast- Out CP Stores Data Fetch Return
384MB eDRAM Inclusive L4 2 SC Chips
Figure 2-13 Cache levels structure
Each core has its own 160 KB cache Level 1 (L1), split into 96 KB for data (D-cache) and 64 KB for instructions (I-cache). The L1 cache is designed as a store-through cache, meaning that altered data is also stored to the next level of memory. The next level is the private cache Level 2 (L2) located on each core, having 2 MB, spitted into 1 MB D-cache and 1 MB I-cache and also designed as a store-through cache. The cache Level 3 (L3) is also located on the PUs chip and shared by the six cores, having 48 MB and designed as a store-in cache. Cache levels L2 and L3 are implemented on the PU chip to reduce the latency between the processor and the large shared cache L4, which is located on the two SC chips. Each SC chip has 192 MB, resulting in 384 MB of L4 cache, which is shared by all PUs on the MCM. The L4 cache uses a store-in design.
47
8049ch02.fm
2.5 Memory
Maximum physical memory size is directly related to the number of books in the system. Each book can contain up to 960 GB of physical memory, for a total of 3840 GB (3.75TB) of installed memory per system. AzEC12 has more memory installed than ordered. Part of the physical installed memory is used to implement the redundant array of independent memory (RAIM) design, resulting on up to 768 GB of available memory per book and up to 3072 GB (3 TB) per system. Table 2-3 on page 48 shows the maximum and minimum memory sizes customer can order for each zEC12 model.
Table 2-3 zEC12 memory sizes Model H20 H43 H66 H89 HA1 Number of books 1 2 3 4 4 Customer memory (GB) 32 - 704 32 - 1392 32 - 2272 32 - 3040 32 - 3040
The minimum physical installed memory is 80 GB per book, and the minimum initial amount of memory that can be ordered is 32 GB for all zEC12 models. The maximum customer memory size is based on the physical installed memory minus RAIM and minus HSA memory, which has a fixed amount of 32 GB. Table 2-4 shows the memory granularity based on the installed customer memory.
Table 2-4 Memory granularity Granularity (GB) 32 64 96 112 128 256 Customer memory (GB) 32 - 256 320 - 512 608 - 896 1008 1136 - 1520 1776 - 3056
With the zEC12 the memory granularity varies from 32 GB (for customer memory sizes from 32 to 256 GB) up to 256 GB (for CPCs having from 1776 GB to 3056 GB of customer memory).
48
8049ch02.fm
2.5.1 Memory subsystem topology

The zEC12 memory subsystem uses high speed, differential ended communications memory channels to link a host memory to the main memory storage devices. Figure 2-14 shows an overview of the book memory topology of a zEC12.
MCU2
MD21 MD22 MD23 MD24
MCU1
MD11 MD12 MD13 MD14 MD15 MD16 MD17 MD18 Master (a) Slave (b) MD19 MD20
MCU0
MD01 MD02 MD03 MD04 MD05 MD06 MD07 MD08 MD09 MD10 Channel 0
Channel 1
DIMMs
MD25 MD26 MD27 MD28 MD29 MD30
Channel 2
Channel 3
Channel 4
MCU2
PU2
MCU1
PU1
MCU0
PU0
GX2 GX6
GX1 GX7
GX0 PSI
MCM
SC1 PU3
SC0 SC0 PU4 PU5
GX3
GX4
GX5 PSI
Figure 2-14 Book memory topology
Each book has from 10 to 30 dual in-line memory modules (DIMMs). DIMMs are connected to the MCM through three memory control units (MCUs) located on PU0, PU1 and PU2. Each MCU uses five channels, one of them for RAIM implementation, on a 4 +1 (parity) design. Each channel has one or two chained DIMMs, so a single MCU can have five or ten DIMMs. Each DIMM has 4, 16 or 32 GB, and there is no mixing of DIMM sizes on a book.
2.5.2 Redundant array of independent memory (RAIM)

For a fully fault-tolerant N+1 design the zEC12 use the technology of redundant array of independent memory (RAIM). The RAIM design detects and recovers from DRAM, socket, memory channel or DIMM failures.
49
8049ch02.fm
The RAIM design requires the addition of one memory channel that is dedicated for RAS, as shown in Figure 2-15.
Level 4 Cache
16B 16B 16B 16B 16B 16B
Key Cache
Key Cache
Key Cache
MCU 0
2B
MCU 1
2B
MCU 2
2B
DATA CHECK
DATA CHECK ECC RAIM Parity
Extra column provides RAIM function
Figure 2-15 RAIM DIMMs
The parity of the four data DIMMs are stored in the DIMMs attached to the fifth memory channel. Any failure in a memory component can be detected and corrected dynamically. This design takes the RAS of the memory subsystem to another level, making it essentially a fully fault tolerant N+1 design.
2.5.3 Memory configurations

Memory sizes in each book do not have to be similar. Different books can contain different amounts of memory. Table 2-5 shows the physically installed memory on each book for all zEC12 models.
Table 2-5 Physically installed memory Memory (GB) 32 64 96 128 Model H20 Book 1 80a 100 160 240 Model H43 Book 1 40 60 80 100 Book 3 40 40 60 80 Model H66 Book 1 40 40 60 60 Book 2 40 40 40 60 Book 3 40 40 40 60 Model H89 Model HA1 Book 0 40 40 40 60 Book 1 40 40 40 40 Book 2 40 40 40 40 Book 3 40 40 40 40
50
8049ch02.fm
Memory (GB) 160 192 224 256 320 384 448 512 608 704 800 896 1008 1136 1264 1392 1520 1776 2032 2288 2544 2800 3056
Model H20 Book 1 240 320 320 400 480 640 640 800 800 960 N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A
Model H43 Book 1 120 160 160 240 240 320 320 400 400 480 640 640 640 800 800 960 960 N/A N/A N/A N/A N/A N/A Book 3 100 120 160 160 240 240 320 320 400 480 480 640 640 640 800 800 960 N/A N/A N/A N/A N/A N/A
Model H66 Book 1 80 100 100 120 160 240 240 240 320 320 400 400 480 480 640 640 640 800 960 960 N/A N/A N/A Book 2 80 80 100 120 160 160 240 240 240 320 320 400 400 480 480 640 640 800 800 960 N/A N/A N/A Book 3 60 80 100 100 100 160 160 240 240 320 320 400 400 480 480 480 640 640 800 960 N/A N/A N/A
Model H89 Model HA1 Book 0 60 80 80 100 120 160 160 240 240 240 320 320 320 400 400 480 480 640 640 800 800 960 960 Book 1 60 60 80 80 100 120 160 160 240 240 240 320 320 400 400 480 480 640 640 800 800 960 960 Book 2 60 60 80 80 100 120 160 160 160 240 240 320 320 320 400 400 480 480 640 640 800 800 960 Book 3 40 60 60 80 100 100 100 160 160 240 240 240 320 320 400 400 480 480 640 640 800 800 960
a. 80 GB for a one book system. However if System is ordered with >1 book, 40 GB installed
Physically, memory is organized as follows: A book always contains a minimum of 10 DIMMs of 4 GB each (40 GB). A book has more memory installed than enabled. The amount of memory that can be enabled by the customer is the total physically installed memory minus the RAIM amount and minus the 32 GB HSA memory. A book can have available unused memory, which can be ordered on a memory upgrade.
51
8049ch02.fm
Figure 2-16 illustrates how the physical installed memory is allocated on a zEC12, showing HSA memory, RAIM, customer memory, and the remaining available unused memory that can be enabled by licensed internal code (LIC) when required.
A vai lab le u nu Available sed unused memo ry memory
Physical installed memory
C usto mer
Customer o rd ered ordered mem ory memory
RAIM
RAIM
HSA 32 GB)
H SA (1 6 GB) (
Figure 2-16 Memory allocation diagram
As an example, a zEC12 model H43 (two books) ordered with 192 GB of memory would have memory sizes as follows. See Figure 2-16. Physical installed memory is 280 GB: 160 GB on book 1 and 120 GB on book 3. Book 1 has the 32 GB HSA memory and up to 96 GB for customer memory, while book 2 has up to 96 GB for customer memory, resulting in 192 GB of available memory for the customer. As the customer has ordered 176 GB, provided the granularity rules are met, there are still 16 GB (192 - 176 GB) available to be used in conjunction with additional memory for future upgrades by LIC. Memory upgrades are satisfied from already installed unused memory capacity until it is exhausted. When no more unused memory is available from the installed memory cards (DIMMs), one of the following additions must occur: Memory cards have to be upgraded to a higher capacity. An additional book with additional memory is necessary. Memory cards (DIMMs) must be added. A memory upgrade is concurrent when it requires no change of the physical memory cards. A memory card change is disruptive when no use is made of enhanced book availability. See 2.7.2, Enhanced book availability on page 60. When activated, a logical partition can use memory resources located in any book. No matter in which book the memory resides, a logical partition has access to that memory for up to a maximum of 1 TB. Despite the book structure, thezEC12 is still a symmetric multiprocessor (SMP). For more information, see 3.6, Logical partitioning on page 107.
52
8049ch02.fm
2.5.4 Memory upgrades

For a model upgrade that results in the addition of a book, the minimum memory increment is added to the system. As previously mentioned, the minimum physical memory size in book configuration is as follows: 1-book system has 80 GB of pysical memory 2-book system has 80 GB of pysical memory for each book 3-book and 4-book systems have 80 GB of pysical memory in the first book and second book, and 40 GB of pysical memory for the third book and forth book During a model upgrade, the addition of a book is a concurrent operation. The addition of the physical memory that is in the added book is also concurrent. If all or part of the additional memory is enabled for installation use (if it has been purchased), it becomes available to an active logical partition if this partition has reserved storage defined. For more information, see 3.6.3, Reserved storage on page 115. Alternately, additional memory can be used by an already-defined logical partition that is activated after the memory addition.
2.5.5 Book replacement and memory

With enhanced book availability as supported for zEC12 (see 2.7.2, Enhanced book availability on page 60), sufficient resources must be available to accommodate resources that are lost when a book is removed for upgrade or repair. Most of the time, removal of a book results in removal of active memory. With the flexible memory option (see 2.5.6, Flexible Memory Option on page 53), evacuating the affected memory and reallocating its use elsewhere in the system is possible. This requires additional available memory to compensate for the memory lost with the removal of the book.
2.5.6 Flexible Memory Option

With the Flexible Memory Option, additional physical memory is supplied to support activation of the actual purchased memory entitlement in the event of a single book failure, or to be available during an enhanced book availability action. When ordering memory, you can request additional flexible memory. The additional physical memory, if required, is calculated by the configurator and priced accordingly. Flexible memory is available only on the H43, H66, H89, and HA1 models. Table 2-6 shows the flexible memory sizes available for the zEC12.
Table 2-6 zEC12 memory sizes Model H20 H43 H66 H89 HA1 Standard memory (GB) 32 - 704 32 - 1392 32 - 2272 32 - 3040 32 - 3040 Flexible memory (GB) N/A 32 - 704 32 - 1392 32 - 2272 32 - 2272
53
8049ch02.fm
Table 2-7 shows the memory granularity for the Flexible Memory Option.
Table 2-7 Flexible memory granularity Granularity (GB) 32 64 96 112 128 256 Flexible memory (GB) 32 - 256 320 - 512 608 - 896a 1008 1136 - 1520b 1776 - 2288
a. Model H43 limit is 704 GB b. Model H66 limit is 1392 GB
Flexible memory can be purchased but cannot be used for normal everyday use. For that reason, a different purchase price for the flexible memory is offered to increase the overall availability of the system.
2.5.7 Preplanned Memory

Preplanned Memory provides the ability to plan for nondisruptive permanent memory upgrades. It differs from the flexible memory option. The flexible memory option is meant to anticipate nondisruptive book replacement. The usage of flexible memory is therefore temporary, in contrast with plan-ahead memory. When preparing in advance for a future memory upgrade, note that memory can be pre-plugged, based on a target capacity. The pre-plugged memory can be made available through a LIC configuration code (LICCC) update. You can order this LICCC through these channels: The IBM Resource Link (login is required): http://www.ibm.com/servers/resourcelink/ Your IBM representative The installation and activation of any pre-planned memory requires the purchase of the required feature codes (FC), described in table Table 2-8. The payment for plan-ahead memory is a two-phase process. One charge takes place when the plan-ahead memory is ordered, and another charge takes place when the prepaid memory is activated for actual usage. For the exact terms and conditions, contact your IBM representative.
54
8049ch02.fm
Table 2-8 Feature codes for plan-ahead memory Memory Pre-planned memory Charged when physical memory is installed. Used for tracking the quantity of physical increments of plan-ahead memory capacity. Pre-planned memory activation Charged when plan-ahead memory is enabled. Used for tracking the quantity of increments of plan-ahead memory that is being activated. zEC12 feature code FC 1996
FC 1901
Installation of pre-planned memory is done by ordering FC 1996. The ordered amount of plan-ahead memory is charged with a reduced price compared to the normal price for memory. One FC 1996 is needed for each 16 GB of usable memory (20 GB RAIM). Activation of installed pre-planned memory is achieved by ordering FC 1901, which causes the other portion of the previously contracted charge price to be invoiced. One FC 1901 is needed for each additional 16 GB to be activated. Memory: Normal memory upgrades use up the plan-ahead memory first.
2.6 Reliability, availability, serviceability (RAS)

IBM System z continues to deliver enterprise class RAS with the IBM zEnterprise EC12. But what is RAS really about? The main philosophy behind RAS is about preventing, or even masking outages. This can be planned- or unplanned outages. A few general examples of planned and unplanned outages are given (examples are not related to the RAS features of System z servers): A planned outage because of the addition of extra processor capacity A planned outage because of the addition of extra I/O cards An unplanned outage because of a failure of a power supply An unplanned outage because of a memory failure
Actually, it is highly unlikely that either one of the above examples could occur on a zEC12 server. The System z hardware has gone through decades of intense engineering, and this resulted in a very robust and reliable platform. Not only the hardware has a lot of RAS features built-in, but also the software has. IBMs parallel sysplex architecture is a good example of this.
2.6.1 RAS in the CPC memory subsystem

Patented error correction technology in the memory subsystem provides IBM's most robust error correction to date. Two full DRAM failures per rank can be spared and a third full DRAM failure corrected. DIMM level failures, including components such as the controller Application Specific Integrated Circuit (ASIC), the power regulators, the clocks and the board, can be corrected. Channel failures such as signal lines, control lines and drivers/receivers on the MCM can be corrected. Up stream and down stream data signals can be spared using two spare wires on both the upstream and downstream paths. One of these signals can be used to spare a clock signal line (one up stream and one down stream). Further improvements that were added by the zEC12 are:
55
8049ch02.fm
Improved error detection for L3/L4 eDRAM configuration latches. Improved error detection for L3/L4 wordline failures. Improved recovery for unresponsive memory DIMM ASIC. Improved recovery of memory command words by changing code points. Improved chip marking technology (marking a chip as defective).
Taken together, this provides the most robust memory subsystem of all generations of System z server.
2.6.2 General zEC12 RAS features

The zEC12 provides true N+1 (fully redundant) functionality for both radiator cooled and chilled water cooled models. In the unlikely case of a failure of both components (both water pumps or both radiators), there is backup in the form of air-cooling. The power supplies for the zEC12 are also based on the N+1 philosophy. A defect power supply will therefore not cause an unplanned outage of the machine. The IBM zEnterprise System CPCs have improved chip packaging (encapsulated chip connectors) and use soft error rate (SER) hardened latches throughout the design. Fully fault protected N+2 voltage transformation module (VTM) power conversion. This redundancy protects processor from loss of voltage due to VTM failures. System z uses triple redundancy on the environmental sensors (humidity and altitude) for reliability. Improved zBX Fibre Channel link/path testing and diagnostics Coupling Facility Control Code (CFCC) service enhancements: Structure control info to be included in CF dumps Enhanced tracing support Method to measure burst activity Trigger non-disruptive dumping for other soft-failure cases beyond break-duplexing
IBM zEnterprise EC12 continues to deliver robust server designs through exciting new technologies, hardening both new- and classic redundancy. For more information see Chapter 10, RAS on page 355.
56
8049ch02.fm
2.7 Connectivity
Connections to I/O cages, I/O drawers and Parallel Sysplex InfiniBand coupling (PSIFB) are driven from the host channel adapter (HCA) fanouts that are located on the front of the book, while connections to PCIe I/O drawers are driven from the PCIe fanouts also located on the front of the books. Figure 2-17 shows the location of the fanouts and connectors for a two-book system. In the figure, ECF is the External Clock Facility card for the Pulse Per Second (PPS) connectivity, OSC is the oscillator card; FSP is the flexible service processor; and LG is the location code for logic card.
ECF
OSC D1 I/O D2 I/O D3 FSP D4 FSP D5 I/O D6 I/O D7 I/O D8 I/O D9 I/O DA I/O
OSC
ECF D1 I/O D2 I/O D3 FSP D4 FSP D5 I/O D6 I/O D7 I/O D8 I/O D9 I/O DA I/O
filler
filler
LG01
LG06
LG10
LG15
Figure 2-17 Location of the host channel adapter fanouts
Each book has up to eight fanouts (numbered D1, D2, and D5 through DA). A fanout can be repaired concurrently with the use of redundant I/O interconnect. See 2.7.1, Redundant I/O interconnect on page 59. These are the six types of fanouts: Host Channel Adapter2-C (HCA2-C): Copper connections for InfiniBand I/O interconnect to all FICON, OSA, and Crypto cards in I/O cages and I/O drawers. PCIe fanout: Copper connections for PCIe I/O interconnect to all FICON, OSA, Crypto and Flash Express in PCIe I/O drawers. Host Channel Adapter2-O (HCA2-O): Optical connections for 12x InfiniBand for coupling links (IFB). The HCA2-O (12xIFB) provides a point-to-point connection over a distance of up to 150 m using OM3 fiber optic cables (50/125 m). zEC12 to z196, z114 or System z10 connections use 12-lane InfiniBand link at 6 GBps. Host Channel Adapter2-O Long Reach (HCA2-O LR):
57
8049ch02.fm
Optical connections for 1x IFB and supports IFB Long Reach (IFB LR) coupling links for distances of up to 10 km and up to 100 km when repeated through a System z qualified DWDM. IFB LR coupling links operate at up to 5.0 Gbps between two CPCs, or automatically scale down to 2.5 Gbps depending on the capability of the attached equipment. Host Channel Adapter3-O (HCA3-O): Optical connections for 12x IFB or 12x IFB3 for coupling links. For details, see 12x IFB and 12x IFB3 protocols on page 140. The HCA3-O (12xIFB) provides a point-to-point connection over a distance of up to 150 m using OM3 fiber optic cables (50/125 m). Host Channel Adapter3-O Long Reach (HCA3-O LR): Optical connections for 1x InfiniBand and supports IFB Long Reach (IFB LR) coupling links for distances of up to 10 km and up to 100 km when repeated through a System z qualified DWDM. IFB LR coupling links operate at up to 5.0 Gbps between two CPCs, or automatically scale down to 2.5 Gbps depending on the capability of the attached equipment. Fanout positions: On a model H20 and a model H43, all fanout positions can be populated. On a model H66, all fanout positions can be populated only on the first book. Positions D1 and D2 must remain free of fanouts on both the second and third books. On models H89 and HA1, all D1 and D2 positions must remain free of any fanout. When configuring for availability, the channels, coupling links, and OSAs should be balanced across books. In a system configured for maximum availability, alternate paths maintain access to critical I/O devices, such as disks, networks, and so on. Enhanced book availability (EBA) allows a single book in a multibook CPC to be concurrently removed and reinstalled for an upgrade or a repair. Removing a book means that the connectivity to the I/O devices connected to that book is lost. To prevent connectivity loss, the redundant I/O interconnect feature allows you to maintain connection to critical devices, except for Parallel Sysplex InfiniBand coupling, when a book is removed.
58
8049ch02.fm
2.7.1 Redundant I/O interconnect

Redundancy is provided for both InfiniBand I/O as well as for PCIe I/O interconnects.
InfiniBand I/O connection

Redundant I/O interconnect is accomplished by the facilities of the InfiniBand I/O connections to the InfiniBand Multiplexer (IFB-MP) card. Each IFB-MP card is connected to a jack located in the InfiniBand fanout of a book. IFB-MP cards are half-high cards and are interconnected with cards called STI-A8 and STI-A4, allowing redundant I/O interconnect in case the connection coming from a book ceases to function, as happens when, for example, a book is removed. A conceptual view of how redundant I/O interconnect is accomplished is shown in Figure 2-18.
CEC Cage 2827 Book 0 2827 Book 1

HCA-C
...
HCA-C
...
...
...
Up to 16 x 6 GB/sec per book

IFB-MP A IFB-MP B
Slot 5
Interconnect
I/O Cage or I/O Drawer
B andwidth to I/O card depends o n card type 2 GB /sec 1 GB/sec 500 MB/sec 333 MB/sec
Domain 0 Domain 1
Figure 2-18 Redundant I/O interconnect
Normally, the HCA2-C fanout in the first book connects to the IFB-MP (A) card and services domain 0 in a I/O cage or I/O drawer. In the same fashion, the HCA2-C fanout of the second book connects to the IFB-MP (B) card and services domain 1 in a I/O cage or I/O drawer. If the second book is removed, or the connections from the second book to the cage or drawer are removed, connectivity to domain 1 is maintained by guiding the I/O to domain 1 through the interconnect between IFB-MP (A) and IFB-MP (B).
PCIe I/O connection

The PCIe I/O Drawer supports up to 32 I/O cards. They are organized in 4 hardware domains per drawer as shown in Figure 2-19. Each domain is driven through a PCIe switch card. Always two PCIe switch cards provide a backup path for each other through the passive connection in the PCIe I/O Drawer backplane, so that in case of a PCIe fanout or cable failure, all 16 I/O cards in the two domains can be driven through a single PCIe switch card.
59
8049ch02.fm
To support Redundant I/O Interconnect (RII) between front to back domain pairs 0,1 and 2,3, the two interconnects to each pair must be driven from 2 different PCIe fanouts. Normally, each PCIe interconnect in a pair supports the eight I/O cards in its domain. In backup operation mode, one PCIe interconnect supports all 16 I/O cards in the domain pair.
Book 0 Memor y
PU PU PU
Book 1 Memory
PU PU PU
SC1, SC0 (F BC)

PU PU PU
SC1, SC0 (FBC)

PU PU PU
Fanouts
PCIe (8x)
x16 PCIe Gen2 8 GBp s
PCIe (8x)
PCIe switch
PCIe switch
PCIe switch
PCIe switch
4 GB/s PCIe Gen 2 x8
FICON Express8S
OSA-Express4S
PCIe I/O drawer
Figure 2-19 Redundant I/O interconnect for PCIe I/O drawer
2.7.2 Enhanced book availability

With enhanced book availability, the impact of book replacement is minimized. In a multiple book system, a single book can be concurrently removed and reinstalled for an upgrade or repair. Removing a book without affecting the workload requires sufficient resources in the remaining books. Before removing the book, the contents of the PUs and memory from the book to be removed must be relocated. Additional PUs must be available on the remaining books to replace the deactivated book, and sufficient redundant memory must be available if no degradation of applications is allowed. To ensure that the CPC configuration supports removal of a book with minimal impact to the workload, consider the flexible memory option. Any book can be replaced, including the first book, which initially contains the HSA. Removal of a book also removes the book connectivity to the I/O cages. The impact of the removal of the book on the system is limited by the use of redundant I/O interconnect, which is described in 2.7.1, Redundant I/O interconnect on page 59. However, all PSIFB links on the removed book must be configured offline. If the enhanced book availability and flexible memory options are not used when a book must be replaced (for example because of an upgrade or a repair action), the memory in the failing book is removed also. Until the removed book is replaced, a power-on reset of the system with the remaining books is supported.
60
8049ch02.fm
2.7.3 Book upgrade

All fanouts used for I/O and HCA fanouts used for IFB are concurrently rebalanced as part of a book addition.
2.8 Model configurations

When a zEC12 order is configured, PUs are characterized according to their intended usage. They can be ordered as any of the following items: CP The processor purchased and activated that supports the z/OS, z/VSE, z/VM, z/TPF, and Linux on System z operating systems. It can also run Coupling Facility Control Code.
Capacity marked CP A processor purchased for future use as a CP is marked as available capacity. It is offline and not available for use until an upgrade for the CP is installed. It does not affect software licenses or maintenance charges. IFL The Integrated Facility for Linux is a processor that is purchased and activated for use by z/VM for Linux guests and Linux on System z operating systems. A processor purchased for future use as an IFL. It is offline and cannot be used until an upgrade for the IFL is installed. It does not affect software licences or maintenance charges. An Internal coupling facility (ICF) processor purchased and activated for use by the Coupling Facility Control Code. A z196 Application Assist Processor (zAAP) purchased and activated to run eligible workloads such as Java code under control of z/OS JVM or z/OS XML System Services. A z196 Integrated Information Processor (zIIP) purchased and activated to run eligible workloads such as DB2 DRDA or z/OS3 Communication Server IPSec. An optional processor that is purchased and activated for use as a system assist processor (SAP).
Unassigned IFL
ICF zAAP
zIIP
Additional SAP
A minimum of one PU characterized as a CP, IFL, or ICF is required per system. The maximum number of CPs, IFLs, and ICFs is 101, The maximum number of zAAPs is 50, but requires an equal or greater number of characterized CPs. The maximum number of zIIPs is also 40 and requires an equal or greater number of characterized CPs. The sum of all zAAPs and zIIPs cannot be larger than two times the number of characterized CPs. Not all PUs on a given model are required to be characterized. The zEC12 model nomenclature is based on the number of PUs available for customer use in each configuration. The models are summarized in Table 2-9.
z/VM V5R4 and above support zAAP and zIIP processors for guest configurations.
61
8049ch02.fm
Table 2-9 z196 configurations Spares / reserved 2/ 1 2/ 1 2/ 1 2/ 1 2/ 1 Yes Yes Yes Yes Model Books PUs per MCM 27 54 81 108 120 CPs IFLs/ uIFL ICFs zAAPs zIIPs Add. SAPs Std. SAPs
H20 H43 H66 H89 HA1
1 2 3 4 4
020 043 066 089 0101
020 019 043 042 066 065 089 088 0101 0100
020 043 066 089 0101
010 021 033 044 044
010 021 033 044 044
034 08 012 016 016
4 8 12 16 16
A capacity marker identifies that a certain number of CPs have been purchased. This number of purchased CPs is higher than or equal to the number of CPs actively used. The capacity marker marks the availability of purchased but unused capacity intended to be used as CPs in the future. They usually have this status for software-charging reasons. Unused CPs are not a factor when establishing the MSU value that is used for charging MLC software, or when charged on a per-processor basis.
2.8.1 Upgrades
Concurrent upgrades of CPs, IFLs, ICFs, zAAPs, zIIPs, or SAPs are available for the zEC12. However, concurrent PU upgrades require that additional PUs are installed (at a prior time), but not activated. Spare PUs are used to replace defective PUs. There are always two spare PUs on a zEC12. Note that in the rare event of a PU failure, a spare PU is concurrently and transparently activated, and assigned the characteristics of the failing PU. If an upgrade request cannot be accomplished within the given configuration, a hardware upgrade is required. The upgrade enables the addition of one or more books to accommodate the desired capacity. Additional books can be installed concurrently. Although upgrades from one zEC12 model to another zEC12 model are concurrent, meaning that one or more books can be added, there is one exception. Upgrades from any zEC12 (model H20, H43, H66, H89) to a model HA1 is disruptive, because this upgrade requires the replacement of all four books. Table 2-10 shows the possible upgrades within the zEC12 configuration range.
Table 2-10 zEC12 to zEC12 upgrade paths To 2827 From 2827 Model H20 Model H43 Model H66 Model H89 Model H20 Model H43 Yes Model H66 Yes Yes Model H89 Yes Yes Yes Model HA1a
62
8049ch02.fm
a. disruptive upgrade
You can also upgrade a z10 EC or a z196 to a zEC12, while preserving the CPC serial number (S/N). The I/O cards can also be carried forward (with certain restrictions) to the zEC12. Attention: Upgrades from System z10 and System z196 are always disruptive. Upgrade paths from any z10 Enterprise Class to any zEC12 are supported as listed in Table 2-11.
Table 2-11 z10 EC to zEC12 upgrade paths To 2827 From 2097 Model E12 Model E26 Model E40 Model E56 Model E64 Model H20 Yes Yes Yes Yes Yes Model H43 Yes Yes Yes Yes Yes Model H66 Yes Yes Yes Yes Yes Model H89 Yes Yes Yes Yes Yes Model HA1 Yes Yes Yes Yes Yes
Upgrades from any z196 to any zEC12 are supported as listed in Table 2-12.
Table 2-12 z196 to z2120 upgrade paths To 2827 From 2817 Model M15 Model M32 Model M49 Model M66 Model M80 Model H20 Yes Yes Yes Yes Yes Model H43 Yes Yes Yes Yes Yes Model H66 Yes Yes Yes Yes Yes Model H89 Yes Yes Yes Yes Yes Model HA1 Yes Yes Yes Yes Yes
2.8.2 Concurrent PU conversions

Assigned CPs, assigned IFLs, and unassigned IFLs, ICFs, zAAPs, zIIPs, and SAPs can be converted to other assigned or unassigned feature codes. Most conversions are not disruptive. In exceptional cases, the conversion can be disruptive, for example, when a model H20 with 20 CPs is converted to an all IFL system. In addition, a logical partition might be disrupted when PUs must be freed before they can be converted. Conversion information is summarized in Table 2-13.
Table 2-13 Concurrent PU conversions To From CP IFL Yes Yes CP IFL Unassigned IFL Yes Yes ICF Yes Yes zAAP Yes Yes zIIP Yes Yes SAP Yes Yes
63
8049ch02.fm
To From Unassigned IFL ICF zAAP zIIP SAP
CP Yes Yes Yes Yes Yes
IFL Yes Yes Yes Yes Yes
Unassigned IFL Yes Yes Yes Yes
ICF Yes Yes Yes Yes
zAAP Yes Yes Yes Yes
zIIP Yes Yes Yes Yes
SAP Yes Yes Yes Yes -
2.8.3 Model capacity identifier

To recognize how many PUs are characterized as CPs, the store system information (STSI) instruction returns a value that can be seen as a model capacity identifier (MCI), which determines the number and speed of characterized CPs. Characterization of a PU as an IFL, an ICF, a zAAP, or a zIIP is not reflected in the output of the STSI instruction, because these have no effect on software charging. More information about the STSI output is in Processor identification on page 349. Four distinct model capacity identifier ranges are recognized (one for full capacity and three for granular capacity): For full-capacity engines, model capacity identifiers 701 to 7A1 are used. They express the 101 possible capacity settings from one to 101 characterized CPs. Three model capacity identifier ranges offer a unique level of granular capacity at the low end. They are available when no more than 20 CPs are characterized. These three subcapacity settings applied to up to twenty CPs offer 60 additional capacity settings. See Granular capacity on page 64.
Granular capacity
The zEC12 offers 60 capacity settings at the low end of the processor. Only 20 CPs can have granular capacity. When subcapacity settings are used, other PUs, beyond 20, can only be characterized as specialty engines. The three defined ranges of subcapacity settings have model capacity identifiers numbered from 401 to 420, 501 to 520, and 601 to 620. CPs: Within a zEC12, all CPs have the same capacity identifier. Specialty engines (IFLs, zAAPs, zIIPs, ICFs) operate at full speed.
List of model capacity identifiers

Table 2-14 shows that regardless of the number of books, a configuration with one characterized CP is possible. For example, model HA1 might have only one PU characterized as a CP.
Table 2-14 Model capacity identifiers zEC12 Model H20 Model H43 Model H66 Model capacity identifier 701720, 601620, 501520, 401420 701743, 601620, 501520, 401420 701766, 601620, 501520, 401420
64
8049ch02.fm
zEC12 Model H89 Model HA1
Model capacity identifier 701789, 601620, 501520, 401420 7017A1, 601620, 501520, 401420
Attention: On zEC12, model capacity identifier 400 is used for IFL or ICF only configurations.
2.8.4 Model capacity identifier and MSU value

All model capacity identifiers have a related MSU value (millions of service units), which are used to determine the software license charge for MLC software. Tables with MSU values are published on the Mainframe Exhibits for IBM Servers website, at: http://www-03.ibm.com/systems/z/resources/swprice/reference/exhibits/hardware.html
2.8.5 Capacity Backup

Capacity Backup (CBU) delivers temporary backup capacity in addition to what an installation might have already available in numbers of assigned CPs, IFLs, ICFs, zAAPs, zIIPs, and optional SAPs. The six CBU types are: CBU for CP CBU for IFL CBU for ICF CBU for zAAP CBU for zIIP Optional SAPs When CBU for CP is added within the same capacity setting range (indicated by the model capacity indicator) as the currently assigned PUs, the total number of active PUs (the sum of all assigned CPs, IFLs, ICFs, zAAPs, zIIPs, and optional SAPs) plus the number of CBUs cannot exceed the total number of PUs available in the system. When CBU for CP capacity is acquired by switching from one capacity setting to another, no more CBU can be requested than the total number of PUs available for that capacity setting.
CBU and granular capacity

When CBU for CP is ordered, it replaces lost capacity for disaster recovery. Specialty engines (ICFs, IFLs, zAAPs, and zIIPs) always run at full capacity, and also when running as CBU to replace lost capacity for disaster recovery. When you order CBU, specify the maximum number of CPs, ICFs, IFLs, zAAPs, zIIPs, and SAPs to be activated for disaster recovery. If disaster strikes, you decide how many of each of the contracted CBUs of any type must be activated. The CBU rights are registered in one or more records in the CPC. Up to eight records can be active, and that can contain a several CBU activation variations that apply to the installation. You can test the CBU. The number of free CBU test activations in each CBU record is now determined by the number of years that are purchased with the CBU record. (For example, a three year CBU record has three tests activations, a one year CBU record has one test activation.) You can increase the number of tests up to a maximum of 15 for each CBU record. The real activation of CBU lasts up to 90 days with a grace period of two days to
65
8049ch02.fm
prevent sudden deactivation when the 90-day period expires. The contract duration can be set from one to five years. The CBU record describes the following properties related to the CBU: Number of CP CBUs allowed to be activated Number of IFL CBUs allowed to be activated Number of ICF CBUs allowed to be activated Number of zAAP CBUs allowed to be activated Number of zIIP CBUs allowed to be activated Number of SAP CBUs allowed to be activated Number of additional CBU tests allowed for this CBU record Number of total CBU years ordered (duration of the contract) Expiration date of the CBU contract The record content of the CBU configuration is documented in IBM configurator output, shown in Example 2-1. In the example, one CBU record is made for a 5-year CBU contract without additional CBU tests for the activation of one CP CBU.
Example 2-1 Simple CBU record and related configuration features
On Demand Capacity Selecions: NEW00001 - CBU - CP(1) - Years(5) - Tests(5) Resulting feature numbers in configuration: 6817 6818 6820 Total CBU Years Ordered CBU Records Ordered Single CBU CP-Year 5 1 5
In Example 2-2, a second CBU record is added to the configuration in Example 2-1for two CP CBUs, two IFL CBUs, two zAAP CBUs, and two zIIP CBUs, with five additional tests and a 5-year CBU contract. The result is now a total number of 10 years of CBU ordered, which is the five years in the first record and five years in the second record. The two CBU records are independent and can be activate individually. Five additional CBU tests have been requested, and because there is a total of five years contracted for a total of three CP CBUs, two IFL CBUs, two zAAPs, and two zIIP CBUs, they are shown as 15, 10, 10, and 10 CBU years for their respective types.
Example 2-2 Second CBU record and resulting configuration features
NEW00001 - CBU - Replenishment is required to reactivate Expiration(06/21/2017) NEW00002 - CBU - CP(2) - IFL(2) - zAAP(2) - zIIP(2) Total Tests(10) - Years(5)
Resulting cumulative feature numbers in configuration: 6805 6817 6818 6820 6822 6826 6828 5 Additional CBU Tests Total CBU Years Ordered CBU Records Ordered Single CBU CP-Year Single CBU IFL-Year Single CBU zAAP-Year Single CBU zIIP-Year 5 10 2 15 10 10 10
66
8049ch02.fm
CBU for CP rules

Consider the following guidelines when planning for CBU for CP capacity: The total CBU CP capacity features are equal to the number of added CPs plus the number of permanent CPs changing capacity level. For example, if 2 CBU CPs are added to the current model 503, and the capacity level does not change, the 503 becomes 505: (503 + 2 = 505) If the capacity level changes to a 606, the number of additional CPs (3) are added to the 3 CPs of the 503, resulting in a total number of CBU CP capacity features of 6: (3 + 3 = 6) The CBU cannot decrease the number of CPs. The CBU cannot lower the capacity setting. Tip: Activation of CBU for CPs, IFLs, ICFs, zAAPs, zIIPs, and SAPs can be activated together with On/Off Capacity on Demand temporary upgrades. Both facilities can reside on one system and can be activated simultaneously.
CBU for specialty engines

Specialty engines (ICFs, IFLs, zAAPs, and zIIPs) run at full capacity for all capacity settings. This also applies to CBU for specialty engines. Table 2-15 shows the minimum and maximum (min-max) numbers of all types of CBUs that might be activated on each of the models. Note that the CBU record can contain larger numbers of CBUs than can fit in the current model.
Table 2-15 Capacity BackUp matrix Model Total PUs available 20 43 66 89 101 CBU CPs min-max 020 043 066 089 0101 CBU IFLs min-max 020 043 066 089 0101 CBU ICFs min-max 020 043 066 089 0101 CBU zAAPs min-max 010 021 033 044 050 CBU zIIPs min-max 010 021 033 044 050 CBU SAPs min-max 0-4 0-8 0-12 0-16 0-16
Model H20 Model H43 Model H66 Model H89 Model HA1
Unassigned IFLs are ignored. They are considered spares and are available for use as CBU. When an unassigned IFL is converted to an assigned IFL, or when additional PUs are characterized as IFLs, the number of CBUs of any type that can be activated is decreased.
2.8.6 On/Off Capacity on Demand and CPs

On/Off Capacity on Demand (CoD) provides temporary capacity for all types of characterized PUs. Relative to granular capacity, On/Off CoD for CPs is treated similarly to the way CBU is handled.
On/Off CoD and granular capacity

When temporary capacity requested by On/Off CoD for CPs matches the model capacity identifier range of the permanent CP feature, the total number of active CP equals the sum of the number of permanent CPs plus the number of temporary CPs ordered. For example,
67
8049ch02.fm
when a model capacity identifier 504 has two CP5s added temporarily, it becomes a model capacity identifier 506. When the addition of temporary capacity requested by On/Off CoD for CPs results in a cross-over from one capacity identifier range to another, the total number of CPs active when the temporary CPs are activated is equal to the number of temporary CPs ordered. For example, when a CPC with model capacity identifier 504 specifies six CP6 temporary CPs through On/Off CoD, the result is a CPC with model capacity identifier 606. A cross-over does not necessarily mean that the CP count for the additional temporary capacity will increase. The same 504 could temporarily be upgraded to a CPC with model capacity identifier 704. In this case, the number of CPs does not increase, but additional temporary capacity is achieved.
On/Off CoD guidelines

When you request temporary capacity, consider the following guidelines Temporary capacity must be greater than permanent capacity. Temporary capacity cannot be more than double the purchased capacity. On/Off CoD cannot decrease the number of engines on the CPC. Adding more engines than are currently installed is not possible. For more information about temporary capacity increases, see Chapter 9, System upgrades on page 311.
2.9 Power and cooling

zEC12 power and cooling system is a continuation of z196 with the expansion of some significant new developed technologies. The power service specifications of zEC12 are same as z196. InzEC12, new radiator unit was designed to replace the modular refrigeration unit (MRU) in z196. Water cooling system is still an option of zEC12. New Top Exit Power feature is available for zEC12, combined with Top Exit I/O Cabling feature, you will have more flexible choice to plan your computer room cabling. About more details of zEC12 Top Exit features, you can reference 11.3, IBM zEnterprise EC12 physical planning on page 382.
2.9.1 Power consumption

The system operates with two completely redundant power feeds. Each power feed is either 1 or 2 line cords. The number of line cords required depends on machine configuration (the number of BPR pairs installed). Line cords attach to either 3 phase, 50/60 Hz, 200 to 480 V AC power source or 380 to 520 V DC power source. There is no impact to system operation with the total loss of one power feed. A Balanced Power Plan Ahead feature is available for future growth also assuring adequate and balanced power for all possible configurations. With this feature downtimes for upgrading a server will eliminated by including the maximum power requirements in terms of Bulk Power Regulators (BPRs) and line cords to your installation. For ancillary equipment such as the Hardware Management Console, its display, and its switch, additional single phase outlets are required. The power requirements depend on the cooling facility installed and on number of books, and the number of I/O units installed. Information about the different requirements in relation to the number of installed I/O units can be found in 11.1.1, Power consumption on page 376.
68
8049ch02.fm
2.9.2 High voltage DC power

The High Voltage Direct Current power feature is an option for IBM zEnterprise EC12. It allows zEC12 to directly use the high voltage DC distribution as shown in Figure 2-20. A direct HV DC data center power design can improve data center energy efficiency by removing the need for a DC to AC inversion step. The zEC12 bulk power supplies have been modified to support HV DC so the only difference in shipped HW to implement the option is the DC line cords. Because HV DC is a new technology, there are multiple proposed standards. The zEC12 supports both ground referenced and dual polarity HV DC supplies. such as +/-190V or +/-260V, or +380V, and so on. Beyond the data center UPS and power distribution energy savings, a zEC12 run on HV DC power will draw 13% less input power. HV DC does not change the number of line cords a system requires.
Tr an sfer Sw itch
UPS
Step dow n
High Voltage AC Power
480 VAC 3 phase
AC/DC
480VAC
DC/AC
PDU
208 VA C 3 phase
AC/DC
4 10V DC
DC/DC
12V D C
Batteries
PSU
Lo sses ~ 13%
Server
Ste p down subs tation

T ran sf er Swit ch
480VAC 3 phas e
UPS AC/ DC DC/DC
12 V DC
520VDC
Batteri es
AC/DC
PSU
High Voltage DC Power

T ran sfer Switch
Lo sses ~4%
Server
UPS
D Input C
DC/DC
12V DC
DC/ DC
520 VDC S tep down s ubst ation

Batteries
AC/ DC
PSU
Server
Figure 2-20 AC versus DC Distribution
2.9.3 Internal Battery Feature (IBF)

IBF is an optional feature on the zEC12 server. See Figure 2-1 on page 32 for air cooled models and Figure 2-2 on page 33 for water cooled models for a pictorial view of the location of this feature. This optional IBF provides the function of a local uninterrupted power source. The IBF further enhances the robustness of the power design, increasing power line disturbance immunity. It provides battery power to preserve processor data in case of a loss of power on all two power feeds from the computer room. The IBF can hold power briefly during a brownout, or for orderly shutdown in case of a longer outage. The IBF provides up to 10 minutes of full power. More information about the hold times depending on the I/O configuration can be found in 11.1.2, Internal Battery Feature on page 377.
69
8049ch02.fm
2.9.4 Power capping and power saving

zEC12 support power capping, which gives the ability to limit the maximum power consumption and reduce cooling requirements (especially with zBX). To use power capping, FC 0020, Automate Firmware Suite, is required. This feature is used to enable the Automate suite of functionality associated with the zEnterprise Unified Resource Manager. The Automate suite includes representation of resources in a workload context, goal-oriented monitoring and management of resources, and energy management.
2.9.5 Power estimation tool

The Power estimation tool for the zEC12 allows you to enter your precise server configuration to produce an estimate of power consumption. Login to Resource Link with your userid. Navigate to Planning, then to Tools, then to Power Estimation Tools. Specify the quantity for the features that are installed in your machine. This tool estimates the power consumption for the specified configuration. The tool does not verify that the specified configuration can be physically built. Power: The exact power consumption for your machine will vary. The object of the tool is produce an estimation of the power requirements to aid you in planning for your machine installation. Actual power consumption after installation can be confirmed on the HMC Monitors Dashboard task.
2.9.6 Cooling
Water cooling technology is fully utilized inzEC12 MCM cooling. ForzEC12, it still has air cooled models and water cooled models.
Air cooled models

In zEC12, books, I/O drawers, and power enclosures are all cooled by forced air with blowers controlled by Move Device Assembly (MDA). The MCMs in books are cooled by water comes from radiator unit. Radiator unit is a new designed cooling component inzEC12 to replace modular refrigeration unit (MRU) as in z196. Accordingly, evaporator is redesigned and replaced by MCM cold plate. Internal closed loop water takes heat away from MCM by circulating between radiator heat exchanger and cold plate mounted on MCM. For more details, please reference 2.9.7, Radiator Unit. Even MCM is cooled by water, the heat finally exhausts into the room from radiator heat exchanger by forced air with blowers. From system level, it is still an air cooling system.
Water cooled models

Water cooling system is an option ofzEC12. With water cooling unit (WCU) technology, zEC12 can transfer the majority of the total heat generated to the building chilled water, effectively reducing heat output to room. Different with radiator in air cooled models, a WCU has two water loops, internal closed water loop and external chilled water loop. External water loop connects to customer supplied building chilled water, internal water loop circulates between WCU heat exchanger and MCM code plate, taking heat away from MCM and transfer it to external water loop in WCU heat exchanger. For more details, please reference 2.9.8, Water Cooling Unit (WCU).
70
8049ch02.fm
In addition to the MCMs, the internal water loop also circulates through two heat exchangers located in the path of the exhaust air in the rear of the frames. These heat exchangers will remove approximately 60% to 65% of the residual heat from the I/O drawers, the air cooled logic in the books and the heat generated within the power enclosures. Almost 2/3 of the total heat generated to be removed from the room by the chilled water. Selection of air cooled models or water cooled models is done at ordering and the appropriate equipment is factory installed. MES from air cooled model to water cooled model is not allowed, vice versa.
2.9.7 Radiator Unit

Radiator Unit provides cooling to MCM with closed loop water, no connection to external chilled water supply required. For zEC12, the internal circulating water is conditioned water and will be added to radiator unit during machine installation with specific Fill and Drain Tool (FC 3378). Water and Drain Tool will be shipped with new zEC12. As Figure 2-21 shows, water pumps, manifold assembly, radiator assembly (includes heat exchanger), and blowers are main components of zEC12 radiator unit.
Blow er s
Radiator Assem bly (Heat Exchanger)
Mani fo ld Assemb ly
Pum ps
Figure 2-21 Radiator unit
The radiator unit can connect to all four books and cool all MCMs simultaneously. The cooling capability is redundant design, a single working pump and blower can support entire load. Replacement of pump or blower is fully redundant, without performance impact. Figure 2-22 shows the closed water loop in radiator unit. The warm water exiting from the MCM code plates enters pumps through a common manifold, being pumped through a heat exchanger where heat extracted by the air flowing across the heat exchanger fins. The cooled water is then recirculated back into the MCM cold plates. The radiator air cooling system is open loop, that is to say there is no specific target temperature but only an estimated range based upon MCM power and ambient temperature. However the radiator blowers will be speed up in increment of air flow when MCM temperature increasing.
71
8049ch02.fm
MC M Cold Plate
Radiator Unit Blower
Book 0 Return Mani fold Pump Heat Exchang er
Book 1
Blower
Book 2 Pump Book 3
Figure 2-22 Radiator cooling system
Same as MRU design in z196, backup blowers is the redundant design of radiator unit in zEC12. If MCM temperature increases to threshold limit for any abnormal situations, such as radiator unit fails or environment temperature is too high, the backup blowers will be turned on to strengthen MCM cooling, and cycle steering mode is required. Please reference section 2.9.9, Backup air cooling system to learn more about backup blower and cycle steering mode. In zEC12 backup blowers are also provided primarily to permit concurrent replacement of the radiator (heat exchanger) or the return manifold. At this time, cycle steering mode may required.
2.9.8 Water Cooling Unit (WCU)

The zEC12 continues the ability to cool systems with building chilled water by employing the water cooling unit (WCU) technology. The MCM in the book is cooled by internal closed loop water coming from WCU. The internal closed loop water exchanges heat with building chilled water. The source of building chilled water is provided by customer. A very simplified high level diagram illustrating the principle is shown in Figure 2-23.
Customer water system
WCU Heat Ex changer
MCM Co ld Plate
Chilled
Cold
Internal water closed loop
Warmer
Pump
Warm er
Figure 2-23 WCU water loop
72
8049ch02.fm
The water in the closed loop within the system exchanges heat with the continuous supply of building chilled water. The internal water loop contains approximately 23 l (6 gallons) of conditioned water which circulates between the MCM cold plate and a heat exchanger within the WCU. Here the system water comes in 'contact' with the customer supplied chilled water. Heat from the MCM transfers to the cold plate where it is in turn transferred to the circulating system water. Finally the system water gives up its heat to the building chilled water within the WCU's heat exchanger and the MCM is efficiently cooled in this manner. The zEC12 operates with two fully redundant water cooling units (WCUs). These water cooling units each have their own facility feed and return water connections. If water is interrupted to one of the water cooling units, the other water cooling unit will pick up the entire load and the server will continue to operate without interruption. The customer is required to provide independent redundant water loops to the water cooling units in order to obtain the full redundancy. In the event of a total loss of building chilled water or in the unlikely event that both water cooling units fail, the backup blowers will be turned on to keep server running, same as air cooled models. At this time, cycle time degradation is required, please reference 2.9.9, Backup air cooling system for more details. The internal circulating water in water cooling unit is conditioned water and will be added to the two WCUs during machine installation with specific Fill and Drain Tool (FC 3378), same as radiator unit. Water and Drain Tool will be shipped with new zEC12.
Exhaust Air Heat Exchanger

InzEC12, all water cooled models, two Exhaust Air Heat Exchanger units are installed on the rear of the A&Z frames, as shown in Figure 2-24. It designed to remove heat from internal system water loop and internal air exiting the server into the hot air exhaust aisle.
Figure 2-24 Water cooled model - rear view
73
8049ch02.fm
In addition to the MCM cold plates, the internal water loop also circulates through these two heat exchangers located in the path of the exhaust air in the rear of the frames. These two heat exchangers will remove approximately 65% of the residual heat from the I/O drawers, the air cooled logic in the book and the heat generated within the power enclosures. The goal is for 2/3 of the total heat generated to be removed from the room by the chilled water. A diagrammatic view of the entire circulation system is shown in Figure 2-25.
Chil led water feed Nod Feed Manifold WCU 1 Chil led water return Chil led water feed WCU 2 Chil led water return Boo k 3 Boo k 0 Retur n Manifo ld A F rame Heat Exchanger
Boo k 1
Boo k 2
Z Frame Heat Exchanger
System F eed Manifold
Figure 2-25 WCU complete water loop
Above figure shows the complete water loop of WCU. Two customer water feeds connect to the two redundant WCUs. The WCUs feed water to a common feed manifold which supplies water to the frame heat exchangers. Two frame heat exchangers are fed in parallel from this manifold. The node feed manifold feeds the four book positions in parallel. The heat is exchanged with the customer supplied chilled water in the WCUs which also pump the system water around the loop. If one customer water supply or one WCU fails, the remaining feed will maintain MCM cooling. WCUs and the associated drive card are concurrently replaceable. Also, the heat exchangers can be disconnected and removed from the system concurrently.
Considerations before ordering

The water cooling option is preferable because it can substantially lower the total power consumption of the zEC12 and thus positively influence the total cost of ownership for the CPC. This is particularly true for the bigger models of the zEC12 (see Table 2-16). The water cooling option cannot be installed in the field, so careful consideration of present and future computer room and CPC configuration options should be exercised before deciding what cooling option to order. For more information, please reference section 11.1.4, Cooling requirements on page 378.
Table 2-16 Power consumption based on temperature Temperature 3 book typical configuration 4 book typical configuration 4 book maximum power configuration
Water cooled system power in normal room/hot room -est. 12.9 kW / 14.1 kW 17.4 kW / 19.0 kW 24.7 kW / 26.3 kW
74
8049ch02.fm
Temperature
3 book typical configuration
4 book typical configuration
4 book maximum power configuration
Inlet air temperature Heat to water and as % of total system heat load 18 C 23 C 27 C 32 C (hot room) 7.3 kW (57%) 9.5 kW (74%) 11.5 kW (89%) 14.8 kW (105%) 9.8 kW (56%) 12.6 kW (72%) 14.8 kW (85%) 18.2 kW (96%) 12.6 kW (51%) 15.6 kW (63%) 18.0 kW (73%) 21.6 kW (82%)
2.9.9 Backup air cooling system

The zEC12 has backup air cooling system design to lower power consumption of MCM, it works on both air cooled models and water cooled models. In zEC12, the radiator water cooling system of air cooled models or the WCU water cooling system of water cooled models is the primary cooling source of MCM. If water cooling system can not provide enough cooling capitally to maintain MCM temperature within a normal range, two backup blowers are switched on to compensate for the lost cooling capability with additional air cooling. At the same time, the oscillator card is set to a slower cycle time, slowing the system down to allow the degraded cooling capacity to maintain the proper temperature range of MCM. Running at a slower clock speed, the MCM produces less heat. The slowdown process is done in steps, based on the temperature of the MCM.
75
8049ch02.fm
2.10 Summary of zEC12 structure

Table 2-17 summarizes all aspects of the zEC12 structure.
Table 2-17 System structure summary Description Number of MCMs Total number of PUs Maximum number of characterized PUs Number of CPs Number of IFLs Number of ICFs Number of zAAPs Number of zIIPs Standard SAPs Additional SAPs Standard spare PUs Enabled memory sizes L1 cache per PU L2 cache per PU L3 shared cache per PU chip L4 shared cache Cycle time (ns) Clock frequency Maximum number of fanouts I/O interface per IFB cable I/O interface per PCIe cable Number of support elements External AC power Optional external DC Internal Battery Feature Model H20 1 27 20 0-20 0-20 0-20 0-10 0-10 4 04 2 32704 GB 64-I/96-D KB 1-I/1-D MB 48MB 384 MB 0.178 5.5 GHz 8 6 GBps 8 GBps 2 3 phase 570V/380V Optional Model H43 2 54 43 0-43 0-43 0-43 0-21 0-21 8 08 2 3211392 GB 64-I/96-D KB 1-I/1-D MB 48MB 384 MB 0.178 5.5 GHz 16 6 GBps 8 GBps 2 3 phase 570V/380V Optional Model H66 3 81 66 0-66 0-66 0-66 0-33 0-33 12 012 2 322272 GB 64-I/96-D KB 1-I/1-D MB 48MB 384 MB 0.178 5.5 GHz 20 6 GBps 8 GBps 2 3 phase 570V/380V Optional Model H89 4 108 89 0-89 0-89 0-89 0-44 0-44 16 016 2 323040 GB 64-I/96-D KB 1-I/1-D MB 48MB 384 MB 0.178 5.5 GHz 24 6 GBps 8 GBps 2 3 phase 570V/380V Optional Model HA1 4 120 101 0-101 0-101 0-101 0-50 0-50 16 016 2 323040 GB 64-I/96-D KB 1-I/1-D MB 48MB 384 MB 0.178 5.5 GHz 24 6 GBps 8 GBps 2 3 phase 570V/380V Optional
76
8049ch03.fm
Chapter 3.
CPC System design

This chapter explains how the zEC12 processor unit is designed. This information can be used to understand the functions that make the zEC12 a system that suits a broad mix of workloads for large enterprises. This chapter discusses the following topics: 3.1, Design highlights on page 78 3.2, Book design on page 79 3.3, Processor unit design on page 82 3.4, Processor unit functions on page 91 3.5, Memory design on page 103 3.6, Logical partitioning on page 107 3.7, Intelligent resource director on page 117 3.8, Clustering technology on page 118 The design of the zEC12 symmetric multiprocessor (SMP) is the next step in an evolutionary trajectory stemming from the introduction of CMOS technology back in 1994. Over time the design has been adapted to the changing requirements dictated by the shift towards new types of applications that customers are becoming more and more dependent on. The zEC12 offers very high levels of serviceability, availability, reliability, resilience, and security, and fits in the IBM strategy in which mainframes play a central role in creating an intelligent, energy efficient, and integrated infrastructure. The zEC12 is designed in such a way that not only is the system considered important for the infrastructure, but also everything around it (operating systems, middleware, storage, security, and network technologies supporting open standards) in helping customers achieve their business goals. The modular book design aims to reduce or, in some cases, even eliminate, planned and unplanned outages by offering concurrent repair, replace, and upgrade functions for processors, memory, and I/O. For more information about the zEC12 RAS features see Chapter 10, RAS on page 355. The zEC12 with its ultra-high frequency, very large high speed buffers (caches) and memory, superscalar processor design, out-of-order core execution, and flexible configuration options is the next implementation to address the ever-changing IT environment.
77
8049ch03.fm
3.1 Design highlights

The physical packaging of the zEC12 is comparable to the packaging used for z196 systems. Its modular book design creates the opportunity to address the ever-increasing costs related to building systems with ever-increasing capacities. The modular book design is flexible and expandable, offering unprecedented capacity to meet consolidation needs, and might contain even larger capacities in the future. zEC12 continues the line of backward-compatible mainframe processors, introducing more complex instructions executed by millicode and additional complex instructions broken down into multiple operations. It uses 24, 31, and 64-bit addressing modes, multiple arithmetic formats, and multiple address spaces for robust inter-process security. The main objectives of the zEC12 system design, which are discussed in this and subsequent chapters, are as follows: Offer a flexible infrastructure to concurrently accommodate a wide range of operating systems and applications, from the traditional systems (for example z/OS and z/VM) to the world of Linux and cloud computing. Offer state-of-the-art integration capability for server consolidation, using virtualization capabilities, such as: Logical partitioning, which allows 60 independent logical servers. z/VM, which can virtualize hundreds to thousands of servers as independently running virtual machines. HiperSockets, which implement virtual LANs between logical partitions within the system. This allows for a logical and virtual server coexistence and maximizes system utilization and efficiency, by sharing hardware resources. Offer high performance to achieve the outstanding response times required by new workload-type applications, based on high frequency, superscalar processor technology, improved out-of-order core execution, large high speed buffers (cache) and memory, architecture, and high bandwidth channels, which offer second-to-none data rate connectivity. Offer the high capacity and scalability required by the most demanding applications, both from single-system and clustered-systems points of view. Offer the capability of concurrent upgrades for processors, memory, and I/O connectivity, avoiding system outages in planned situations. Implement a system with high availability and reliability, from the redundancy of critical elements and sparing components of a single system, to the clustering technology of the Parallel Sysplex environment. Have broad internal and external connectivity offerings, supporting open standards such as Gigabit Ethernet (GbE), and Fibre Channel Protocol (FCP). Provide leading cryptographic performance in which every PUs has a dedicated CP Assist for Cryptographic Function (CPACF). Optional Crypto Express features with Cryptographic Coprocessors providing the highest standardized security certification1 and Cryptographic Accelerators for Secure Sockets Layer Transport Layer Security (SSL/TLS) transactions can be added. Be self-managing and self-optimizing, adjusting itself on workload changes to achieve the best system throughput, through the Intelligent Resource Director or the Workload Manager functions, assisted by HiperDispatch.
1
78
8049ch03.fm
Have a balanced system design, providing large data rate bandwidths for high performance connectivity along with processor and system capacity. The following sections describe the zEC12 system structure, showing a logical representation of the data flow from PUs, caches, memory cards, and a variety of interconnect capabilities.
3.2 Book design

A zEC12 system has up to four books in a fully connected topology (see Figure 3-3 on page 82), up to 101 processor units that can be characterized, and up to 3 TB of memory capacity. Memory has up to 12 memory controllers, using 5-channel redundant array of independent memory (RAIM) protection, with DIMM bus cyclic redundancy check (CRC) error retry. The 4-level cache hierarchy is implemented with eDRAM (embedded) caches. Until recently, eDRAM was considered to be too slow for this kind of use, however, a break-through in technology made by IBM has negated that. In addition, eDRAM offers higher density, less power utilization, fewer soft-errors, and better performance. Concurrent maintenance allows dynamic book add and repair. The zEC12 uses 32nm chip technology, with advanced low latency pipeline design, leveraging high speed yet power efficient circuit designs. The multichip module (MCM) has a dense packaging, allowing closed water loop cooling. The head exchange from the closed loop is either air cooled by a radiator unit or as an option, water cooled. The water cooling option is best, because it can lower the total power consumption of the machine. This is particularly true for the larger configurations as discussed in 2.9.6, Cooling on page 70.
3.2.1 Cache levels and memory structure

The zEC12 memory subsystem focuses on keeping data closer to the processor unit, increasing all cache levels beginning from L2, in which chip-level shared cache (L3) and book-level shared cache (L4) is doubled in size. Figure 3-1 shows the zEC12 cache levels and memory hierarchy.
Chapter 3. CPC System design
79
8049ch03.fm
Main Storage DRAM
0.75 TB per book / over 30 DIMMs / 3.0 TB per CEC
DIMMs One cache per Book over two SC Chip (fully shared) SC Chips One per PU Chip (shared by 6 cores) up to 30 (PUs)
384 MB
Level 4 eDRAM
up to four (books)
Level 3
48 MB
eDRAM
Level 2 Level 1
1 MB / 1 MB 64 KB I-cache 96 KB D-cache
SRAM
S RAM
1 per core, 6 per PU Chip PU Chip
Figure 3-1 zEC12 cache levels and memory hierarchy
The 4-level cache structure is implemented within the MCM. The first three levels (L1, L2, and L3) are located on each PU chip and the last level (L4) resides on SC chips: L1 and L2 caches use static random access memory (SRAM) and are private for each core. L3 cache uses embedded dynamic static random access memory (eDRAM) and is shared by all six cores within the PU chip. Each book has six L3 caches and a four-book system has 24 of them, resulting in 1152 MB (48 x 24 MB) of this shared PU chip level cache. L4 cache also uses eDRAM and is shared by all PU chips on the MCM. A four-book system has 1536 MB (4 x 384 MB) of shared L4 cache. Main storage has up to 0.75 TB per book, using up to 30 DIMMs. A four-book system can have up to 3 TB of main storage. Cache sizes are being limited by ever-diminishing cycle times because they must respond quickly without creating bottlenecks. Access to large caches costs more cycles. Instruction and data caches (L1) sizes must be limited because larger distances must be traveled to reach long cache lines. This L1 access time should occur in one cycle, avoiding increased latency. Also, the distance to remote caches as seen from the microprocessor becomes a significant factor. An example of this is the L4 cache that is not on the microprocessor (and might not even be in the same book). Although the L4 cache is rather large, the reduced cycle time has the effect that more cycles are needed to travel the same distance. In order to overcome this and avoid potential latency, zEC12 uses two additional cache levels (L2 and L3) within the PU chip, with denser packaging. This design reduces traffic to and from the shared L4 cache, which is located on another chip (SC chip). Only when there is a cache miss in L1, L2, and L3, the request is sent to L4. L4 is the coherence manager, meaning that all memory fetches must be in the L4 cache before that data can be used by the processor.
80
8049ch03.fm
Another approach is available for avoiding L4 cache access delays (latency) as much as possible. The L4 cache straddles up to four books. This means relatively large distances exist between the higher-level caches in the processors and the L4 cache content. To overcome the delays that are inherent to the book design and to save cycles to access the remote L4 content, it is beneficial to keep instructions and data as close to the processors as possible by directing as much work of a given logical partition workload on the processors located in the same book as the L4 cache. This is achieved by having the PR/SM scheduler and the z/OS dispatcher work together to keep as much work as possible within the boundaries of as few processors and L4 cache space (which is best within a book boundary) as can be achieved without affecting throughput and response times. Preventing PR/SM and the dispatcher from scheduling and dispatching a workload on any processor available, and keeping the workload in as small a portion of the system as possible, contributes to overcoming latency in a high-frequency processor design such as the zEC12. The cooperation between z/OS and PR/SM has been bundled in a function called HiperDispatch. HiperDispatch exploits the zEC12 cache topology, with reduced cross-book help, and better locality for multi-task address spaces. For more information on HiperDispatch, see 3.6, Logical partitioning on page 107. Figure 3-2 compares the cache structures of the zEC12 with the System z previous generation system, the z196.
z196
4 L4 Cac hes 192 MB Shared eDRAM L4
2 4 MB Sh r eD RAM L3
L2 L2 L2 L2 L1 L1 L1 L1
EC12
4 L4 Caches 384 MB Shar ed eDRAM L4
48 MB Shr eDR AM L 3
L2 L2 L2 L2 L1 L1 L1 L1 L2 L1 L2 L1
24 MB Sh r 6 L3s, eD RAM L3 24 L1 / L2s

L2 L1 L2 L1
6 L3s, 3 6 L1 / L2s
48 MB Shr eD RAM L3
L2 L2 L1 L1 L2 L2 L2 L2 L1 L1 L1 L1
L2 L2 L1 L1
L1 :
64 K I + 12 8 KD 8w DL1, 4 w IL1 25 6 B line size Private 1 .5 MB Inclus ive of L1s 12 w Set As soc iative 25 6 B c ac he line s ize Sha red 24 MB Inc lusive of L2s 12 w Set As soc iative 25 6 B c ac he line s ize 19 2 MB Inclus iv e 24 w Set As soc iative 25 6 B c ac he line s ize
L1:
64 K I + 96 K D 8w DL1, 4 w IL1 25 6 B line size Private 1 MB Inclusive of DL1 Private 1 MB Inclusive of IL1 8w Set Ass oc ia tiv e 25 6 B c ac he line s ize Sha red 48 MB Inc lusiv e of L2s 12 w Set As sociative 25 6 B c ac he line s ize 38 4 MB Inclus ive 24 w Set As sociative 25 6 B c ac he line s ize
L2
L2
L3
L3
L4
L4
Figure 3-2 zEC12 and z196 cache levels comparison
Compared to z196, the zEC12 cache design has much larger cache level sizes, except for the L1 private cache on each core (as its access time should occur in one cycle). The zEC12 cache level structure is focused on keeping more data closer to the processor unit. This design can improve system performance on many production workloads.
81
8049ch03.fm
3.2.2 Book interconnect topology

Books are interconnected in a point-to-point connection topology, allowing every book to communicate with every other book. Data transfer never has to go through another book (cache) to address the requested data or control information. Figure 3-3 shows a simplified topology for a four-book system.
6PU
6PU
6PU
6PU
6PU
6PU
L4 cache
6PU 6PU 6PU
L4 cache
6PU 6PU 6PU
6PU
6PU
6PU
6PU
6PU
6PU
L4 cache
6PU 6PU 6PU
L4 cache
6PU 6PU 6PU
Figure 3-3 Point-to-point topology for book-to-book communication
Inter-book communication takes place at the L4 cache level, which is implemented on Storage Controller (SC) cache chips in each MCM. The SC function regulates coherent book-to-book traffic.
3.3 Processor unit design

Today, systems design is driven by processor cycle time, though this does not automatically mean that the performance characteristics of the system improve. Processor cycle time is especially important for CPU-intensive applications. The System z10 EC introduced a dramatic PU cycle time improvement, and its succeeding generation, the z196, lowers it even further, reaching 0.192 nanoseconds (5.2 GHz). The zEC12 improved even this world class leading numbers to 0,178 ns (5.5 GHz). Besides the cycle time, other processor design aspects, such as pipeline, execution order, branch prediction, and high speed buffers (caches), also greatly contribute to the performance of the system. The zEC12 processor unit core is a superscalar, out of order processor, with six execution units where, of the instruction which are not directly executed by the hardware, some are executed by millicode, and still others cracked into multiple operations. zEC12 introduces architectural extensions, with instructions designed to allow reduced processor quiesce effects, reduced cache misses, and reduced pipeline disruption. The zEC12 new architecture includes the following features: Improvements in branch prediction and handling Performance per watt improvements, compared to the z196 system 82
8049ch03.fm
Numerous improvements in the out of order design Enhanced instruction dispatch and grouping efficiency Enhanced branch prediction structure and sequential instruction fetching Millicode improvements Transactional execution (TX) facility Runtime instrumentation (RI) facility Enhanced DAT-2 for 2GB pages support Decimal floating point (DFP) improvements. The zEC12 enhanced instruction set architecture (ISA) includes a set of instructions added to improve compiled code efficiency. This results on optimized processor units to meet the demands of a wide variety of business workload types without compromising the performance characteristics of traditional workloads.
3.3.1 Out-of-order execution

The z196 was the first System z to implement an out-of-order (OOO) core. The zEC12 improves this technology by increasing the OOO resources, increasing the execution and completion throughput and improving the instruction dispatch and grouping efficiency. OOO is yields significant performance benefit for compute intensive applications through re-ordering instruction execution, allowing later (younger) instructions to be executed ahead of a stalled instruction, and re-ordering storage accesses and parallel storage accesses. OOO maintains good performance growth for traditional applications. Out-of-order execution can improve performance in the following ways: Re-ordering instruction execution: Instructions stall in a pipeline because they are waiting for results from a previous instruction or the execution resource they require is busy. In an in-order core, this stalled instruction stalls all later instructions in the code stream. In an out-of-order core, later instructions are allowed to execute ahead of the stalled instruction. Re-ordering storage accesses: Instructions which access storage can stall because they are waiting on results needed to compute storage address. In an in-order core, later instructions are stalled. In an out-of-order core, later storage-accessing instructions which can compute their storage address are allowed to execute. Hiding storage access latency: Many instructions access data from storage. Storage accesses can miss the L1 and require 7 to 50 additional cycles to retrieve the storage data. In an in-order core, later instructions in the code stream are stalled. In an out-of-order core, later instructions that are not dependent on this storage data are allowed to execute. The zEC12 is the second generation of out of order (OOO) System z processor design with advanced micro-architectural innovations providing: Maximization of ILP for a better CPI design by reviewing every part of the z196 design Maximization of performance per watt. Two cores are added, as compared to the z196 chip, at slightly higher chip power (~300 watts) Enhancements of instruction dispatch and grouping efficiency
83
8049ch03.fm
Increased OOO resources (Global Completion Table entries, physical GPR entries, physical FPR entries) Improved completion rate Reduced cache/TLB miss penalty Improved execution of D-Cache store and reload and new Fixed-point divide. New OSC (load-hit-store conflict) avoidance scheme Enhanced branch prediction structure and sequential instruction fetching.
Program results
The OOO execution does not change any program results. Execution can occur out of (program) order, but all program dependencies are honored, ending up with same results of the in order (program) execution. This implementation requires special circuitry to make execution and memory accesses appear in order to software. The logical diagram of a zEC12 core is shown in Figure 3-4.
Out Of Order (OOO) addition
Clump of instruction (up to 3)
Decode/Dispatch1 InstQ GCT Rename VBQ VBU Issue Queue Fixed Pt GPR Fixed Pt
Group of micro-ops (up to 3)
0cycE
Group of micro-ops (up to 3)
Issue Queue Load/ Store2

TLB
TLB0
VBQ
micro-ops issue (up to 7)
Load/ Store2
Bin/Hex Floating Pt Dec Flt Pt Fx Div
VBU
on-the-fly OSC detection bubble-free instruction expansion 2) banking for concurrent load/stores prefetching enhancements
1)
L1 Dir
L2 Dir
FPR
Figure 3-4 zEC12 PU core logical diagram
Memory address generation and memory accesses can occur out of (program) order. This capability can provide a greater exploitation of the zEC12 superscalar core, and can improve system performance. Figure 3-5 shows how OOO core execution can reduce the execution time of a program.
84
8049ch03.fm
In-or der core execution

Instructions 1 2 3 4 5 6 7 T ime
L1 m iss
z196 Out-of-order core execution

Instructions 1 2 3 4 5 6 7 Instructions 1 2
L1 m iss L1 mis s
Shorter L1 Miss latency
Faster millicode execution
Dependency Execution Storage access
3 4 5 6 7 Time
Better Instruction Delivery
EC12 Out-of-order core execution
Figure 3-5 In-order and zEC12 out-of-order core execution improvements
The left side of the example shows an in-order core execution. Instruction 2 has a big delay due to an L1 cache miss, and the next instructions wait until instruction 2 finishes. In the usual in-order execution, the next instruction waits until the previous one finishes. Using OOO core execution, shown on the right side of the example, instruction 4 can start its storage access and execution while instruction 2 is waiting for data, if no dependencies exist between both instructions. So, when the L1 cache miss is solved, instruction 2 can also start its execution while instruction 4 is executing, Instruction 5 might need the same storage data required by instruction 4, and as soon as this data is on L1 cache, instruction 5 starts execution at the same time. The zEC12 superscalar PU core can have up to seven instructions/operations in execution per cycle. Compared to the z196, the first IBM System z CPC that used the OOO technology, further enhancements to the execution cycle are integrated in the cores, which results in a shorter execution time.
Example of branch prediction

If the branch prediction logic of the microprocessor makes the wrong prediction, removing all instructions in the parallel pipelines might be necessary. The wrong branch prediction is more costly in a high-frequency processor design, as discussed previously. Therefore, the branch prediction techniques used are very important to prevent as many wrong branches as possible. For this reason, a variety of history-based branch prediction mechanisms are used, as shown on the in-order part of the zEC12 PU core logical diagram in Figure 3-4 on page 84. The branch target buffer (BTB) runs ahead of instruction cache pre-fetches to prevent branch misses in an early stage. Furthermore, a branch history table (BHT) in combination with a pattern history table (PHT) and the use of tagged multi-target prediction technology branch prediction offer an extremely high branch prediction success rate. zEC12 microprocessor improves the branch prediction structure by increasing the size of the branch buffer (BTB2), having a faster prediction throughput in BTB1 by usage of a fast re-indexing table (FIT), and improving the sequential instruction stream delivery.
85
8049ch03.fm
3.3.2 Superscalar processor

A scalar processor is a processor that is based on a single-issue architecture, which means that only a single instruction is executed at a time. A superscalar processor allows concurrent execution of instructions by adding additional resources onto the microprocessor to achieve more parallelism by creating multiple pipelines, each working on its own set of instructions. A superscalar processor is based on a multi-issue architecture. In such a processor, where multiple instructions can be executed at each cycle, a higher level of complexity is reached, because an operation in one pipeline stage might depend on data in another pipeline stage. Therefore, a superscalar design demands careful consideration of which instruction sequences can successfully operate in a long pipeline environment. On the zEC12, up to three instructions can be decoded per cycle and up to seven instructions or operations can be in execution per cycle. Execution can occur out of (program) order.
Challenges of creating a superscalar processor: Many challenges exist in creating an efficient superscalar processor. The superscalar design of the PU has made big strides in avoiding address generation interlock (AGI) situations. Instructions requiring information from memory locations can suffer multi-cycle delays to get the desired memory content, and because high-frequency processors wait faster, the cost of getting the information might become prohibitive.
3.3.3 Compression and cryptography accelerators on a chip

This section describes the compression and cryptography features.
Coprocessor units
There is one coprocessor (CoP) unit for compression and cryptography on each core in the chip (see Figure 3-6 on page 87). The compression engine uses static dictionary compression and expansion. The dictionary size is up to 64KB, with 8K entries, and has a local 16 KB cache for dictionary data. The cryptography engine is used for CP assist for cryptographic function (CPACF), which offers a set of symmetric cryptographic functions for high encrypting and decrypting performance of clear key operations.
86
8049ch03.fm
IF U
I D U
RU IS U
F XU D F U B FU
I-c a c h e XU In s tr. L2
L SU
L2 C o n tro l
D a ta -L 2
CO P
Figure 3-6 Compression and cryptography accelerators on a core in the chip
CP assist for cryptographic function

The CP assist for cryptographic function (CPACF) accelerates the encrypting and decrypting of SSL/TLS transactions, VPN-encrypted data transfers, and data-storing applications that do not require FIPS2 140-2 level 4 security. The assist function uses a special instruction set for symmetrical clear key cryptographic encryption and decryption, as well as for hash operations. This group of instructions is known as the Message-Security Assist (MSA). For information about these instructions, see z/Architecture Principles of Operation, SA22-7832. For more information about cryptographic functions on zEC12, see Chapter 6, Cryptography on page 183.
3.3.4 Decimal floating point accelerator

The decimal floating point (DFP) accelerator function is present on each of the microprocessors (cores) on the quad-core chip. Its implementation meets business application requirements for better performance, precision, and function. Base 10 arithmetic is used for most business and financial computation. Floating point computation that is used for work typically done in decimal arithmetic has involved frequent necessary data conversions and approximation to represent decimal numbers. This has made floating point arithmetic complex and error-prone for programmers using it for applications in which the data is typically decimal data. Hardware decimal-floating-point computational instructions provide data formats of 4, 8, and 16 bytes, an encoded decimal (base 10) representation for data, instructions for performing
2
87
8049ch03.fm
decimal floating point computations, and an instruction that performs data conversions to and from the decimal floating point representation. The decimal floating point (DFP) architecture on zEC12 has been improved to facilitate better performance on traditional zoned-decimal operations for Cobol programs. Additional instructions are provided to convert zoned-decimal data into DFP format in FPR registers.
Benefits of the DFP accelerator

The DFP accelerator offers the following benefits: Avoids rounding issues such as those happening with binary-to-decimal conversions. Has better functionality over existing binary coded decimal (BCD) operations. Follows the standardization of the dominant decimal data and decimal operations in commercial computing supporting industry standardization (EEE 745R) of decimal floating point operations. Instructions are added in support of the Draft Standard for Floating-Point Arithmetic, which is intended to supersede the ANSI/IEEE Std 754-1985. Cobol programs using zoned-decimal operations can take advantage of this zEC12 introduced architecture.
Software support
Decimal floating point is supported in various programming languages and products such as: Release 4 and later of the High Level Assembler C/C++ (requires z/OS 1.10 with program temporary fixes, PTFs, for full support) Enterprise PL/I Release 3.7 and Debug Tool Release 8.1 Java Applications using the BigDecimal Class Library SQL support as of DB2 Version 9
3.3.5 IEEE floating point

Binary and hexadecimal floating-point instructions are implemented in zEC12. They incorporate IEEE Standards into the platform. The key point is that Java and C/C++ applications tend to use IEEE Binary Floating Point operations more frequently than earlier applications. This means that the better the hardware implementation of this set of instructions, the better the performance of applications will be.
3.3.6 Processor error detection and recovery

The PU uses a process called transient recovery as an error recovery mechanism. When an error is detected, the instruction unit retries the instruction and attempts to recover the error. If the retry is not successful (that is, a permanent fault exists), a relocation process is started that restores the full capacity by moving work to another PU. Relocation under hardware control is possible because the R-unit has the full architected state in its buffer. The principle is shown in Figure 3-7.
88
8049ch03.fm
Soft Error
Instruction State Checkpoint
Hard Error
PU x
PU y
R-Unit No Error Fault
Figure 3-7 PU error detection and recovery
3.3.7 Branch prediction

Because of the ultra high frequency of the PUs, the penalty for a wrongly predicted branch is high. So a multi-pronged strategy for branch prediction, based on gathered branch history combined with other prediction mechanisms, is implemented on each microprocessor. The branch history table (BHT) implementation on processors has a large performance improvement effect. Originally introduced on the IBM ES/9000 9021 in 1990, the BHT has been continuously improved. The BHT offers significant branch performance benefits. The BHT allows each PU to take instruction branches based on a stored BHT, which improves processing times for calculation routines. Besides the BHT, the zEC12 uses a variety of techniques to improve the prediction of the correct branch to be executed. The following techniques are included: Branch history table (BHT) Branch target buffer (BTB) Pattern history table (PHT) BTB data compression The success rate of branch prediction contributes significantly to the superscalar aspects of the zEC12. This is because the architecture rules prescribe that, for successful parallel execution of an instruction stream, the correctly predicted result of the branch is essential. zEC12 architecture introduces new instructions BPP/BPRL to allow software to preload future branch and its target into the branch target buffer (BTB).
3.3.8 Wild branch

When a bad pointer is used or when code overlays a data area containing a pointer to code, a random branch is the result, causing a 0C1 or 0C4 abend. Random branches are very hard to diagnose because clues about how the system got there are not evident. With the wild branch hardware facility, the last address from which a successful branch instruction was executed is kept. z/OS uses this information in conjunction with debugging aids, such as the SLIP command, to determine where a wild branch came from and might collect data from that storage location. This approach decreases the many debugging steps necessary when looking for where the branch came from.
89
8049ch03.fm
3.3.9 Translation look-aside buffer

The translation look-aside buffer (TLB) in the instruction and data L1 caches use a secondary TLB to enhance performance. In addition, a translator unit is added to translate misses in the secondary TLB. The size of the TLB is kept as small as possible because of its low access time requirements and hardware space limitations. Because memory sizes have recently increased significantly, as a result of the introduction of 64-bit addressing, a smaller working set is represented by the TLB. To increase the working set representation in the TLB without enlarging the TLB, large page support is introduced and can be used when appropriate. See Large page support on page 104. With the new enhanced DAT-2 (EDAT-2) improvements, zEC12 introduces architecture enhancements to allow 2 GB page frames support.
3.3.10 Instruction fetching, decode, and grouping

The superscalar design of the microprocessor allows for the decoding of up to three instructions per cycle and the execution of up to seven instructions per cycle. Both execution and storage accesses for instruction and operand fetching can occur out of sequence. Instruction fetching Instruction fetching normally tries to get as far ahead of instruction decoding and execution as possible because of the relatively large instruction buffers available. In the microprocessor, smaller instruction buffers are used. The operation code is fetched from the I-cache and put in instruction buffers that hold prefetched data awaiting decoding. Instruction decoding The processor can decode up to three instructions per cycle. The result of the decoding process is queued and subsequently used to form a group. Instruction grouping From the instruction queue, up to five instructions can be completed on every cycle. A complete description of the rules is beyond the scope of this book. The compilers and JVMs are responsible for selecting instructions that best fit with the superscalar microprocessor and abide by the rules to create code that best exploits the superscalar implementation. All the System z compilers and the JVMs are under constant change to benefit from new instructions as well as advances in microprocessor designs.
3.3.11 Extended translation facility

Instructions have been added to the z/Architecture instruction set in support of the Extended Translation facility. They are used in data conversion operations for data encoded in Unicode, causing applications that are enabled for Unicode or globalization to be more efficient. These data-encoding formats are used in Web services, grid, and on-demand environments where XML and SOAP3 technologies are used. The High Level Assembler supports the Extended Translation Facility instructions.
SOAP - Simple Object Access Protocol
90
8049ch03.fm
3.3.12 Instruction set extensions

The processor supports a large number of instructions to support functions, including these: Hexadecimal floating point instructions for various unnormalized multiply and multiply-add instructions. Immediate instructions, including various add, compare, OR, exclusive OR, subtract, load, and insert formats; use of these instructions improves performance. Load instructions for handling unsigned half words (such as those used for Unicode). Cryptographic instructions, known as the Message-Security Assist (MSA), offer the full complement of the AES, SHA-1, SHA-2 and DES algorithms along with functions for random number generation Extended Translate Facility-3 instructions, enhanced to conform with the current Unicode 4.0 standard. Assist instructions, help eliminate hypervisor overhead.
3.3.13 Transactional execution (TX)

This capability, know in the industry as hardware transactional memory, provides the ability to execute a group of instructions atomically, that is, either all their results are committed or none is, in true transactional way. The execution is optimistic: the instructions are executed but previous state values are saved in a transactional memory. If the transaction succeeds, the saved values are discarded, otherwise they are used to restore the original values. The Transaction Execution facility provides instructions, including declaring the beginning and end of a transaction, and to abort the transaction. TX is expected to provide significant performance benefits and scalability by providing the ability to avoid most locks. This is especially important for heavily threaded applications, such as Java.
3.3.14 Run-time instrumentation (RI)

Run-time instrumentation is a hardware facility introduced with the zEC12 for managed runtimes such as the Java Runtime Environment (JRE), that allow dynamic optimization of code generation as it is being executed. It requires much lower overhead than the current software only profiling, and provides information on hardware as well as program characteristics. It enhances JRE in making the right decision by providing real time feedback on the execution.
3.4 Processor unit functions

This section describes the processor unit (PU) functions.
3.4.1 Overview
All PUs on a zEC12 are physically identical. When the system is initialized, PUs can be characterized to specific functions: CP, IFL, ICF, zAAP, zIIP, or SAP. The function assigned to a PU is set by the licensed internal code (LIC), which is loaded when the system is initialized (at power-on reset) and the PU is characterized. Only characterized PUs have a designated function. Non-characterized PUs are considered spares. At least one CP, IFL, or ICF must be ordered on a zEC12.
91
8049ch03.fm
This design brings outstanding flexibility to the zEC12, because any PU can assume any available characterization. This also plays an essential role in system availability, because PU characterization can be done dynamically, with no system outage, allowing the actions discussed in the following sections. Also see Chapter 8, Software support on page 243 for information about software level support on functions and features.
Concurrent upgrades
Except on a fully configured model, concurrent upgrades can be done by the licensed internal code (LIC), which assigns a PU function to a previously non-characterized PU. Within the book boundary or boundary of multiple books, no hardware changes are required, and the upgrade can be done concurrently through the following facilities: Customer Initiated Upgrade (CIU) facility for permanent upgrades On/Off Capacity on Demand (On/Off CoD) for Temporary upgrades Capacity Backup (CBU) for Temporary upgrades Capacity for Planned Event (CPE) for temporary upgrades. If the MCMs in the installed books have no available remaining PUs, an upgrade results in a model upgrade and the installation of an additional book (up to the limit of 4 books). Book installation is nondisruptive, but can take more time than a simple LIC upgrade. For more information about Capacity on Demand, see Chapter 9, System upgrades on page 311.
PU sparing
In the rare event of a PU failure, the failed PUs characterization is dynamically and transparently reassigned to a spare PU. The zEC12 has two spare PUs. PUs not characterized on a CPC configuration can also be used as additional spare PUs. More information about PU sparing is provided in 3.4.11, Sparing rules on page 102.
PU pools
PUs defined as CPs, IFLs, ICFs, zIIPs, and zAAPs are grouped together in their own pools, from where they can be managed separately. This significantly simplifies capacity planning and management for logical partitions. The separation also has an effect on weight management because CP, zAAP, and zIIP weights can be managed separately. For more information, see PU weighting on page 93. All assigned PUs are grouped together in the PU pool. These PUs are dispatched to online logical PUs. As an example, consider a zEC12 with ten CPs, three zAAPs, two IFLs, two zIIPs, and one ICF. This system has a PU pool of 18 PUs, called the pool width. Subdivision of the PU pool defines: A CP pool of ten CPs An ICF pool of one ICF An IFL pool of two IFLs A zAAP pool of three zAAPs A zIIP pool of two zIIPs. PUs are placed in the pools according to the following occurrences: When the system is power-on reset At the time of a concurrent upgrade As a result of an addition of PUs during a CBU 92
8049ch03.fm
Following a capacity on demand upgrade, through On/Off CoD or CIU. PUs are removed from their pools when a concurrent downgrade takes place as the result of removal of a CBU, and through On/Off CoD and conversion of a PU. Also, when a dedicated logical partition is activated, its PUs are taken from the proper pools, as is the case when a logical partition logically configures a PU on, if the width of the pool allows. By having different pools, a weight distinction can be made between CPs, zAAPs, and zIIPs. On earlier machines specialty engines such as zAAPs automatically received the weight of the initial CP. For a logical partition, logical PUs are dispatched from the supporting pool only. This means that logical CPs are dispatched from the CP pool, logical zAAPs are dispatched from the zAAP pool, logical zIIPs from the zIIP pool, logical IFLs from the IFL pool, and the logical ICFs from the ICF pool.
PU weighting
Because zAAPs, zIIPs, IFLs, and ICFs have their own pools from where they are dispatched, they can be given their own weights. For more information about PU pools and processing weights, see zEnterprise System Processor Resource/Systems Manager Planning Guide, SB10-7156.
3.4.2 Central processors

A central processor (CP) is a PU that uses the full z/Architecture instruction set. It can run z/Architecture-based operating systems (z/OS, z/VM, TPF, z/TPF, z/VSE, Linux), the Coupling Facility Control Code (CFCC) and IBM zAware. Up to 101 PUs can be characterized as CPs, depending on the configuration. The zEC12 can only be initialized in LPAR mode. CPs are defined as either dedicated or shared. Reserved CPs can be defined to a logical partition to allow for nondisruptive image upgrades. If the operating system in the logical partition supports the logical processor add function, reserved processors are no longer needed. Regardless of the installed model, a logical partition can have up to 101 logical CPs defined (the sum of active and reserved logical CPs). In practice, define no more CPs than the operating system supports; i.e. z/OS LPAR supports a maximum of 99 logical CPs. All PUs characterized as CPs within a configuration are grouped into the CP pool. The CP pool can be seen on the HMC workplace. Any z/Architecture operating systems, CFCCs and IBM zAware can run on CPs that are assigned from the CP pool.
Granular capacity
The zEC12 recognizes four distinct capacity settings for CPs. Full-capacity CPs are identified as CP7. In addition to full-capacity CPs, three subcapacity settings (CP6, CP5, and CP4), each for up to 20 CPs, are offered. The four capacity settings appear in hardware descriptions, as follows: CP7 feature code 1908 CP6 feature code 1907 CP5 feature code 1906 CP4 feature code 1905 Granular capacity adds 60 subcapacity settings to the 101 capacity settings that are available with full capacity CPs (CP7). Each of the 60 subcapacity settings applies only to up to 20 CPs, independently of the model installed.
93
8049ch03.fm
Information about CPs in the remainder of this chapter applies to all CP capacity settings, CP7, CP6, CP5, and CP4, unless indicated otherwise. See 2.8, Model configurations on page 61, for more details about granular capacity.
3.4.3 Integrated Facility for Linux

An Integrated Facility for Linux (IFL) is a PU that can be used to run Linux, Linux guests on z/VM operating systems or IBM zAware. Up to 101 PUs can be characterized as IFLs, depending on the configuration. IFLs can be dedicated to a Linux, a z/VM, or a IBM zAware logical partition, or can be shared by multiple Linux guests, z/VM logical partitions or IBM zAware running on the same zEC12. Only z/VM, Linux on System z operating systems, IBM zAware and designated software products can run on IFLs. IFLs are orderable by feature code (FC 1909).
IFL pool
All PUs characterized as IFLs within a configuration are grouped into the IFL pool. The IFL pool can be seen on the HMC workplace. IFLs do not change the model capacity identifier of the zEC12. Software product license charges based on the model capacity identifier are not affected by the addition of IFLs.
Unassigned IFLs
An IFL that is purchased but not activated is registered as an unassigned IFL (FC 1914). When the system is subsequently upgraded with an additional IFL, the system recognizes that an IFL was already purchased and is present.
3.4.4 Internal Coupling Facility

An Internal Coupling Facility (ICF) is a PU used to run the Coupling Facility Control Code (CFCC) for Parallel Sysplex environments. Within the capacity of the sum of all unassigned PUs in up to four books, up to 101 ICFs can be characterized, depending on the model. Still, the maximum number of ICFs that can be defined on a coupling facility logical partition is limited to 16. ICFs are orderable by feature code (FC 1910). ICFs exclusively run CFCC. ICFs do not change the model capacity identifier of the zEC12. Software product license charges based on the model capacity identifier are not affected by the addition of ICFs. All ICFs within a configuration are grouped into the ICF pool. The ICF pool can be seen on the HMC workplace. The ICFs can only be used by coupling facility logical partitions. ICFs are either dedicated or shared. ICFs can be dedicated to a CF logical partition, or shared by multiple CF logical partitions running in the same system. However, having a logical partition with dedicated and shared ICFs at the same time is not possible.
Coupling Facility combinations

Thus, a coupling facility image can have one of the following combinations defined in the image profile: Dedicated ICFs Shared ICFs Dedicated CPs Shared CPs 94
8049ch03.fm
Shared ICFs add flexibility. However, running only with shared coupling facility PUs (either ICFs or CPs) is not a desirable production configuration. It is preferable for a production CF to operate by using dedicated ICFs. In Figure 3-8, the CPC on the left has two environments defined (production and test), each having one z/OS and one coupling facility image. The coupling facility images are sharing the same ICF.
Test Sysplex
ICF
CP Partition Image Profile
z/OS Test
z/OS
Prod
CF CF
z/OS Test
z/OS CF Prod
HMC
Setup
Figure 3-8 ICF options; shared ICFs
The logical partition processing weights are used to define how much processor capacity each coupling facility image can have. The capped option can also be set for a test coupling facility image to protect the production environment. Connections between these z/OS and coupling facility images can use internal coupling links (ICs) to avoid the use of real (external) coupling links and to get the best link bandwidth available.
Dynamic Coupling Facility dispatching

The dynamic coupling facility dispatching function has a dispatching algorithm that lets you define a backup coupling facility in a logical partition on the system. When this logical partition is in backup mode, it uses very little processor resources. When the backup CF becomes active, only the resource necessary to provide coupling is allocated.
3.4.5 System z Application Assist Processors

A System z Application Assist Processor (zAAP) reduces the standard processor (CP) capacity requirements for z/OS Java or XML system services applications, freeing up capacity for other workload requirements. zAAPs do not increase the MSU value of the processor and therefore do not affect the IBM software license charges. The zAAP is a PU that is used for running IBM designated z/OS workloads such as Java or z/OS XML System Services. IBM SDK for z/OS Java 2 Technology Edition (the Java Virtual Machine), in cooperation with z/OS dispatcher, directs JVM processing from CPs to zAAPs. Also, z/OS XML parsing performed in TCB mode is eligible to be executed on the zAAP processors. zAAPs include the following benefits:
95
8049ch03.fm
Potential cost savings. Simplification of infrastructure as a result of the co-location and integration of new applications with their associated database systems and transaction middleware (such as DB2, IMS, or CICS). Simplification can happen, for example, by introducing a uniform security environment, reducing the number of TCP/IP programming stacks and system interconnect links. Prevention of processing latencies that would occur if Java application servers and their database servers were deployed on separate server platforms. One CP must be installed with or prior to installing a zAAP. The number of zAAPs in a CPC cannot exceed the number of purchased CPs. Within the capacity of the sum of all unassigned PUs in up to four books, up to 50 zAAPs on a model HA1 can be characterized. Table 3-1 shows the allowed number of zAAPs for each model.
Table 3-1 Number of zAAPs per model Model zAAPs H20 0 10 H43 0 21 H66 0 33 H89 0 44 HA1 0 50
The number of permanent zAAPs plus temporary zAAPs cannot exceed the number of purchased (permanent plus unassigned) CPs plus temporary CPs. Also, the number of temporary zAAPs cannot exceed the number of permanent zAAPs. PUs characterized as zAAPs within a configuration are grouped into the zAAP pool. This allows zAAPs to have their own processing weights, independent of the weight of parent CPs. The zAAP pool can be seen on the hardware console. zAAPs are orderable by feature code (FC 1912). Up to one zAAP can be ordered for each CP or marked CP configured in the CPC.
zAAPs and logical partition definitions

zAAPs are either dedicated or shared, depending on whether they are part of a logical partition with dedicated or shared CPs. In a logical partition, you must have at least one CP to be able to define zAAPs for that partition. You can define as many zAAPs for a logical partition as are available in the system. Logical partition: A zEC12 cannot have more zAAPs than CPs. However, in a logical partition, as many zAAPs as are available can be defined together with at least one CP.
How zAAPs work

zAAPs are designed for supporting designated z/OS workloads. The initial exploiter was Java code execution. When Java code must be executed (for example, under control of WebSphere), the z/OS Java Virtual Machine (JVM) calls the function of the zAAP. The z/OS dispatcher then suspends the JVM task on the CP it is running on and dispatches it on an available zAAP. After the Java application code execution is finished, z/OS redispatches the JVM task on an available CP, after which normal processing is resumed. This process reduces the CP time needed to run Java WebSphere applications, freeing capacity for other workloads. Figure 3-9 shows the logical flow of Java code running on a zEC12 that has a zAAP available. When JVM starts execution of a Java program, it passes control to the z/OS dispatcher that will verify the availability of a zAAP.
96
8049ch03.fm
Availability is treated as follows: If a zAAP is available (not busy), the dispatcher suspends the JVM task on the CP, and assigns the Java task to the zAAP. When the task returns control to the JVM, it passes control back to the dispatcher that reassigns the JVM code execution to a CP. If no zAAP is available (all busy) at that time, the z/OS dispatcher can allow a Java task to run on a standard CP, depending on the option used in the OPT statement in the IEAOPTxx member of SYS1.PARMLIB.
Standard Processor
WebSphere WebSphere Execute JAVA Code Execute JAVACode
zAAP z/OS Dispatcher z/OS z/OS Dispatcher Dispatcher Dispatch JVM task on z/OS Dispatch JVM task on z/OS zAAP logical processor zAAP processor logical logical processor
JVM JVM Switch to zAAP Switch to zAAP
JVM JVM
z/OS Dispatcher z/OS Dispatcher Suspend JVM task on z/OS Suspend JVM task on z/OS standard logical processor standard logical processor z/OS Dispatcher z/OS Dispatcher z/OS Dispatcher Java Application Code Java ApplicationCode Executing on a zAAP Executing on a zAAP logical processor logical processor
Dispatch JVM task on z/OS Dispatch JVM task on z/OS standard logical processor standard logical processor
JVM
JVM JVM
JVM JVM JVM Switch to Switch to standard processor Switch to standard processor
z/OS Dispatcher z/OS Dispatcher Suspend JVM task on z/OS Suspend JVM task on z/OS zAAP logical processor zAAP logical processor
WebSphere WebSphere
Figure 3-9 Logical flow of Java code execution on a zAAP
A zAAP executes only IBM authorized code. This includes the z/OS JVM in association with parts of system code, such as the z/OS dispatcher and supervisor services. A zAAP is not able to process I/O or clock comparator interruptions and does not support operator controls such as IPL. Java application code can either run on a CP or a zAAP. The installation can manage the use of CPs such that Java application code runs only on CPs, only on zAAPs, or on both. Three execution options for Java code execution are available. These options are user specified in IEAOPTxx and can be dynamically altered by the SET OPT command. The options that are currently supported for z/OS V1R10 and later releases are as follows. Option 1: Java dispatching by priority (IFAHONORPRIORITY=YES): This is the default option and specifies that CPs must not automatically consider zAAP-eligible work for dispatching on them. The zAAP-eligible work is dispatched on the zAAP engines until Workload Manager (WLM) considers that the zAAPs are overcommitted. WLM then requests help from the CPs. When help is requested, the CPs consider dispatching zAAP-eligible work on the CPs themselves based on the dispatching priority relative to other workloads. When the zAAP engines are no longer overcommitted, the CPs stop considering zAAP-eligible work for dispatch.
97
8049ch03.fm
This option has the effect of running as much zAAP-eligible work on zAAPs as possible and only allowing it to spill over onto the CPs when the zAAPs are overcommitted. Option 2: Java dispatching by priority (IFAHONORPRIORITY=NO): zAAP-eligible work executes on zAAPs only while at least one zAAP engine is online. zAAP-eligible work is not normally dispatched on a CP, even if the zAAPs are overcommitted and CPs are unused. The exception to this is that zAAP-eligible work may sometimes run on a CP to resolve resource conflicts, and other reasons. Therefore, zAAP-eligible work does not affect the CP utilization that is used for reporting through SCRT, no matter how busy the zAAPs are. Option 3: Java discretionary crossover (IFACROSSOVER=YES or NO): As of z/OS V1R8 (and the IBM zIIP Support for z/OS V1R7 Web deliverable), the IFACROSSOVER parameter is no longer honored. If zAAPs are defined to the logical partition but are not online, the zAAP-eligible work units are processed by CPs in order of priority. The system ignores the IFAHONORPRIORITY parameter in this case and handles the work as though it had no eligibility to zAAPs.
3.4.6 System z Integrated Information Processor

A System z Integrated Information Processor (zIIP) enables eligible z/OS workloads to have a portion of the workloads enclave service request block (SRB) work directed to the zIIP. The zIIPs do not increase the MSU value of the processor and therefore do not affect the IBM software license changes. z/OS Communications Server and DB2 UDB for z/OS version 8 (and later) exploit the zIIP by indicating to z/OS which portions of the work are eligible to be routed to a zIIP. Here we list eligible DB2 UDB for z/OS V8 (and later) workloads executing in SRB mode: Query processing of network-connected applications that access the DB2 database over a TCP/IP connection using Distributed Relational Database Architecture (DRDA). DRDA enables relational data to be distributed among multiple platforms. It is native to DB2 for z/OS, thus reducing the need for additional gateway products that can affect performance and availability. The application uses the DRDA requestor or server to access a remote database. (DB2 Connect is an example of a DRDA application requester.) Star schema query processing, mostly used in Business Intelligence (BI) work. A star schema is a relational database schema for representing multidimensional data. It stores data in a central fact table and is surrounded by additional dimension tables holding information about each perspective of the data. A star schema query, for example, joins various dimensions of a star schema data set. DB2 utilities that are used for index maintenance, such as LOAD, REORG, and REBUILD. Indices allow quick access to table rows, but over time, as data in large databases is manipulated, they become less efficient and have to be maintained. The zIIP runs portions of eligible database workloads and in doing so helps to free up computer capacity and lower software costs. Not all DB2 workloads are eligible for zIIP processing. DB2 UDB for z/OS V8 and later gives z/OS the information to direct portions of the work to the zIIP. The result is that in every user situation, different variables determine how much work is actually redirected to the zIIP. On a zEC12 the following workloads can also benefit from zIIPs.
98
8049ch03.fm
z/OS Communications Server exploits the zIIP for eligible Internet protocol security (IPSec) network encryption workloads. This requires z/OS V1R10 or later releases. Portions of IPSec processing take advantage of the zIIPs, specifically end-to-end encryption with IPSec. The IPSec function moves a portion of the processing from the general-purpose processors to the zIIPs. In addition to performing the encryption processing, the zIIP also handles cryptographic validation of message integrity and IPSec header processing. z/OS Global Mirror, formerly known as Extended Remote Copy (XRC), exploits the zIIP too. Most z/OS DFSMS system data mover (SDM) processing associated with zGM is eligible to run on the zIIP. This requires z/OS V1R10 or later releases. The first IBM exploiter of z/OS XML system services is DB2 V9. With regard to DB2 V9 prior to the z/OS XML system services enhancement, z/OS XML system services non-validating parsing was partially directed to zIIPs when used as part of a distributed DB2 request through DRDA. This enhancement benefits DB2 V9 by making all z/OS XML system services non-validating parsing eligible to zIIPs when processing is used as part of any workload running in enclave SRB mode. z/OS Communications Server also allows the HiperSockets Multiple Write operation for outbound large messages (originating from z/OS) to be performed by a zIIP. Application workloads based on XML, HTTP, SOAP, Java, etc as well as traditional file transfer, can benefit. For business intelligence, IBM Scalable Architecture for Financial Reporting provides a high-volume, high performance reporting solution by running many diverse queries in z/OS batch and can also be eligible for zIIP. For more information, see the IBM zIIP website: http://www-03.ibm.com/systems/z/hardware/features/ziip/about.html
zIIP installation information

One CP must be installed with or prior to any zIIP being installed. The number of zIIPs in a system cannot exceed the number of CPs and unassigned CPs in that system. Within the capacity of the sum of all unassigned PUs in up to four books, up to 50 zIIPs on a model HA1 can be characterized. Table 3-2 shows the allowed number of zIIPs for each model.
Table 3-2 Number of zIIPs per model Model Maximum zIIPs H20 0 10 H43 0 21 H66 0 33 H89 0 44 HA1 0 50
zIIPs are orderable by feature code (FC 1913). Up to one zIIP can be ordered for each CP or marked CP configured in the system. If the installed books have no remaining unassigned PUs, the assignment of the next zIIP might require the installation of an additional book. PUs characterized as zIIPs within a configuration are grouped into the zIIP pool. By doing this, zIIPs can have their own processing weights, independent of the weight of parent CPs. The zIIP pool can be seen on the hardware console. The number of permanent zIIPs plus temporary zIIPs cannot exceed the number of purchased CPs plus temporary CPs. Also, the number of temporary zIIPs cannot exceed the number of permanent zIIPs.
99
8049ch03.fm
zIIPs and logical partition definitions

zIIPs are either dedicated or shared depending on whether they are part of a logical partition with dedicated or shared CPs. In a logical partition, at least one CP must be defined before zIIPs for that partition can be defined. The number of zIIPs available in the system is the number of zIIPs that can be defined to a logical partition. Logical partition: A system cannot have more zIIPs than CPs. However, in a logical partition, as many zIIPs as are available can be defined together with at least one CP.
3.4.7 zAAP on zIIP capability

As described previously, zAAPs and zIIPs support different types of workloads. However, there are installations that do not have enough eligible workloads to justify buying a zAAP or a zAAP and a zIIP. IBM is now making available the possibility of combining zAAP and zIIP workloads on zIIP processors, provided that no zAAPs are installed on the system. This combination can provide the following benefits: The combined eligible workloads can make the zIIP acquisition more cost effective. When zIIPs are already present, investment is maximized by running the Java and z/OS XML System Services-based workloads on existing zIIPs. This capability does not eliminate the need to have one CP for every zIIP processor in the system. Support is provided by z/OS. See 8.3.2, zAAP support on page 258. When zAAPs are present4 this capability is not available, as it is neither intended as a replacement for zAAPs, which continue to be available, nor as an overflow possibility for zAAPs. Do not convert zAAPs to zIIPs in order to take advantage of the zAAP to zIIP capability, for the following reasons: Having both zAAPs and zIIPs maximizes the system potential for new workloads. zAAPs were announced on April 7, 2004. Having been available for so many years, there might exist applications or middleware with zAAP-specific code dependencies. For example, the code can use the number of installed zAAP engines to optimize multithreading performance. It is a good idea to plan and test before eliminating all zAAPs, because there can be application code dependencies that might affect performance. Statement of Direction: IBM zEnterprise EC12 is planned to be the last high-end System z server to offer support for zAAP specialty engine processors. IBM intends to continue support for running zAAP workloads on zIIP processors (zAAP on zIIP). This is intended to help simplify capacity planning and performance management, while still supporting all the currently eligible workloads. In addition, IBM plans to provide a PTF for APAR OA38829 on z/OS V1.12 and V1.13 in September 2012 to remove the restriction that prevents zAAP-eligible workloads from running on zIIP processors when a zAAP is installed on the server. This is intended only to help facilitate migration and testing of zAAP workloads on zIIP processors.
The zAAP on zIIP capability is available to z/OS when running as a guest of z/VM on machines with zAAPs installed, provided that no zAAPs are defined to the z/VM LPAR. This allows, for instance, testing this capability to estimate usage before committing to production.
100
8049ch03.fm
3.4.8 System Assist Processors

A System Assist Processor (SAP) is a PU that runs the channel subsystem licensed internal code (LIC) to control I/O operations. All SAPs perform I/O operations for all logical partitions. All models have standard SAPs configured. The number of standard SAPs depends on the zEC12 model, as shown in Table 3-3.
Table 3-3 SAPs per model Model Standard SAPs H20 4 H43 8 H66 12 H89 16 HA1 16
SAP configuration
A standard SAP configuration provides a very well-balanced system for most environments. However, there are application environments with very high I/O rates (typically various TPF environments). In this case, optional additional SAPs can be ordered. Assignment of additional SAPs can increase the capability of the channel subsystem to perform I/O operations. In zEC12 systems, the number of SAPs can be greater than the number of CPs.
Optional additional orderable SAPs

An option available on all models is additional orderable SAPs (FC 1911). These additional SAPs increase the capacity of the channel subsystem to perform I/O operations, usually suggested for Transaction Processing Facility (TPF) environments. The maximum number of optional additional orderable SAPs depends on the configuration and the number of available uncharacterized PUs. The number of SAPs are listed in Table 3-4.
Table 3-4 Optional SAPs per model Model Optional SAPs H20 04 H43 08 H66 0 12 H89 0 16 HA1 0 16
Optionally assignable SAPs

Assigned CPs can be optionally reassigned as SAPs instead of CPs by using the reset profile on the Hardware Management Console (HMC). This reassignment increases the capacity of the channel subsystem to perform I/O operations, usually for specific workloads or I/O-intensive testing environments. If you intend to activate a modified system configuration with a modified SAP configuration, a reduction in the number of CPs available reduces the number of logical processors that can be activated. Activation of a logical partition can fail if the number of logical processors that you attempt to activate exceeds the number of CPs available. To avoid a logical partition activation failure, verify that the number of logical processors assigned to a logical partition does not exceed the number of CPs available.
3.4.9 Reserved processors

Reserved processors are defined by the Processor Resource/Systems Manager (PR/SM) to allow for a nondisruptive capacity upgrade. Reserved processors are like spare logical processors, and can be shared or dedicated. Reserved CPs can be defined to a logical partition dynamically to allow for nondisruptive image upgrades.
101
8049ch03.fm
Reserved processors can be dynamically configured online by an operating system that supports this function, if enough unassigned PUs are available to satisfy this request. The PR/SM rules regarding logical processor activation remain unchanged. Reserved processors provide the capability to define to a logical partition more logical processors than the number of available CPs, IFLs, ICFs, zAAPs, and zIIPs in the configuration. This makes it possible to configure online, nondisruptively, more logical processors after additional CPs, IFLs, ICFs, zAAPs, and zIIPs have been made available concurrently with one of the Capacity on Demand options. The maximum number of reserved processors that can be defined to a logical partition depends on the number of logical processors that are already defined. The maximum number of logical processors plus reserved processors is 101. Do not define more active and reserved processors than the operating system for the logical partition can support. For more information about logical processors and reserved processors and their definition, see 3.6, Logical partitioning on page 107.
3.4.10 Processor unit assignment

Processor unit assignment of characterized PUs is done at power-on reset (POR) time, when the system is initialized. The intention of the initial assignment rules is to keep PUs of the same characterization type grouped together as much as possible regarding PU chips and books boundaries, to optimize shared cache usage. The PU assignment is based on book plug ordering. This defines the low order and the high order books, as follows: Book 0: Plug order 4 (when plugged this is the low order book) Book 1: Plug order 1 (when Book 0 is not plugged this is the low order book) Book 2: Plug order 3 Book 3: Plug order 2 Assignment rules isolate the PUs as much as possible on different books (and even on different PU chips) so that the operating systems do not use the same shared caches. For example, CPs, zAAPs, and zIIPs are all used by z/OS, and can benefit by using the same shared caches. Whereas, IFLs are used by z/VM and Linux, and ICFs are used by CFCC, so for performance reasons the assignment rules prevent them from sharing L3 and L4 caches with z/OS processors. This initial PU assignment done at POR can be dynamically rearranged by LPAR, to improve system performance (see LPAR dynamic PU reassignment on page 112). When an additional book is added concurrently after POR and new logical partitions are activated, or processor capacity for active partitions is dynamically expanded, the additional PU capacity can be assigned from the new book. Only after the next POR do the processor unit assignment rules take into consideration the newly installed book.
3.4.11 Sparing rules

On a zEC12 system, two PUs are reserved as spares. The reserved spares are available to replace any two characterized PUs, what ever it may be: a CP, IFL, ICF, zAAP, zIIP, or SAP.
102
8049ch03.fm
Systems with a failed PU for which no spare is available will call home for a replacement. A system with a failed PU that has been spared and requires an MCM to be replaced (referred to as a pending repair) can still be upgraded when sufficient PUs are available.
Transparent CP, IFL, ICF, zAAP, zIIP, and SAP sparing

Depending on the model, sparing of CP, IFL, ICF, zAAP, zIIP, and SAP is completely transparent and does not require an operating system or operator intervention. With transparent sparing, the status of the application that was running on the failed processor is preserved and continues processing on a newly assigned CP, IFL, ICF, zAAP, zIIP, or SAP (allocated to one of the spare PUs) without customer intervention.
Application preservation
If no spare PU is available, application preservation (z/OS only) is invoked. The state of the failing processor is passed to another active processor used by the operating system and, through operating system recovery services, the task is resumed successfully (in most cases, without customer intervention).
Dynamic SAP sparing and reassignment

Dynamic recovery is provided in case of failure of the SAP. If the SAP fails, and if a spare PU is available, the spare PU is dynamically assigned as a new SAP. If no spare PU is available, and more than one CP is characterized, a characterized CP is reassigned as an SAP. In either case, customer intervention is not required. This capability eliminates an unplanned outage and permits a service action to be deferred to a more convenient time.
3.4.12 Increased flexibility with z/VM-mode partitions

zEC12 provides a capability for the definition of a z/VM-mode logical partition that contains a mix of processor types including CPs and specialty processors, such as IFLs, zIIPs, zAAPs, and ICFs. z/VM V5R4 and later support this capability, which increases flexibility and simplifies systems management. In a single logical partition, z/VM can: Manage guests that exploit Linux on System z on IFLs, z/VSE and z/OS on CPs. Execute designated z/OS workloads, such as parts of DB2 DRDA processing and XML, on zIIPs. Provide an economical Java execution environment under z/OS on zAAPs. If the only operating system to run under z/VM is Linux then it is preferable to define a Linux-only logical partition.
3.5 Memory design

This section describes various considerations regarding the zEC12 memory design.
3.5.1 Overview
The zEC12 memory design also provides flexibility and high availability, allowing upgrades: Concurrent memory upgrades (if the physically installed capacity is not yet reached):
103
8049ch03.fm
The zEC12 can have more physically installed memory than the initial available capacity. Memory upgrades within the physically installed capacity can be done concurrently by LIC, and no hardware changes are required. Note that memory upgrades cannot be done through Capacity BackUp (CBU) or On/Off CoD. Concurrent memory upgrades (if the physically installed capacity is reached): Physical memory upgrades require a book to be removed and re-installed after having replaced the memory cards in the book. Except for a model H20, the combination of enhanced book availability and the flexible memory option allow you to concurrently add memory to the system. For more information, see 2.5.5, Book replacement and memory on page 53, and 2.5.6, Flexible Memory Option on page 53. When the total capacity installed has more usable memory than required for a configuration, the licensed internal code configuration control (LICCC) determines how much memory is used from each card. The sum of the LICCC provided memory from each card is the amount available for use in the system.
Memory allocation
Memory assignment or allocation is done at power-on reset (POR) when the system is initialized. PR/SM is responsible for the memory assignments. PR/SM has knowledge of the amount of purchased memory and how it relates to the available physical memory in each of the installed books. PR/SM has control over all physical memory and therefore is able to make physical memory available to the configuration when a book is nondisruptively added. PR/SM also controls the reassignment of the content of a specific physical memory array in one book to a memory array in another book. This is known as the memory copy/reassign function, which is used to reallocate the memory content from the memory in a book to another memory location. It is used when enhanced book availability (EBA) is applied to concurrently remove and re-install a book in case of an upgrade or repair action. Because of the memory allocation algorithm, systems that undergo a number of miscellaneous equipment specification (MES) upgrades for memory can have a variety of memory mixes in all books of the system. If, however unlikely, memory fails, it is technically feasible to power-on reset the system with the remaining memory resources. After power-on reset, the memory distribution across the books is now different, and so is the amount of available memory.
Large page support

By default, page frames are allocated with a 4 KB size. The zEC12 also supports large page sizes of 1 MB or 2 GB. The first z/OS release that supports 1 MB large pages is z/OS V1R9. Linux on System z support for 1 MB large pages is available in SUSE Linux Enterprise Server (SLES) 10 SP2 and Red Hat RHEL 5.2. The translation look-aside buffer (TLB) exists to reduce the amount of time required to translate a virtual address to a real address by dynamic address translation (DAT) when it needs to find the correct page for the correct address space. Each TLB entry represents one page. Like other buffers or caches, lines are discarded from the TLB on a least recently used (LRU) basis. The worst-case translation time occurs when there is a TLB miss and both the segment table (needed to find the page table) and the page table (needed to find the entry for the particular page in question) are not in cache. In this case, there are two complete real memory access delays plus the address translation delay. The duration of a processor cycle is much smaller than the duration of a memory cycle, so a TLB miss is relatively costly.
104
8049ch03.fm
It is very desirable to have one's addresses in the TLB. With 4 K pages, holding all the addresses for 1 MB of storage takes 256 TLB lines. When using 1 MB pages, it takes only 1 TLB line. This means that large page size exploiters have a much smaller TLB footprint. Large pages allow the TLB to better represent a large working set and suffer fewer TLB misses by allowing a single TLB entry to cover more address translations. Exploiters of large pages are better represented in the TLB and are expected to see performance improvement in both elapsed time and CPU time. This is because DAT and memory operations are part of CPU busy time even though the CPU waits for memory operations to complete without processing anything else in the meantime. Overhead is associated with creating a 1 MB page. To overcome that overhead, a process has to run for a period of time and maintain frequent memory access to keep the pertinent addresses in the TLB. Very short-running work does not overcome the overhead; short processes with small working sets are expected to receive little or no improvement. Long-running work with high memory-access frequency is the best candidate to benefit from large pages. Long-running work with low memory-access frequency is less likely to maintain its entries in the TLB. However, when it does run, a smaller number of address translations is required to resolve all the memory it needs. So, a very long-running process can benefit somewhat even without frequent memory access. You should weight the benefits of whether something in this category should use large pages as a result of the system-level costs of tying up real storage. There is a balance between the performance of a process using large pages, and the performance of the remaining work on the system. On zEC12, 1 MB large pages become pageable if Flash Express is enabled. They are only available for 64-bit virtual private storage such as virtual memory located above 2 GB. One would be inclined to think, that increasing the TLB size is a feasible option to deal with TLB-miss situations. However, this is not as straightforward as it seems. As the size of the TLB increases, so does the overhead involved in managing the TLBs contents. Correct sizing of the TLB is subject to very complex statistical modelling in order to find the optimal trade-off between size and performance.
3.5.2 Central storage

Central storage (CS) consists of main storage, addressable by programs, and storage not directly addressable by programs. Non-addressable storage includes the hardware system area (HSA). Central storage provides: Data storage and retrieval for PUs and I/O Communication with PUs and I/O Communication with and control of optional expanded storage Error checking and correction Central storage can be accessed by all processors, but cannot be shared between logical partitions. Any system image (logical partition) must have a central storage size defined. This defined central storage is allocated exclusively to the logical partition during partition activation.
105
8049ch03.fm
3.5.3 Expanded storage

Expanded storage can optionally be defined on zEC12. Expanded storage is physically a section of processor storage. It is controlled by the operating system and transfers 4 KB pages to and from central storage.
Storage considerations
Except for z/VM, z/Architecture operating systems do not use expanded storage. Because they operate in 64-bit addressing mode, they can have all the required storage capacity allocated as central storage. z/VM is an exception because, even when operating in 64-bit mode, it can have guest virtual machines running in 31-bit addressing mode, which can use expanded storage. In addition, z/VM exploits expanded storage for its own operations. Defining expanded storage to a coupling facility image is not possible. However, any other image type can have expanded storage defined, even if that image runs a 64-bit operating system and does not use expanded storage. The zEC12 only runs in LPAR mode. Storage is placed into a single storage pool called LPAR single storage pool, which can be dynamically converted to expanded storage and back to central storage as needed when partitions are activated or de-activated.
LPAR single storage pool

In LPAR mode, storage is not split into central storage and expanded storage at power-on reset. Rather, the storage is placed into a single central storage pool that is dynamically assigned to expanded storage and back to central storage, as needed. On the hardware management console (HMC), the storage assignment tab of a reset profile shows the customer storage, which is the total installed storage minus the 32 GB of hardware system area. Logical partitions are still defined to have central storage and, optionally, expanded storage. Activation of logical partitions and dynamic storage reconfiguration cause the storage to be assigned to the type needed (central or expanded), and do not require a power-on reset.
3.5.4 Hardware system area

The hardware system area (HSA) is a non-addressable storage area that contains system Licensed Internal Code and configuration-dependent control blocks. On the zEC12 the HSA has a fixed size of 32 GB and is not part of the purchased memory that you order and install. The fixed size of the HSA eliminates planning for future expansion of the HSA because HCD/IOCP always reserves space for the following items: Four channel subsystems (CSSs) Fifteen logical partitions in each CSS for a total of 60 logical partitions Subchannel set 0 with 63.75 K devices in each CSS Subchannel set 1 with 64 K devices in each CSS Subchannel set 2 with 64 K devices in each CSS The HSA has sufficient reserved space allowing for dynamic I/O reconfiguration changes to the maximum capability of the processor.
106
8049ch03.fm
3.6 Logical partitioning

This section discusses logical partitioning features.
3.6.1 Overview
Logical partitioning (LPAR) is a function implemented by the Processor Resource/Systems Manager (PR/SM) on the zEC12. The zEC12 runs only in LPAR mode. This means that all system aspects are controlled by PR/SM functions. PR/SM is aware of the book structure on the zEC12. Logical partitions, however, do not have this awareness. Logical partitions have resources allocated to them from a variety of physical resources. From a systems standpoint, logical partitions have no control over these physical resources, but the PR/SM functions do. PR/SM manages and optimizes allocation and the dispatching of work on the physical topology. Most physical topology that was previously handled by the operating systems is the responsibility of PR/SM. As shown in 3.4.10, Processor unit assignment on page 102, the initial PU assignment is done during power-on-reset (POR), using rules to optimize cache usage. This is the physical step, where CPs, zIIPs, zAAPs, IFLs, ICFs, and SAPs are allocated on books. When a logical partition is activated, PR/SM builds logical processors and allocates memory for the logical partition. Memory allocation is spread across all books, using a round robin algorithm with three increments per book, to match the number of memory controllers (MCs) per book. This memory allocation design is driven by performance results, also minimizing variability for the majority of workloads. Logical processors are dispatched by PR/SM on physical processors. The assignment topology used by PR/SM to dispatch logical on physical PUs is also based on cache usage optimization. Book level assignments are more important, because this optimizes L4 cache usage. So logical processors from a given logical partition are packed into a book (or books) as much as possible. Then PR/SM optimizes chip assignments within the assigned book (or books), to maximize L3 cache efficiency. So logical processors from a logical partition are dispatched on physical processors on the same PU chip as much as possible. Note that the number of processors per chip (six) matches the number of z/OS processor affinity queues (also six), used by HiperDispatch, achieving optimal cache usage within an affinity node. PR/SM also tries to redispatch a logical processor on the same physical processor to optimize private caches (L1 and L2) usage.
HiperDispatch
PR/SM and z/OS work in tandem to more efficiently use processor resources. HiperDispatch is a function that combines the dispatcher actions and the knowledge that PR/SM has about the topology of the system.
107
8049ch03.fm
Performance can be optimized by redispatching units of work to same processor group, keeping processes running near their cached instructions and data, and minimizing transfers of data ownership among processors/books. The nested topology is returned to z/OS by the Store System Information (STSI) instruction, and HiperDispatch utilizes the information to concentrate logical processors around shared caches (L3 at PU chip level, and L4 at book level), and dynamically optimizes assignment of logical processors and units of work. z/OS dispatcher manages multiple queues, called affinity queues, with a target number of six processors per queue, which fits nicely into a single PU chip. These queues are used to assign work to as few logical processors as are needed for a given logical partition workload. So, even if the logical partition is defined with a large number of logical processors, HiperDispatch optimizes this number of processors nearest to the required capacity. The optimal number of processors to be used are kept within a book boundary where possible.
Logical partitions
PR/SM enables the zEC12 to be initialized for a logically partitioned operation, supporting up to 60 logical partitions. Each logical partition can run its own operating system image in any image mode, independently from the other logical partitions. A logical partition can be added, removed, activated, or deactivated at any time. Changing the number of logical partitions is not disruptive and does not require power-on reset (POR). Certain facilities might not be available to all operating systems, because the facilities might have software co-requisites. Each logical partition has the same resources as a real CPC. Those are processors, memory, and channels: Processors: Called logical processors, they can be defined as CPs, IFLs, ICFs, zAAPs, or zIIPs. They can be dedicated to a logical partition or shared among logical partitions. When shared, a processor weight can be defined to provide the required level of processor resources to a logical partition. Also, the capping option can be turned on, which prevents a logical partition from acquiring more than its defined weight, limiting its processor consumption. Logical partitions for z/OS can have CP, zAAP, and zIIP logical processors. All three logical processor types can be defined as either all dedicated or all shared. The zAAP and zIIP support is available in z/OS. The weight and the number of online logical processors of a logical partition can be dynamically managed by the LPAR CPU Management function of the Intelligent Resource Director (IRD) to achieve the defined goals of this specific partition and of the overall system. The provisioning architecture of the zEC12, described in Chapter 9, System upgrades on page 311, adds another dimension to the dynamic management of logical partitions. For z/OS Workload License Charge (WLC) pricing metric, and metrics based on it such as AWLC, a logical partition defined capacity can be set, enabling the soft capping function. Workload charging introduces the capability to pay software license fees based on the processor utilization of the logical partition on which the product is running, rather than on the total capacity of the system, as follows: In support of WLC, the user can specify a defined capacity in millions of service units (MSUs) per hour. The defined capacity sets the capacity of an individual logical partition when soft capping is selected. The defined capacity value is specified on the Options tab on the Customize Image Profiles panel 108
8049ch03.fm
WLM keeps a 4-hour rolling average of the CPU usage of the logical partition, and when the 4-hour average CPU consumption exceeds the defined capacity limit, WLM dynamically activates LPAR capping (soft capping). When the rolling 4-hour average returns below the defined capacity, the soft cap is removed. For more information regarding WLM, see System Programmer's Guide to: Workload Manager, SG24-6472. For a review of software licensing see 8.12, Software licensing considerations on page 305. Weight settings: When defined capacity is used to define an uncapped logical partitions capacity, looking carefully at the weight settings of that logical partition is important. If the weight is much smaller than the defined capacity, PR/SM will use a discontinuous cap pattern to achieve the defined capacity setting. This means PR/SM will alternate between capping the LPAR at the MSU value corresponding to the relative weight settings, and no capping at all. It is best to avoid this case, and try to establish a defined capacity which is equal or close to the relative weight. Memory: Memory, either central storage or expanded storage, must be dedicated to a logical partition. The defined storage must be available during the logical partition activation. Otherwise, the activation fails.
Reserved storage can be defined to a logical partition, enabling nondisruptive memory

addition to and removal from a logical partition, using the LPAR dynamic storage reconfiguration (z/OS and z/VM). For more information, see 3.6.5, LPAR dynamic storage reconfiguration on page 116. Channels: Channels can be shared between logical partitions by including the partition name in the partition list of a channel path identifier (CHPID). I/O configurations are defined by the input/output configuration program (IOCP) or the hardware configuration dialog (HCD) in conjunction with the CHPID mapping tool (CMT). The CMT is an optional, but strongly preferred, tool used to map CHPIDs onto physical channel identifiers (PCHIDs) that represent the physical location of a port on a card in an I/O cage, I/O drawer or PCIe I/O drawer. IOCP is available on the z/OS, z/VM, and z/VSE operating systems, and as a stand-alone program on the hardware console. HCD is available on z/OS and z/VM operating systems. FICON channels can be managed by the Dynamic CHPID Management (DCM) function of the Intelligent Resource Director. DCM enables the system to respond to ever-changing channel requirements by moving channels from lesser-used control units to more heavily used control units, as needed.
Modes of operation
Table 3-5 shows the nodes of operation, summarizing all available mode combinations: operating modes and their processor types, operating systems, and addressing modes. Only the currently in-support versions of operating systems are considered.
109
8049ch03.fm
Table 3-5 zEC12 modes of operation Image mode ESA/390 PU type CP and zAAP/zIIP CP CP ESA/390 TPF Coupling facility LINUX-only CP only ICF or CP IFL or CP Operating system z/OS z/VM z/VSE and Linux on System z (64-bit) Linux on System z (31-bit) z/TPF CFCC Linux on System z (64-bit) z/VM Linux on System z (31-bit) z/VM zAware CP, IFL, zIIP, zAAP, ICF IFL or CP z/VM zAware 31-bit 64-bit 64-bit Addressing mode 64-bit 64-bit 31-bit 64-bit 64-bit 64-bit
The 64-bit z/Architecture mode has no special operating mode because the architecture mode is not an attribute of the definable images operating mode. The 64-bit operating systems are IPLed in 31-bit mode and, optionally, can change to 64-bit mode during their initialization. The operating system is responsible for taking advantage of the addressing capabilities provided by the architectural mode. For information about operating system support, see Chapter 8, Software support on page 243.
Logically partitioned mode

The zEC12 only runs in LPAR mode. Each of the 60 logical partitions can be defined to operate in one of the following image modes: ESA/390 mode, to run the following systems: A z/Architecture operating system, on dedicated or shared CPs An ESA/390 operating system, on dedicated or shared CPs A Linux on System z operating system, on dedicated or shared CPs z/OS, on any of the following processor units: Dedicated or shared CPs Dedicated CPs and dedicated zAAPs or zIIPs Shared CPs and shared zAAPs or zIIPs Attention: zAAPs and zIIPs can be defined to an ESA/390 mode or z/VM-mode image (see Table 3-5 on page 110). However, zAAPs and zIIPs are exploited only by z/OS. Other operating systems cannot use zAAPs or zIIPs, even if they are defined to the logical partition. z/VM V5R4 and later provide support for real and virtual zAAPs and zIIPs to guest z/OS systems. ESA/390 TPF mode, to run the z/TPF operating system, on dedicated or shared CPs Coupling facility mode, by loading the CFCC code into the logical partition defined as: 110
8049ch03.fm
Dedicated or shared CPs Dedicated or shared ICFs Linux-only mode, to run these systems: A Linux on System z operating system, on either: Dedicated or shared IFLs Dedicated or shared CPs Dedicated or shared IFLs Dedicated or shared CPs
A z/VM operating system, on either:
z/VM-mode to run z/VM on dedicated or shared CPs or IFLs, plus zAAPs, zIIPs, and ICFs. zAware mode, to run by loading the zAware code into the logical partition defined as: Dedicated or shared CPs Dedicated or shared IFLs Table 3-6 shows all LPAR modes, required characterized PUs, operating systems, and the PU characterizations that can be configured to a logical partition image. The available combinations of dedicated (DED) and shared (SHR) processors are also shown. For all combinations, a logical partition can also have reserved processors defined, allowing nondisruptive logical partition upgrades.
Table 3-6 LPAR mode and PU usage LPAR mode ESA/390 PU type CPs Operating systems z/Architecture operating systems ESA/390 operating systems Linux on System z z/OS z/VM (V5R4 and later for guest exploitation) z/TPF CFCC Linux on System z z/VM z/VM (V5R4 and later) PUs usage CPs DED or CPs SHR
CPs and zAAPs or zIIPs ESA/390 TPF Coupling facility LINUX only z/VM CPs ICFs or CPs IFLs or CPs CPs, IFLs, zAAPs, zIIPs, ICFs IFLs, or CPs
CPs DED and zAAPs DED, and (or) zIIPs DED or CPs SHR and zAAPs SHR or zIIPs SHR CPs DED or CPs SHR ICFs DED or ICFs SHR, or CPs DED or CPs SHR IFLs DED or IFLs SHR, or CPs DED or CPs SHR All PUs must be SHR or DED
zAware
zAware
IFLs DED or IFLs SHR, or CPs DED or CPs SHR
Dynamic add or delete of a logical partition name

Dynamic add or delete of a logical partition name is the ability to add or delete logical partitions and their associated I/O resources to or from the configuration without a power-on reset. The extra channel subsystem and multiple image facility (MIF) image ID pairs (CSSID/MIFID) can later be assigned to a logical partition for use (or later removed) through dynamic I/O
111
8049ch03.fm
commands using the Hardware Configuration Definition (HCD). At the same time, required channels have to be defined for the new logical partition. Partition profile: Cryptographic coprocessors are not tied to partition numbers or MIF IDs. They are set up with Adjunct Processor (AP) numbers and domain indices. These are assigned to a partition profile of a given name. The customer assigns these AP and domains to the partitions and continues to have the responsibility to clear them out when their profiles change.
Adding crypto feature to a logical partition

You can preplan the addition of a Crypto Express3 or Crypto Express4S features to a logical partition on the crypto page in the image profile by defining the Cryptographic Candidate List, Cryptographic Online List and, Usage and Control Domain Indices in advance of installation. By using the Change LPAR Cryptographic Controls task, it is possible to add crypto adapters dynamically to a logical partition without an outage of the LPAR. Also, dynamic deletion or moving of these features does not require pre-planning. Support is provided in z/OS, z/VM, z/VSE, and Linux on System z.
LPAR dynamic PU reassignment

System configuration has been enhanced to optimize the PU-to-book assignment of physical processors dynamically. The initial assignment of customer usable physical processors to physical books, as described on 3.4.10, Processor unit assignment, can change dynamically to better suit the actual logical partition configurations that are in use. Swapping of specialty engines and general processors with each other, with spare PUs, or with both, can occur as the system attempts to compact logical partition configurations into physical configurations that span the least number of books. LPAR dynamic PU reassignment can swap customer processors of different types between books. For example, reassignment can swap an IFL on book 1 with a CP on book 2. Swaps can also occur between PU chips within a book, and includes spare PUs in what can be swapped within books. The goals are to further pack the logical partition on fewer books and also on fewer PU chips, based on the zEC12 book topology. The effect of this process is evident in dedicated and shared logical partitions that use HiperDispatch. LPAR dynamic PU reassignment is transparent to operating systems.
LPAR group capacity limit

The group capacity limit feature allows the definition of a group of LPARs on a zEC12 system and a limit for the combined capacity usage by those LPARs. This allows the system to manage the group in such a way that the group capacity limits in MSUs per hour will not be exceeded. To take advantage of this feature, you must be running z/OS V1.10 or later in the all logical partitions in the group. PR/SM and WLM work together to enforce the capacity defined for the group and also enforce the capacity optionally defined for each individual logical partition.
3.6.2 Storage operations

In the zEC12, memory can be assigned as a combination of central storage and expanded storage, supporting up to 60 logical partitions. Expanded storage is only used by the z/VM operating system.
112
8049ch03.fm
Before activating a logical partition, central storage (and, optionally, expanded storage) must be defined to the logical partition. All installed storage can be configured as central storage. Each individual logical partition can be defined with a maximum of 1 TB of central storage. Central storage can be dynamically assigned to expanded storage and back to central storage as needed without a power-on reset (POR). For details, see LPAR single storage pool on page 106. Memory cannot be shared between system images. It is possible to dynamically reallocate storage resources for z/Architecture logical partitions running operating systems that support dynamic storage reconfiguration (DSR). This is supported by z/OS, and z/VM V5R4 and later releases. z/VM in turn virtualizes this support to its guests. For details, see 3.6.5, LPAR dynamic storage reconfiguration on page 116. Operating systems running under z/VM can exploit the z/VM capability of implementing virtual memory to guest virtual machines. The z/VM dedicated real storage can be shared between guest operating systems. Table 3-7 shows the zEC12 storage allocation and usage possibilities, depending on the image mode.
Table 3-7 Storage definition and usage possibilities Image mode Architecture mode (addressability) Maximum central storage Architecture 16 EB 2 GB 2 GB 1.5 TB 16 EB 2 GB 16 EB 16 EB zEC12 definition 1 TB 128 GB 2 GB 1 TB 256 GB 2 GB 256 GB 1 TB Expanded storage zEC12 definable Yes Yes Yes No Yes Yes Yes Yes Operating system usagea Yes Yes No No Only by z/VM Only by z/VM Yes No
ESA/390
z/Architecture (64-bit) ESA/390 (31-bit)
ESA/390 TPF Coupling facility Linux only
ESA/390 (31-bit) CFCC (64-bit) z/Architecture (64-bit) ESA/390 (31-bit)
z/VMb zAware
z/Architecture (64-bit) zAware (64-bit)
a. z/VM supports the use of expanded storage. b. z/VM-mode is supported by z/VM V5R4 and later.
ESA/390 mode
In ESA/390 mode, storage addressing can be 31 or 64 bits, depending on the operating system architecture and the operating system configuration. An ESA/390 mode image is always initiated in 31-bit addressing mode. During its initialization, a z/Architecture operating system can change it to 64-bit addressing mode and operate in the z/Architecture mode. Certain z/Architecture operating systems, such as z/OS, always change the 31-bit addressing mode and operate in 64-bit mode. Other z/Architecture operating systems, such as z/VM, can be configured to change to 64-bit mode or to stay in 31-bit mode and operate in the ESA/390 architecture mode.
113
8049ch03.fm
The following modes are provided: z/Architecture mode: In z/Architecture mode, storage addressing is 64-bit, allowing for virtual addresses up to 16 exabytes (16 EB). The 64-bit architecture theoretically allows a maximum of 16 EB to be used as central storage. However, the current central storage limit for logical partitions is 1 TB of central storage. The operating system that runs in z/Architecture mode has to be able to support the real storage. Currently, z/OS for example, supports up to 4 TB of real storage (z/OS V1R10 and higher releases). Expanded storage can also be configured to an image running an operating system in z/Architecture mode. However, only z/VM is able to use expanded storage. Any other operating system running in z/Architecture mode (such as a z/OS or a Linux on System z image) does not address the configured expanded storage. This expanded storage remains configured to this image and is unused. ESA/390 architecture mode: In ESA/390 architecture mode, storage addressing is 31-bit, allowing for virtual addresses up to 2 GB. A maximum of 2 GB can be used for central storage. Because the processor storage can be configured as central and expanded storage, memory above 2 GB can be configured as expanded storage. In addition, this mode permits the use of either 24-bit or 31-bit addressing, under program control. Because an ESA/390 mode image can be defined with up to 128 GB of central storage, the central storage above 2 GB is not used, but remains configured to this image. Storage resources: Either a z/Architecture mode or an ESA/390 architecture mode operating system can run in an ESA/390 image on a zEC12. Any ESA/390 image can be defined with more than 2 GB of central storage and can have expanded storage. These options allow you to configure more storage resources than the operating system is capable of addressing.
ESA/390 TPF mode

In ESA/390 TPF mode, storage addressing follows the ESA/390 architecture mode; the z/TPF operating system runs in the 64-bit addressing mode.
Coupling Facility mode

In Coupling Facility mode, storage addressing is 64-bit for a coupling facility image running CFCC Level 12 or later, allowing for an addressing range up to 16 EB. However, the current zEC12 definition limit for logical partitions is 1 TB of storage. CFCC Level 18, which is available for the zEC12, introduces the several enhancements in the performance, reporting and serviceability areas. For details, see Coupling facility control code on page 120. Expanded storage cannot be defined for a coupling facility image. Only IBM CFCC can run in Coupling Facility mode.
Linux-only mode
In Linux-only mode, storage addressing can be 31-bit or 64-bit, depending on the operating system architecture and the operating system configuration, in exactly the same way as in ESA/390 mode.
114
8049ch03.fm
Only Linux and z/VM operating systems can run in Linux-only mode. Linux on System z 64-bit distributions (SUSE SLES 10 and later, Red Hat RHEL 5 and later) use 64-bit addressing and operate in the z/Architecture mode. z/VM also uses 64-bit addressing and operates in the z/Architecture mode.
z/VM mode
In z/VM mode, certain types of processor units can be defined within one LPAR. This increases flexibility and simplifies systems management by allowing z/VM to perform the following tasks all in the same z/VM LPAR: Manage guests to operate Linux on System z on IFLs. Operate z/VSE and z/OS on CPs. Offload z/OS system software overhead, such as DB2 workloads on zIIPs. Provide an economical Java execution environment under z/OS on zAAPs or on zIIPs.
zAware mode
In IBM zAware mode, storage addressing is 64-bit for a IBM zAware image running IBM System z Advanced Workload Analysis Reporter firmware, allowing for an addressing range up to 16 EB. However, the current zEC12 definition limit for logical partitions is 1 TB of storage. IBM zAware feature, which is exclusive to zEC12, allows the following capabilities: Help in detecting and diagnosing unusual behavior of z/OS images in near real time. Reduces problem determination time and improve service availability beyond standard z/OS features. Provides an easy to use graphical user interface with quick drill-down capabilities to view analytical data about z/OS behavior. For details, see Appendix A, IBM zAware on page 429. Only IBM zAware firmware can run in IBM zAware mode.
3.6.3 Reserved storage

Reserved storage can optionally be defined to a logical partition, allowing a nondisruptive image memory upgrade for this partition. Reserved storage can be defined to both central and expanded storage, and to any image mode, except the coupling facility mode. A logical partition must define an amount of central storage and, optionally (if not a coupling facility image), an amount of expanded storage. Both central and expanded storages can have two storage sizes defined: The initial value is the storage size allocated to the partition when it is activated. The reserved value is an additional storage capacity beyond its initial storage size that a logical partition can acquire dynamically. The reserved storage sizes defined to a logical partition do not have to be available when the partition is activated. They are simply predefined storage sizes to allow a storage increase, from a logical partition point of view. Without the reserved storage definition, a logical partition storage upgrade is disruptive, requiring the following actions: 1. Partition deactivation 2. An initial storage size definition change 3. Partition activation. The additional storage capacity to a logical partition upgrade can come from:
115
8049ch03.fm
Any unused available storage Another partition that has released storage A memory upgrade. A concurrent logical partition storage upgrade uses dynamic storage reconfiguration (DSR). z/OS uses the reconfigurable storage unit (RSU) definition to add or remove storage units in a nondisruptive way. z/VM V5R4 and later releases support the dynamic addition of memory to a running logical partition by using reserved storage, and also virtualizes this support to its guests. Removal of storage from the guests or z/VM is disruptive. SUSE Linux Enterprise Server (SLES) 11 supports both concurrent add and remove.
3.6.4 Logical partition storage granularity

Granularity of central storage for a logical partition depends on the largest central storage amount defined for either initial or reserved central storage, as shown in Table 3-8.
Table 3-8 Logical partition main storage granularity Logical partition: Largest main storage amount Central storage amount <= 128 GB 128 GB < central storage amount <= 256 GB 256 GB < central storage amount <= 512 GB 512 GB < central storage amount <= 1 TB Logical partition: Central storage granularity 256 MB 512 MB 1 GB 2 GB
The granularity applies across all central storage defined, both initial and reserved. For example, for a logical partition with an initial storage amount of 30 GB and a reserved storage amount of 48 GB, the central storage granularity of both initial and reserved central storage is 256 MB. Expanded storage granularity is fixed at 256 MB. Logical partition storage granularity information is required for logical partition image setup and for z/OS Reconfigurable Storage Units (RSU) definition. Logical partitions are limited to a maximum size of 1 TB of central storage. For z/VM V5R4 and later the limitation is 256 GB.
3.6.5 LPAR dynamic storage reconfiguration

Dynamic storage reconfiguration on the zEC12 allows an operating system running in a logical partition to add (nondisruptively) its reserved storage amount to its configuration, if any unused storage exists. This unused storage can be obtained when another logical partition releases storage or when a concurrent memory upgrade takes place. With dynamic storage reconfiguration, the unused storage does not have to be continuous. When an operating system running in a logical partition assigns a storage increment to its configuration, Processor Resource/Systems Manager (PR/SM) determines whether any free storage increments are available and dynamically brings the storage online.
116
8049ch03.fm
PR/SM dynamically takes offline a storage increment and makes it available to other partitions when an operating system running in a logical partition releases a storage increment.
3.7 Intelligent resource director

Intelligent Resource Director (IRD) is a zEC12 and System z capability exploited by z/OS only. IRD is a function that optimizes processor CPU and channel resource utilization across logical partitions within a single System z system. This feature extends the concept of goal-oriented resource management by allowing grouping system images that are resident on the same zEC12 or System z running in LPAR mode, and in the same Parallel Sysplex, into an LPAR cluster. This gives Workload Manager the ability to manage resources, both processor and I/O, not just in one single image, but across the entire cluster of system images. Figure 3-10 shows an LPAR cluster. It contains three z/OS images, and one Linux image managed by the cluster. Note that included as part of the entire Parallel Sysplex is another z/OS image, and a coupling facility image. In this example, the scope that IRD has control over is the defined LPAR cluster.
LPAR Clust er
z/OS
z/OS
z/OS
Linux
S Y S P L E X
z/OS
CF
System z
Figure 3-10 IRD LPAR cluster example
IRD addresses three separate but mutually supportive functions: LPAR CPU management: WLM dynamically adjusts the number of logical processors within a logical partition and the processor weight based on the WLM policy. The ability to move the CPU weights across an LPAR cluster provides processing power to where it is most needed, based on WLM goal mode policy.
117
8049ch03.fm
This function is automatically deactivated when HiperDispatch is active. HiperDispatch was introduced in 3.6, Logical partitioning on page 107. HiperDispatch manages the number of logical CPs in use. It adjusts the number of logical processors within a logical partition in order to achieve the optimal balance between CP resources and the requirements of the workload in the logical partition. HiperDispatch also adjusts the number of logical processors. The goal is to map the logical processor to as few physical processors as possible, attempting use the processor resources more efficiently by trying to stay within the local cache structure, making efficient use of the advantages of the high-frequency microprocessors and improving throughput and response times. Dynamic channel path management (DCM): DCM moves FICON channel bandwidth between disk control units to address current processing needs. The zEC12 supports DCM within a channel subsystem. Channel subsystem priority queuing: This function on the zEC12 and System z allows the priority queuing of I/O requests in the channel subsystem and the specification of relative priority among logical partitions. WLM, executing in goal mode, sets the priority for a logical partition and coordinates this activity among clustered logical partitions. For information about implementing LPAR CPU management under IRD, see z/OS Intelligent Resource Director, SG24-5952.
3.8 Clustering technology

Parallel Sysplex continues to be the clustering technology used with zEC12. Figure 3-11 illustrates the components of a Parallel Sysplex as implemented within the System z architecture. The figure is intended only as an example. It shows one of many possible Parallel Sysplex configurations.
118
8049ch03.fm
z10 EC
CF01
ICF
zEC12
z/OS PSIFB or IC
PS
B IF
PS
IS
* -3
e (p
r ) Time Synchronization provided

by Server Time Protocol
IF B
IS
C -3
z196
(p ee r)
PSIFB ISC-3* (peer)
Sysplex LPARs
z/OS
Sysplex LPARs
CF02 ICF
PSIFB - Parall el Sysplex InfiniBa nd ISC - InterSystem Channel FICON
Disks
Figure 3-11 Sysplex hardware overview
Disks
Disks
Figure 3-11 on page 119 shows a zEC12 system containing multiple z/OS sysplex partitions and an internal coupling facility (CF02), a z10 EC system containing a stand-alone CF (CF01), and a z196 containing multiple z/OS sysplex partitions. STP over coupling links provides time synchronization to all systems. Appropriate CF link technology (1x IFB or 12x IFB) selection depends on the system configuration and how distant they are physically located. ISC-3 links can be carried forward to zEC12 only when upgrading from either z196 or z10 EC. The ICB-4 coupling link is not supported on both zEC12 and zEnterprise CPCs. Link technologies are described in 4.10.1, Coupling links on page 161. Parallel Sysplex technology is an enabling technology, allowing highly reliable, redundant, and robust System z technology to achieve near-continuous availability. A Parallel Sysplex comprises one or more (z/OS) operating system images coupled through one or more Coupling Facilities. The images can be combined together to form clusters. A properly configured Parallel Sysplex cluster maximizes availability, as follows: Continuous (application) availability: Changes can be introduced, such as software upgrades, one image at a time, while the remaining images continue to process work. For details, see Parallel Sysplex Application Considerations, SG24-6523. High capacity: Scales can be from 2 to 32 images. Dynamic workload balancing: Viewed as a single logical resource, work can be directed to any similar operating system image in a Parallel Sysplex cluster having available capacity. Systems management: Architecture provides the infrastructure to satisfy customer requirements for continuous availability, and provides techniques for achieving simplified systems management consistent with this requirement.
119
8049ch03.fm
Resource sharing: A number of base (z/OS) components exploit coupling facility shared storage. This exploitation enables sharing of physical resources with significant improvements in cost, performance, and simplified systems management. Single system image: The collection of system images in the Parallel Sysplex appears as a single entity to the operator, the user, the database administrator, and so on. A single system image ensures reduced complexity from both operational and definition perspectives. N-2 support Multiple hardware generations (normally three) are supported in the same Parallel Sysplex. This provides for a gradual evolution of the systems in the Sysplex, without forcing changing all simultaneously. Similarly, software support for multiple releases or versions is supported. Through state-of-the-art cluster technology, the power of multiple images can be harnessed to work in concert on common workloads. The System z Parallel Sysplex cluster takes the commercial strengths of the platform to improved levels of system management, competitive price for performance, scalable growth, and continuous availability.
3.8.1 Coupling facility control code

Coupling facility control code (CFCC) Level 18 is available on the zEC12. CFCC Level 18 introduces several enhancements in the performance, reporting and serviceability areas.
Performance improvements
CFCC Level 18 introduces improvements in cache structure management: Dynamic structure size alter is enhanced to improve the performance of changing cache structure size. DB2 Global Buffer Pool (GBP) write-around (cache bypass) supports a new conditional write to GBP command. DB2 can use this enhancement during batch update/insert processing to intelligently decide which entries should be written to the GBP cache, and which should just be written around the cache to disk. Before this enhancement, overrunning cache structures with useless directory entries and changed data during batch update/insert jobs (e.g. reorganizations) caused several issues such as CF overhead, thrashing the cache through LRU processing, and leaded to castout processing backlogs and delays. CF castout class contention avoidance reduces latch contention with more granular class assignments. CF storage class contention avoidance improves response time by changing the latching from a suspend lock to a spin lock. CF Engine performance is improved by more efficient use of shared-processor CF images with good service times; as well as latency reduction for asynchronous CF operations and asynchronous CF notifications.
Coupling channel reporting

CFCC Level 18 provides additional Coupling channel characteristics reporting to z/OS by allowing it to know about the underlying InfiniBand hardware. This change would enable RMF to distinguish between CIB CHPID types (12x IFB, 12x IFB3 and 1x IFB), and detect if there is any degradation in performance on CIB channels. RMF exploits changed XES interface and obtains new channel path characteristics. The new channel path characteristics are: Stored in a new channel path data section of SMF record 74 subtype 4 120
8049ch03.fm
Added to the Subchannel Activity and CF To CF Activity sections of the RMF Postprocessor Coupling Facility Activity report Provided on the Subchannels Details panel of the RMF Monitor III Coupling Facility Systems report.
Serviceability enhancements
Serviceability enhancements provides help in debugging problems. Additional structure control info in CF dumps Prior to CFCC Level 18, only CF control structures were dumped and no structure-related controls were included. With CFCC Level 18, new structure control info included in CF dumps, though data elements (customer data) are still not dumped Enhanced CFCC tracing support CFCC Level 18 has significantly enhanced trace points, especially in areas like latching, suspend queue management/dispatching, duplexing protocols and sublist notification. Enhanced Triggers for CF non-disruptive dumping for soft-failure cases beyond break-duplexing The Coupling Facility Control Code (CFCC), the CF Operating System, is implemented using the active wait technique. This technique means that the CFCC is always running (processing or searching for service) and never enters a wait state. This also means that the CF Control Code uses all the processor capacity (cycles) available for the coupling facility logical partition. If the LPAR running the CFCC has only dedicated processors (CPs or ICFs), then using all processor capacity (cycles) is not a problem. However, this can be an issue if the LPAR that is running the CFCC also has shared processors. Therefore, it is best to enable dynamic dispatching on the CF LPAR. CF structure sizing changes are expected when going from CFCC Level 17 (or below) to CFCC Level 18. We suggest reviewing the CF LPAR size by using the CFSizer tool available at the following web page: http://www.ibm.com/systems/z/cfsizer/
3.8.2 Dynamic CF dispatching

Dynamic CF dispatching provides the following function on a coupling facility: 1. If there is no work to do, CF enters a wait state (by time). 2. After an elapsed time, CF wakes up to see whether there is any new work to do (requests in the CF Receiver buffer). 3. If there is no work, CF sleeps again for a longer period of time. 4. If there is new work, CF enters into the normal active wait until there is no more work, starting the process all over again. This function saves processor cycles and is an excellent option to be used by a production backup CF or a testing environment CF. This function is activated by the CFCC command DYNDISP ON. The CPs can run z/OS operating system images and CF images. For software charging reasons, using only ICF processors to run coupling facility images is better. Figure 3-12 shows the dynamic CF dispatching.
121
8049ch03.fm
ICF
CP
Partition Image Profile
With Dynamic CF dispatching enabled, backup CF becomes active only for new work
ACTIVE CF
z/OS
CF
z/OS
BACK UP CF
z/OS
CF
z/OS
HMC
Setup
DYNDISP ON (CFCC cmd) IMAGE Profile setup
Function
Figure 3-12 Dynamic CF dispatching (shared CPs or shared ICF PUs)
For additional details regarding CF configurations, see Coupling Facility Configuration Options, GF22-5042, also available from the Parallel Sysplex website: http://www.ibm.com/systems/z/advantages/pso/index.html
122
8049ch04.fm
Chapter 4.
CPC I/O System Structure

This chapter describes the I/O system structure and the connectivity options available on the IBM zEnterprise EC12 (zEC12). This chapter discusses the following topics: I/O system overview on page 126 I/O cages on page 127 I/O drawers on page 130 PCIe I/O drawers on page 132 I/O cage, I/O drawer and PCIe I/O drawer offerings on page 134 Fanouts on page 136 I/O feature cards on page 144 Connectivity on page 146 Parallel Sysplex connectivity on page 161 Cryptographic functions on page 167 Flash Express on page 167
123
8049ch04.fm
4.1 Introduction to InfiniBand and PCIe

This section describes the infrastructure and other considerations for InfiniBand and PCIe links.
4.1.1 Infrastructure types

The zEC12 supports two different types of internal I/O infrastructure: InfiniBand-based infrastructure for I/O cages and I/O drawers PCIe-based infrastructure for PCIe I/O drawers (new form factor drawer and I/O features)
InfiniBand I/O infrastructure

The InfiniBand I/O infrastructure was first made available on System z10 and is supported on zEC12. It consists of: InfiniBand fanouts supporting the current 6 GBps InfiniBand I/O interconnect InfiniBand I/O card domain multiplexers with redundant I/O interconnect in: The 14U, 28-slot, 7-domain I/O cage The 5U, 8-slot, 2-domain IO drawer
PCIe I/O infrastructure

IBM extends the use of industry standards on the System z platform by offering a Peripheral Component Interconnect Express Generation 2 (PCIe Gen2) I/O infrastructure. The PCIe I/O infrastructure provided by the zEnterprise CPCs improves I/O capability and flexibility, while allowing for the future integration of PCIe adapters and accelerators. The zEC12 PCIe I/O infrastructure consists of the following components: PCIe fanouts supporting 8 GBps I/O bus interconnection for processor book connectivity to the PCIe I/O drawers The 7U, 32-slot, 4-domain PCIe IO drawer for PCIe I/O features The benefits of the zEnterprise PCIe I/O infrastructure include these: Bandwidth: Increased bandwidth from the processor book or drawer to the I/O domain in the PCIe I/O drawer through an eight gigabytes per second (8 GBps) bus. 14% more capacity: Two PCIe I/O drawers occupy the same space as one I/O cage. Up to 128 channels (64 PCIe I/O features) are supported versus the 112 channels (28 I/O features) offered with the I/O cage. Better granularity for the storage area network (SAN) and the local area network (LAN): For the FICON, zHPF and FCP storage area networks, the FICON Express8S has two channels per feature. For LAN connectivity, the OSA-Express4S GbE features have two ports each, the OSA-Express4S 10 GbE features have one port each, and the OSA-Express4S 1000Base-T features have two ports each.
124
8049ch04.fm
4.1.2 InfiniBand specification

The InfiniBand specification defines the raw bandwidth of one lane (referred to as 1x) connection at 2.5 Gbps. Two additional lane widths are specified, referred to as 4x and 12x, as multipliers of the base link width. Similar to Fibre Channel, PCI Express, Serial ATA, and many other contemporary interconnects, InfiniBand is a point-to-point, bidirectional serial link intended for the connection of processors with high-speed peripherals, such as disks. InfiniBand supports various signalling rates and, as with PCI Express, links can be bonded together for additional bandwidth. The serial connection's signalling rate is 2.5 Gbps on one lane in each direction, per physical connection. Currently, InfiniBand also supports 5 Gbps or 10 Gbps signaling rates, respectively.
4.1.3 Data, signalling, and link rates

Links use 8b/10b encoding (every ten bits sent carry eight bits of data), so that the useful data transmission rate is four-fifths of the signalling rate (signalling rate equals raw bit rate). Thus, links carry 2, 4, or 8 Gbps of useful data, respectively, for 1x link. Links can be aggregated in units of 4 or 12, indicated as 4x1 or 12x. A 12x link therefore carries 120 Gbps raw or 96 Gbps of payload (useful) data. Larger systems with 12x links are typically used for cluster and supercomputer interconnects, as implemented on the zEC12, and for inter-switch connections. For details and the standard for InfiniBand, see the InfiniBand website: http://www.infinibandta.org InfiniBand functions on zEC12: Not all properties and functions offered by InfiniBand are implemented on the zEC12. Only a subset is used to fulfill the interconnect requirements that have been defined for zEC12.
4.1.4 PCIe
PCIe is a serial bus with embedded clock and uses 8b/10b encoding, where every eight bits are encoded into a 10-bit symbol that is then decoded at the receiver. Thus, the bus needs to transfer 10 bits to send 8 bits of actual usable data. A PCIe bus generation 2 single lane can transfer 5 Gbps of raw data (duplex connection), which is 10 Gbps of raw data. From these 10 Gbps, only 8 Gbps are actual data (payload). Therefore an x16 (16 lanes) PCIe gen2 bus transfers 160 Gbps encoded, which is 128 Gbps of unencoded data (payload). This is 20 GBps raw data and 16 GBps of encoded data. The new measuring unit for transfer rates for PCIe is GT/s (Giga Transfers per second) which refers to the raw data (even though only 80% of this transfer is actual payload data). The translation between GT/s to GBps is: 5 GT/s equals 20 GBps or 1 GT/s equals 4 GBps. The 16 lanes of the PCIe bus are virtual lanes, always consisting of one transmit and one receive lane. Each of these lanes consists of two physical copper wires. The physical method used to transmit signals is a differential bus, which means that the signal is encoded into the different voltage levels between two wires (as opposed to one voltage level on one wire in
1
zEC12 does not support this data rate
Chapter 4. CPC I/O System Structure
125
8049ch04.fm
comparison to the ground signal). Therefore, each of the 16 PCIe lanes uses actually four copper wires for the signal transmissions.
4.2 I/O system overview

This section lists characteristics and a summary of features.
4.2.1 Characteristics
The zEC12 I/O subsystem design provides great flexibility, high availability, and excellent performance characteristics, as follows: High bandwidth: The zEC12 uses PCIe as an internal interconnect protocol to drive PCIe I/O drawers. The I/O bus infrastructure data rate increases up to 8 GBps. The zEC12 uses InfiniBand as the internal interconnect protocol to drive I/O cages and I/O drawers and CPC to CPC connection. InfiniBand supports I/O bus infrastructure data rate up to 6 GBps. Connectivity options: The zEC12 can be connected to a range of interfaces such as FICON/Fibre Channel Protocol for storage area network connectivity, and for CPC to CPC connectivity by FICON CTCs (FCTC), 10 Gigabit Ethernet, Gigabit Ethernet, and 1000BASE-T Ethernet for local area network connectivity. For CPC to CPC connection, zEC12 uses Parallel Sysplex InfiniBand (PSIFB), ISC-3 coupling links. Concurrent I/O upgrade: You can concurrently add I/O cards to the server if an unused I/O slot position is available. Concurrent PCIe I/O drawer upgrade: Additional PCIe I/O drawers can be installed concurrently without preplanning if there is free space in one of the frames. Dynamic I/O configuration: Dynamic I/O configuration supports the dynamic addition, removal, or modification of channel path, control units, and I/O devices without a planned outage. Pluggable optics: The FICON Express8S, FICON Express8, and FICON Express4 features have Small Form Factor Pluggable (SFP) optics to permit each channel to be individually serviced in the event of a fiber optic module failure. The traffic on the other channels on the same feature can continue to flow if a channel requires servicing. Concurrent I/O card maintenance: Every I/O card plugged in an I/O cage, I/O drawer, or PCIe I/O drawer supports concurrent card replacement in case of a repair action.
4.2.2 Summary of supported I/O features

The following I/O features are supported: Up to 176 FICON Express4 channels 126
8049ch04.fm
Up to 176 FICON Express8 channels Up to 320 FICON Express8S channels Up to 96 OSA-Express3 ports Up to 96 OSA-Express4S ports Up to 48 ISC-3 coupling links Up to 16 InfiniBand fanouts: Up to 32 12x InfiniBand coupling links with HCA2-O fanout, or Up to 32 1x InfiniBand coupling links with HCA2-O LR (1xIFB) fanout, or Up to 32 12x InfiniBand coupling links with HCA3-O fanout, or Up to 64 1x InfiniBand coupling links with HCA3-O LR (1xIFB) fanout Coupling links: The maximum number of external coupling links combined (ISC-3 and IFB coupling links) cannot exceed 112 for each zEC12.
4.3 I/O cages

The I/O cage is fourteen EIA units high. Each cage supports up to seven I/O domains for a total of 28 I/O card slots. Each I/O domain supports four I/O card slots and uses an IFB-MP card in the I/O cage and a copper cable connected to an Host Channel Adapter (HCA-C) fanout in the CPC cage. An eight IFB-MP card is installed to provide an alternate path to I/O cards in slots 29, 30, 31, and 32 in case of a repair action.
Figure 4-1 I/O cage
Figure 4-2 on page 128 illustrates the I/O structure of an I/O cage. An InfiniBand (IFB) cable connects the HCA2-C fanout to an IFB-MP card in the I/O cage. The passive connection between two IFB-MP cards allows for redundant I/O interconnection. The IFB cable between
127
8049ch04.fm
an HCA2-C fanout in a book and each IFB-MP card in the I/O cage supports a 6 GBps bandwidth.
Processor
Memory
HCA2-C Fanout
8 x 6 GBps I/O Interconnect Passive Connection for Redundant I/O Interconnect
IFB-MP
IFB-MP
I/O Domain FICON/FCP
I/O Domain FICON/FCP
Crypto
Crypto
OSA
ISC
OSA
I/O Cage
Figure 4-2 EC12 I/O structure when using I/O cage
Note: Only one I/O cage is supported in zEC12, carry forward only. The I/O cage domains and their related I/O slots are shown in Figure 4-3 on page 129
128
ISC
8049ch04.fm
I/O cage 28 slots 7 I/O domains

1A 2B 3A 4B
RII
B W I R N G D E
25E 26F 27E 28F 17G 18G 19G 20G
IFB-MP
5A 6B 7A 8B 9C 10D 11C 12D
G
21E 22F 23E 24F
C
13C 14D 15C 16D
Front 16
Rear 12
Figure 4-3 I/O domains of I/O cage
Each I/O domain supports up to four I/O cards (FICON, OSA, Crypto, or ISC). All I/O cards are connected to the IFB-MP cards through the backplane board. Table 4-1 lists the I/O domains and their related I/O slots.
Table 4-1 I/O domains of I/O cage Domain number (name) 0 (A) 1 (B) 2 (C) 3 (D) 4 (E) 5 (F) 6 (G) I/O slot in domain 01, 03, 06, 08 02, 04, 07, 09 10, 12, 15, 17 11, 13, 16, 18 19, 21, 24, 26 20, 22, 25, 27 29, 30, 31, 32
Power Sequence Controller (PSC) feature is not supported on zEC12
129
8049ch04.fm
4.4 I/O drawers

The I/O drawer is five EIA units high and supports up to eight I/O feature cards. Each I/O drawer supports two I/O domains (A and B) for a total of eight I/O card slots. Each I/O domain uses an IFB-MP card in the I/O drawer and a copper cable to connect to a Host Channel Adapter (HCA) fanout in the CPC cage. The link between the HCA in the CPC and the IFB-MP in the I/O drawer supports a link rate of up to 6 GBps. All cards in the I/O drawer are installed horizontally. The two distributed converter assemblies (DCAs) distribute power to the I/O drawer. The locations of the DCAs, I/O feature cards, and IFB-MP cards in the I/O drawer are shown in Figure 4-4.
DCA
Front
L G0 1
J00
L G0 2
L G03 L G04
J01
L G05
I/O cards
Rear
LG06
J00
DCA
IFB-MP
LG07 LG08 LG09

J01
LG10 LG11
Figure 4-4 I/O drawer
The I/O structure in a zEC12 server is illustrated in Figure 4-5 on page 131. An IFB cable connects the HCA fanout card to an IFB-MP card in the I/O drawer. The passive connection between two IFB-MP cards allows redundant I/O interconnection (RII). This provides connectivity between an HCA fanout card, and I/O cards in case of concurrent fanout card or IFB cable replacement. The IFB cable between an HCA fanout card and each IFB-MP card supports a 6 GBps link rate.
130
8049ch04.fm
Processor
Memory
HCA2-C Fanout
8 x 6 GBps I/O Interconnect Passive Connection for Redundant I/O Interconnect
IFB-MP
IFB-MP
Crypto I/O Domain FICON/FCP OSA ISC
Crypto I/O Domain FICON/FCP OSA ISC
I/O Drawer
Figure 4-5 zEC12 I/O structure when using I/O drawer
Note: Only two I/O drawers are supported in zEC12, and as carry forward only. The I/O drawer domains and their related I/O slots are shown in Figure 4-6. The IFB-MP cards are installed at slot 09 at the rear side of the I/O drawer. The I/O cards are installed from the front and rear side of the I/O drawer. Two I/O domains (A and B) are supported. Each I/O domain has up to four I/O feature cards (FICON, OSA, Crypto, or ISC). The I/O cards are connected to the IFB-MP card through the backplane boar.
I/O drawer 8 slots 2 I/O domains
1 2 3 4 5
DCA 1 I/O-A
RII
DCA 2 I/O-B I/O-A
6 7 8 9 10 11
I/O-B I/O-B I/O-A
IFB-MP B IFB-MP A
I/O-A I/O-B
Front - 4
Rear - 4
Figure 4-6 I/O domains of I/O drawer
131
8049ch04.fm
Each I/O domain supports four I/O card slots. Balancing I/O cards across both I/O domains on new build servers, or on upgrades, is automatically done when the order is placed. Table 4-2 lists the I/O domains and their related I/O slots.
Table 4-2 I/O domains of I/O drawer Domain A B I/O slot in domain 02, 05, 08, 10 03, 04, 07, 11
Power Sequence Controller (PSC) feature is not supported on zEC12
4.5 PCIe I/O drawers

The PCIe I/O drawer attaches to the processor node through a PCIe bus and uses PCIe as the infrastructure bus within the drawer. The PCIe I/O bus infrastructure data rate is up to 8 GBps. PCIe switch ASICs (Application-Specific Integrated Circuit) are used to fanout the host bus from the processor node to the individual I/O cards. Up to 128 channels (64PCIe I/O features) are supported versus the 112 channels (28 I/O features) offered with the I/O cage. The PCIe drawer is a two sided drawer (I/O cards on both sides) that is 7U high (one half of I/O cage). The drawer contains 32 I/O card slots, four switch cards (two in front, two in rear), two DCAs to provide the redundant power and two air moving device (AMDs) for redundant cooling. The locations of the DCAs, AMDs, PCIe switch cards, and I/O feature cards in the PCIe I/O drawer are shown in Figure 4-7.
AMD
Front
7U (~311 mm)
FICON Express8S
PCIe switch card
OSA-Express4S
Rear
DCA
560 mm (max)
Figure 4-7 PCIe I/O drawer
132
8049ch04.fm
The I/O structure in a zEC12 server is illustrated in Figure 4-8. The PCIe switch card provides the fanout from the high speed x16 PCIe host bus to eight individual card slots. The PCIe switch card is connected to the processor nest through a single x16 PCIe Gen 2 bus from a PCIe fanout card, which converts the book internal bus into two PCIe buses. A switch card in the front is connected to a switch card in the rear through the PCIe I/O drawer board to provide a failover capability in case of a PCIe fanout card failure, book upgrade, and so on. In PCIe I/O drawer, the 8 I/O cards directly attached to the switch card constitute an I/O domain. The PCIe I/O drawer supports concurrent add and delete to enable a client to increase I/O capability as needed without plan ahead.
B oo k 0 Memo ry
PU PU PU
Bo ok 1 Memo ry
PU PU PU
SC1, S C0 (FBC)
PU PU PU
SC1, SC0 (F BC)

PU PU PU
PC Ie (8x)
x16 P CI e Gen2 8 GBps
PCIe (8x)
PC Ie s witc h
PCIe switc h
P CIe switc h
PC Ie s witc h
4 GB/ s PCIe Gen2 x8
F ICON Express8 S
OSA-Express4S 10 GbE
I/O drawer PCIe I/O d rawer
Figure 4-8 zEC12 structure when using PCIe I/O drawers
The PCIe I/O Drawer supports up to 32 I/O cards. They are organized in four hardware domains per drawer. Each domain is driven through a PCIe switch card. Always two PCIe switch cards provide a backup path for each other through the passive connection in the PCIe I/O Drawer backplane, so that in case of a PCIe fanout card or cable failure, all 16 I/O cards in the two domains can be driven through a single PCIe switch card. To support Redundant I/O Interconnect (RII) between front to back domain pairs 0-1 and 2-3 the two interconnects to each pair must be from two different PCIe fanouts (all four domains in one of these cages can be activated with two fanouts). The flexible service processors (FSPs) are used for system control. The PCIe I/O drawer domains and their related I/O slots are shown in Figure 4-9 on page 134.
133
8049ch04.fm
PCIe I/O drawer 32 slots 4 I/O domains

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
0 0 0 0
PCIe interconnect
PCIe interconnect
1 1 1 1 1 1 1 1
38 37 36 35 34 33 32 31 30 29 28 27 26 25 24 23 22 21 20
RII
FSP-1, 1
0 0 0 0 2 2 2 2
FSP-1, 2
PCIe interconnect
PCIe interconnect
3 3 3 3
2 2 2 2 Front 16
3 3 3 3 Rear 16
Figure 4-9 I/O domains of PCIe I/O drawer
Each I/O domain supports up to eight I/O cards (FICON, OSA, Crypto, and Flash Express). All I/O cards are connected to the PCIe switch card through the backplane board. Table 4-3 lists the I/O domains and slots.
Table 4-3 I/O domains of PCIe I/O drawer Domain 0 1 2 3 I/O slot in domain 01, 02, 03, 04, 06, 07, 08, 09 30, 31, 32, 33, 35, 36, 37, 38 11, 12, 13, 14, 16, 17, 18, 19 20, 21, 22, 23, 25, 26, 27, 28
Power Sequence Controller (PSC) feature is not supported on PCIe drawer and zEC12.
4.6 I/O cage, I/O drawer and PCIe I/O drawer offerings
A maximum of five PCIe drawers can be installed supporting up to 160 PCIe I/O features. Customers will not order I/O cages or I/O drawers, they will order I/O features and the configurator will determine the proper mix of I/O cage, I/O drawer, and PCIe I/O drawer.
134
8049ch04.fm
Note: On new build zEC12 only PCIe I/O drawers are supported.
4.6.1 I//O cage and I/O drawer carry forward

A mixture of I/O cages, I/O drawers, and PCIe I/O drawers are only available on upgrades to a zEC12. Depending on the amount of I/O features carried forward by an upgrade, the configurator will determine the number and mix of I/O cages, I/O drawers, and PCIe I/O drawers. Some I/O features are supported only by I/O cages and I/O drawers: FICON Express4 FICON Express8 OSA Express3 ISC-3 Crypto Express3 Depending on the amount of I/O cages and I/O drawers, PCIe I/O drawer will be added to support PCIe I/O features: FICON Express8S OSA Express4S Crypto Express4S Flash Express Table 4-4 gives an overview of the number of I/O cages, I/O drawers and PCIe drawers that can be present in a zEC12.
Table 4-4 I/O cage and drawer summary Description I/O Cage I/O Drawer PCIe I/O Drawer New Build 0 0 0-5 Carry Forward 0-1 0-2 0-5 MES Add 0 0 0-5
A maximum of 44 I/O features can be carried forward. Table 4-5 list the number and mix of I/O cages and I/O drawers depending on amount of legacy I/O features.
Table 4-5 Number and mix of I/O cages and I/O drawers Number of I/O cards carried forward on upgrades 0 1-8 9-16 17-28 29-36 Number of I/O cages 0 0 0 1 1 Number of I/O drawers 0 1 2 0 1
135
8049ch04.fm
Number of I/O cards carried forward on upgrades 37-44
Number of I/O cages 1
Number of I/O drawers 2
4.7 Fanouts
The zEC12 server uses fanout cards to connect the I/O hardware Subsystem to the processor Books and to provide the InfiniBand coupling links for Parallel Sysplex as well. All Fanout cards support concurrent add, delete and move. zEC12 supports two different internal I/O infrastructures for the internal connection. zEC12 uses InfiniBand based infrastructure for the internal connection to I/O cages and I/O drawers and uses PCIe based infrastructure for PCIe I/O drawers in which the cards for the connection to peripheral devices and networks reside. The InfiniBand and PCIe fanouts are located in the front of each book. Each book has eight fanout slots. They are named D1 to DA, top to bottom; slots D3 and D4 are not used for fanouts. Six types of fanout cards are supported by zEC12. Each slot holds one of the following six fanouts: Host Channel Adapter (HCA2-C): This copper fanout provides connectivity to the IFB-MP card in the I/O cage and I/O drawer. PCIe Fanout: This copper fanout provides connectivity to the PCIe switch card in the PCIe I/O drawer. Host Channel Adapter (HCA2-O (12xIFB)): This optical fanout provides 12x InfiniBand coupling link connectivity up to 150 meters distance to a zEC12, z196, z114, and System z10. Host Channel Adapter (HCA2-O LR (1xIFB)): This optical long range fanout provides 1x InfiniBand coupling link connectivity up to 10 km unrepeated distance to a zEC12, z196, z114 and System z10 servers. Host Channel Adapter (HCA3-O (12xIFB)): This optical fanout provides 12x InfiniBand coupling link connectivity up to 150 meters distance to a zEC12, z196, z114 and System z10. Host Channel Adapter (HCA3-O LR (1xIFB)): This optical long range fanout provides 1x InfiniBand coupling link connectivity up to 10 km unrepeated distance to a zEC12, z196, z114 and System z10 servers. The HCA3-O LR (1xIFB) fanout comes with 4 ports and each other fanout comes with two ports. Figure 4-10 on page 137 illustrates the IFB connection from the CPC cage to an I/O cage and an I/O drawer, and the PCIe connection from the CPC cage to a PCIe I/O drawer.
136
8049ch04.fm
Book 0 Memory
PU PU PU PU PU PU
Book 1 Me mory
PU PU PU PU PU PU
Book 2 Memory
PU PU PU PU PU PU
Book 3 Me mory
PU PU PU PU PU PU
SC1, SC0 (FBC)
SC1, SC0 (FBC) (FBC)

PU PU PU PU PU PU
SC1, SC0 (FBC)

PU PU PU PU PU PU
SC1, SC0 (FBC)

PU PU PU PU PU PU
Fanouts
PU PU
PU PU
PU PU
Fanouts
PCIe (8x )
x16 PCIe Gen2 8 GBps RII
PCIe (8x)
HCA2 (8x)
6 GBps
HCA2 (8x)
PCIe switch
PCIe switch
PCIe switch
RII
PCIe switch
IFB-MP RII
IFB-MP
IFB-MP
RII
IFB-MP
2 GBps mSTI 4 GB/s PCIe Gen2 x8 1 GBps mSTI

FP GA
2 GBpsmSTI
Channels FICON Expre ss8S OS A-E xpress4S FICO N Express8
Ports O SA-Express3
PCIe I/O drawer

Figure 4-10 PCIe and InfiniBand I/O infrastructure
I/O Cage & I/O drawer
Figure 4-11 illustrates the zEC12 coupling links.
12x Infini Band Up to 150 meters 1x Infini Band Up to 10/ 100KM 12x Infini Band Up to 150 meters .. .. .. .. 1x InfiniBand
H CA 3 -O
H CA3 -O LR
Up to 10/100KM
E C12, z196, z114, z10 E C, z10 BC
.. .. .. ..
HC A 2-O* H CA2 -O LR*
Fanouts
IFB-MP
ISC-3 ISC-3 ISC-3 ISC-3
ISC-3
Up to 10/100 km
E C12, z196 z114, z10 EC, z10 BC
H CA2 -C
I/O Dr a wer or I/0 C ag e
* * Not supp orted on the z1 14 * * Carry forwa rd onl y
Figure 4-11 EC12 coupling links
F PGA
2GBps mSTI
137
8049ch04.fm
4.7.1 HCA2-C fanout (FC0162)

The HCA2-C fanout is used to connect to an I/O cage or an I/O drawer using a copper cable. The two ports on the fanout are dedicated to I/O. The bandwidth of each port on the HCA2-C fanout supports a link rate of up to 6 GBps. A 12x InfiniBand copper cable of 1.5 to 3.5 meters long is used for connection to the IFB-MP card in the I/O cage or the I/O drawer. An HCA2-2 fanout is support only if carried forward by an upgrade. HCA2-C fanout: The HCA2-C fanout is used exclusively for I/O and cannot be shared for any other purpose
4.7.2 PCIe copper fanout (FC0169)

The PCIe fanout card supports PCIe Gen2 bus and is used to connect to the PCIe I/O drawer. PCIe fanout cards are always plugged in pairs. The bandwidth of each port on the PCIe fanout supports a link rate of up to 8GBps. The PCIe fanout supports FICON Express8S, OSA Express4S, Crypto Express 4S, and Flash Express in PCIe I/O drawer. PCIe fanout: The PCIe fanout is used exclusively for I/O and cannot be shared for any other purpose.
4.7.3 HCA2-O (12xIFB) fanout (FC0163)

The HCA2-O (12xIFB) fanout for 12x InfiniBand provides an optical interface used for coupling links. The two ports on the fanout are dedicated to coupling links to connect to zEC12, z196, z114, and System z10 servers, or to connect to a coupling port in the same server by using a fiber cable. Each fanout has an optical transmitter and receiver module and allows dual simplex operation. Up to 16 HCA2-O (12xIFB) fanouts are supported by zEC12 and provide up to 32 ports for coupling links. The HCA2-O fanout supports InfiniBand 12x optical links that offer configuration flexibility, and high bandwidth for enhanced performance of coupling links. There are 12 lanes (two fibers per lane) in the cable, which means 24 fibers are used in parallel for data transfer. Each port provides one connector for transmit and one connector for receive data. The fiber optic cables are industry standard OM3 (2000 MHz-km) 50 m multimode optical cables with Multi-Fiber Push-On (MPO) connectors. The maximum cable length is 150 meters. There are 12 pairs of fibers, 12 fibers for transmitting, and 12 fibers for receiving. Each connection supports a link rate of 6 GBps when connected to a zEC12, z196, z114, or System z10 server. HCA2-O (12xIFB) fanout: Ports on the HCA2-O (12xIFB) fanout are exclusively used for coupling links and cannot be used or shared for any other purpose.
Ports for optical link connections

A fanout has two ports for optical link connections and supports up to 16 CHPIDs across both ports. These CHPIDs are defined as channel type CIB in the IOCDS. The coupling links can 138
8049ch04.fm
be defined as shared between images within a channel subsystem and they can be also be spanned across multiple CSSs in a server. Each HCA2-O (12xIFB) fanout used for coupling links has an assigned adapter ID (AID) number that must be used for definitions in IOCDS to create a relationship between the physical fanout location and the CHPID number. For details about AID numbering, see Adapter ID number assignment on page 142. For detailed information about how the AID is used and referenced in HCD, see Implementing and Managing InfiniBand Coupling Links on System z SG24-7539. When STP is enabled, IFB coupling links can be defined as timing-only links to other zEC12, z196, z114, and System z10 servers.
4.7.4 HCA2-O LR (1xIFB) fanout (FC0168)

The HCA2-O LR (1xIFB) fanout for 1x InfiniBand provides an optical interface used for coupling links. The two ports on the fanout are dedicated to coupling links to connect to zEC12, z196, z114, and System z10 servers. IFB LR coupling link connectivity to other servers is not supported. Up to 16 HCA2-O LR (1xIFB) fanouts are supported by zEC12 and provide 32 ports for coupling links. The HCA2-O LR (1xIFB) fanout has 1x optical links that offer a longer distance of coupling links. The cable has one lane containing two fibers; one fiber is used for transmitting and one fiber used for receiving data. Each connection supports a link rate of 5 Gbps if connected to a zEC12, z196, a z114, a System z10 server or to a System z qualified Dense Wavelength Division Multiplexer (DWDM), or a data link rate of 2.5 Gbps when connected to a System z qualified DWDM. The link rate is auto-negotiated to the highest common rate. HCA2-O LR (1xIFB) fanout: Ports on the HCA2-O LR (1xIFB) fanout are used exclusively for coupling links and cannot be used or shared for any other purpose The fiber optic cables are 9 m single mode (SM) optical cables terminated with an LC Duplex connector. The maximum unrepeated distance is 10 km and up to 100 km with System z qualified DWDM.

A fanout has two ports for optical link connections and supports up to 16 CHPIDs across both ports. These CHPIDs are defined as channel type CIB in the IOCDS. The coupling links can be defined as shared between images within a channel subsystem and they can be also be spanned across multiple CSSs in a server. Each HCA2-O LR (1xIFB) fanout can be used for link definitions to another server or a link from one port to a port in another fanout on the same server. Definitions of the source and target operating system image, CF image, and the CHPIDs used on both ports in both servers, are defined in IOCDS. Each HCA2-O LR (1xIFB) fanout used for coupling links has an assigned adapter ID (AID) number that must be used for definitions in IOCDS to create a relationship between the physical fanout location and the CHPID number. See Adapter ID number assignment on page 142 for details about AID numbering.
139
8049ch04.fm
When STP is enabled, IFB LR coupling links can be defined as timing-only links to other zEC12, z196, z114, and System z10 servers.
4.7.5 HCA3-O (12xIFB) fanout (FC0171)

The HCA3-O fanout for 12x InfiniBand provides an optical interface used for coupling links. The two ports on the fanout are dedicated to coupling links to connect to zEC12, z196, z114, System z10 servers, or to connect to a coupling port in the same server by using a fiber cable. Up to 16 HCA3-O (12xIFB) fanouts are supported and provide up to 32 ports for coupling links. The fiber optic cables are industry standard OM3 (2000 MHz-km) 50 m multimode optical cables with Multi-Fiber Push-On (MPO) connectors. The maximum cable length is 150 meters. There are 12 pairs of fibers, 12 fibers for transmitting, and 12 fibers for receiving. The HCA3-O (12xIFB) fanout supports a link data rate of 6 GBps. Each port provides one connector for transmit and one connector for receive data. HCA3-O (12xIFB) fanout: Ports on the HCA3-O (12xIFB) fanout are exclusively used for coupling links and cannot be used or shared for any other purpose.

A fanout has two ports for optical link connections and supports up to 16 CHPIDs across both ports. These CHPIDs are defined as channel type CIB in the IOCDS. The coupling links can be defined as shared between images within a channel subsystem and they can be also be spanned across multiple CSSs in a server. Each HCA3-O (12xIFB) fanout used for coupling links has an assigned adapter ID (AID) number that must be used for definitions in IOCDS to create a relationship between the physical fanout location and the CHPID number. For details about AID numbering, see Adapter ID number assignment on page 142. When STP is enabled, IFB coupling links can be defined as timing-only links to other zEC12, z196, z114, and System z10 servers.
12x IFB and 12x IFB3 protocols

There are two protocols supported by the HCA3-O for 12x IFB feature: 12x IFB3 protocol: When HCA3-O (12xIFB) fanouts are communicating with HCA3-O (12xIFB) fanouts and have been defined with four or fewer CHPIDs per port, the 12x IFB3 protocol is utilized. 12x IFB protocol: If more than four CHPIDs are defined per HCA3-O (12xIFB) port, or HCA3-O (12xIFB) features are communicating with HCA2-O (12xIFB) features on zEnterprise or System z10 servers, links will run with the 12x IFB protocol. The HCA3-O feature supporting 12x InfiniBand coupling links has been designed to deliver improved services times. When no more than four CHPIDs are defined per HCA3-O (12xIFB) port, the 12x IFB3 protocol is used. When using the 12x IFB3 protocol, synchronous service times are designed to be 40% faster than when using the 12x IFB protocol.
4.7.6 HCA3-O LR (1xIFB) fanout (FC0170)

The HCA3-O LR fanout for 1x InfiniBand provides an optical interface used for coupling links. The four ports on the fanout are dedicated to coupling links to connect to zEC12, z196, z114,
140
8049ch04.fm
System z10 servers, or to connect to a coupling port in the same server by using a fiber cable. Up to 16 HCA3-O LR (1xIFB) fanouts are supported by zEC12 and provide up to 64 ports for coupling links. The HCA-O LR fanout supports InfiniBand 1x optical links that offer longer distance of coupling links. The cable has one lane containing two fibers; one fiber is used for transmitting and one fiber is used for receiving data. Each connection supports a link rate of 5 Gbps if connected to a zEC12, z196, a z114, a z10 server, or to a System z qualified DWDM, and a data link rate of 2.5 Gbps when connected to a System z qualified DWDM. The link rate is auto- negotiated to the highest common rate. HCA3-O LR (1xIFB) fanout: Ports on the HCA3-O LR (1xIFB) fanout are used exclusively for coupling links and cannot be used or shared for any other purpose The fiber optic cables are 9 m single mode (SM) optical cables terminated with an LC Duplex connector. The maximum unrepeated distance is 10 km and up to 100 km with System z qualified DWDM.

A fanout has four ports for optical link connections and supports up to 16 CHPIDs across all four ports. These CHPIDs are defined as channel type CIB in the IOCDS. The coupling links can be defined as shared between images within a channel subsystem and they can also be spanned across multiple CSSs in a server. This is compatible with the HCA2-O LR (1xIFB) fanout, which has two ports. Each HCA3-O LR (1xIFB) fanout can be used for link definitions to another server or a link from one port to a port in another fanout on the same server. Definitions of the source and target operating system image, CF image, and the CHPIDs used on both ports in both servers, are defined in IOCDS. Each HCA3-O LR (1xIFB) fanout used for coupling links has an assigned adapter ID (AID) number that must be used for definitions in IOCDS to create a relationship between the physical fanout location and the CHPID number. See Adapter ID number assignment on page 142 for details about AID numbering. When STP is enabled, IFB LR coupling links can be defined as timing-only links to other zEC12, z196, z114, and System z10 servers.
4.7.7 Fanout considerations

Because fanout slots in each book can be used to plug different fanouts, where each fanout is designed for a special purpose, certain restrictions might apply to the number of available channels located in the I/O cage, I/O drawer, and PCIe I/O drawer. Depending on the model, the number of fanouts varies. The plugging rules for fanouts for each model are illustrated in Figure 4-12 on page 142.
141
8049ch04.fm
ECF
OSC D1 D2 D3 FSP FSP
OSC
ECF
ECF
OSC D1 D2 D3 D4 D5 D6 D7 D8 D9 DA
ECF
filler
filler
filler
filler
filler
FSP FSP
H20
D4 D5 D6 D7 D8 D9 DA
H43
01
06
10
15
01
06
10
15
ECF
OSC
ECF
ECF
OSC
OSC
ECF
filler
D3 D4 D5 D6 D7 D8 D9 DA
FSP FSP
D3 D4 D5 D6 D7 D8 D9 DA
FSP FSP
H66
H89 H89
D3 D4 D5 D6 D7 D8 D9 DA
FSP FSP
D3 D4 D5 D6 D7 D8 D9 DA
FSP FSP
D3
FSP
D3
FSP
D4 FSP D5 D6 D7 D8 D9 DA
D4 FSP D5 D6 D7 D8 D9 DA
01
06
10
15
01
06
10
15
Figure 4-12 Fanout plugging rules
Adapter ID number assignment

Unlike channels installed in an I/O cage, which are identified by a PCHID number related to their physical location, IFB fanouts and ports are identified by an adapter ID (AID), initially dependent on their physical locations. This AID must be used to assign a CHPID to the fanout in the IOCDS definition. The CHPID assignment is done by associating the CHPID to an AID port. Table 4-6 illustrates the AID assignment for each fanout slot relative to the book location on a new build system.
Table 4-6 AID number assignment Book First Second Third Fourth Slot 06 15 10 01 Fanout slot D1, D2, D5-DA D1, D2, D5-DA D1, D2, D5-DA D1, D2, D5-DA AIDs 08, 09, 0A-0F 18, 19, 1A-1F 10, 11, 12-17 00, 01, 02-07
Fanout slots
The fanout slots are numbered D1 to DA top to bottom, as shown in Table 4-7 on page 143. All fanout locations and their AIDs for all four books are shown in the table for reference only. Fanouts in locations D1 and D2 are not available on all models. Slots D3 and D4 will never have a fanout installed (dedicated for FSPs).
142
8049ch04.fm
Attention: Slots D1 and D2 are not used in a 4-book system, and only partially in a 3-book system.
Table 4-7 Fanout AID numbers Fanout location D1 D2 D3 D4 D5 D6 D7 D8 D9 DA Fourth book 00 01 02 03 04 05 06 07 First book 08 09 0A 0B 0C 0D 0E 0F Third book 10 11 12 13 14 15 16 17 Second book 18 19 1A 1B 1C 1D 1E 1F
Important: The AID numbers in Table 4-7 are valid only for a new build system or for new books added. If a fanout is moved, the AID follows the fanout to its new physical location. The AID assigned to a fanout is found in the PCHID REPORT provided for each new server or for MES upgrade on existing servers. Example 4-1 shows part of a report, named PCHID REPORT, for a model M32. In this example, one fanout is installed in the first book (location 06) and one fanout is installed in the second book (location 15), both in location D5. The assigned AID for the fanout in the first book is 0A; the AID assigned to the fanout in the second book is 1A.
Example 4-1 AID assignment in PCHID report
CHPIDSTART 12345675 PCHID Machine: xxxx-H43 SNXXXXXXX - - - - - - - - - - - - - - - - - Source Cage Slot F/C 06/D5 A25B D506 0163 15/D5 A25B D515 0163
REPORT
Jun xx,2012
- - - - - - - - - - - - - - - - - - - - PCHID/Ports or AID Comment AID=0A AID=1A
4.7.8 Fanout summary

Fanout features supported by the zEC12 server are shown in Table 4-8 on page 144. The table provides the feature type, feature code, and information about the link supported by the fanout feature.
143
8049ch04.fm
Table 4-8 Fanout summary Fanout feature HCA2-C Feature code 0162 Use Connect to I/O cage or I/O drawer Coupling link Coupling link Connect to PCIe I/O drawer Coupling link Coupling link Cable type Copper Connector type n/a Max. distance 3.5 m Link data rate 6 GBps
HCA2-O (12xIFB) HCA2-O LR (1xIFB) PCIe fanout
0163
50 m MM OM3 (2000 MHz-km) 9 m SM Copper
MPO
150 m
6 GBps
0168 0169
LC Duplex n/a
10 kma 3m
5.0 Gbps 2.5 Gbpsb 8 GBps
HCA3-O (12xIFB) HCA3-O LR (1xIFB)
0171
50 m MM OM3 (2000 MHz-km) 9 m SM
MPO
150 m
6 GBpsc
0170
LC Duplex
10 kma
5.0 Gbps 2.5 Gbpsb
a. Up to 100 km with repeaters (System z qualified DWDM) b. Auto-negotiated, depending on DWDM equipment c. When using the 12x IFB3 protocol, synchronous service times are 40% faster than when using the 12x IFB protocol.
4.8 I/O feature cards

I/O cards have ports to connect the zEC12 to external devices, networks, or other servers. I/O cards are plugged into the I/O cage, I/O drawer, and PCIe I/O drawer based on the configuration rules for the server. Different types of I/O cards are available, one for each channel or link type. I/O cards can be installed or replaced concurrently. In addition to I/O cards, Crypto Express cards can be installed in I/O drawers, I/O cages or PCIe drawers, and Flash Express cards can be installed in a PCIe drawer. These feature types occupy one or more I/O slots.
4.8.1 I/O feature card types ordering information

Table 4-9 lists the I/O features supported by zEC12 and the order information for them.
Table 4-9 I/O features and ordering information Channel feature FICON Express4 10KM LX FICON Express8 10KM LX FICON Express8S 10KM LX FICON Express4 SX FICON Express8 SX Feature code 3321 3325 0409 3322 3326 New build N N Y N N Carry forward Y Y N/A Y Y
144
8049ch04.fm
Channel feature FICON Express8S SX OSA-Express3 GbE LX OSA-Express4S GbE LX OSA-Express3 GbE SX OSA-Express4S GbE SX OSA-Express3 1000BASE-T Ethernet OSA-Express4S 1000BASE-T Ethernet OSA-Express3 10 GbE LR OSA-Express4S 10 GbE LR OSA-Express3 10 GbE SR OSA-Express4S 10 GbE SR ISC-3 ISC-3 up to 20 kma HCA2-O (12xIFB) HCA2-O LR (1xIFB) HCA3-O (12xIFB) HCA3-O LR (1xIFB) Crypto Express3 Crypto Express4S Flash Express
Feature code 0410 3362 0404 3363 0405 3367 0408 3370 0406 3371 0407 0217 (ISC-M) 0218 (ISC-D) RPQ 8P2197 (ISC-D) 0163 0168 0171 0170 0864 0865 0402
New build Y N Y N Y N Y N Y N Y N N N N Y Y N Y Y
Carry forward N/A Y Y Y Y Y N/A Y Y Y Y Y Y Y Y Y Y Y Y N/A
a. RPQ 8P2197 enables the ordering of a daughter card supporting 20 km unrepeated distance for 1 Gbps peer mode. RPQ 8P2262 is a requirement for that option, and other than the normal mode, the channel increment is two, meaning that both ports (FC 0219) at the card must be activated.
4.8.2 PCHID report

A Physical Channel ID (PCHID) reflects the physical location of a channel-type interface. A PCHID number is based on the I/O cage, I/O drawer, and PCIe I/O drawer location, the channel feature slot number, and the port number of the channel feature. A CHPID does not directly correspond to a hardware channel port, but is assigned to a PCHID in HCD or IOCP. A PCHID report is created for each new build server and for upgrades on existing servers. The report lists all I/O features installed, the physical slot location, and the assigned PCHID. Example 4-2 on page 146 shows a portion of a sample PCHID report. The AID numbering rules for InfiniBand coupling links are described in Adapter ID number assignment on page 142.
145
8049ch04.fm
Example 4-2 PCHID report

CHPIDSTART 12345675 PCHID REPORT Jun xx,2012 Machine: xxxx-H43 SNXXXXXXX - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Source Cage Slot F/C PCHID/Ports or AID Comment 06/D7 15/D5 06/DA/J01 06/D8/J01 06/D8/J01 06/D9/J02 06/DA/J02 15/DA/J02 15/DA/J02 06/DA/J02 A25B A25B Z15B Z15B Z15B Z15B Z22B Z22B Z22B Z22B D706 0170 D515 0171 04 0405 AID=0C AID=1A 130/J00J01 131/J02J03 210/J00 211/J01 218/J00 219/J01 520/D1 521/D2 580/P00 590/J00 5A0/J00J01 5A1/J02J03 5B0/J00J01 5B1/J02J03
D120 0218 D220 0218 04 02 03 04 05 0409 0865 0407 0408 0408
The following list explains the content of the sample PCHID REPORT: Feature code 0170 (HCA3-O LR (1xIFB)) is installed in the first book (cage A25B, slot 06) location D7 and has AID 0C assigned. Feature code 0171 (HCA3-O (12xIFB)) is installed in the second book (cage A25B, slot 15) location D5 and has AID 1A assigned. Feature code 0405 (OSA-Express4S GbE SX) is installed in cage Z15B slot 4 and has PCHIDs 130 and 131 assigned. PCHID 130 is shared by port 00 and 01, PCHID 131 is shared by port 02 and 03. Feature code 0218 (ISC-3) is installed in cage Z15B slot 20 and has PCHID 210 and 211 assigned to the two ports on the upper daughter card, and PCHID 218 and 219 to the two ports on the lower daughter card. Feature code 0409 (FICON Express8S LX 10 km) is installed in drawer Z15B slot 4 and has PCHIDs 520 and 521 assigned. Feature code 0865 (Crypto Express4S) is installed in drawer Z22B slot 2 and has PCHIDs 580 assigned. Feature code 0407 (OSA-Express4S 10 GbE SR) is installed in drawer Z22B slot 3 and has PCHIDs 590 assigned. Feature code 0408 (OSA-Express4S 1000BASE-T) is installed in drawer Z22B slot 4 and has PCHIDs 5A0 and 5A1 assigned. PCHID 5A0 is shared by port 00 and 01, PCHID 5A1 is shared by port 02 and 03. The pre-assigned PCHID number of each I/O port relates directly to its physical location (jack location in a specific slot).
4.9 Connectivity
I/O channels are part of the channel subsystem (CSS). They provide connectivity for data exchange between servers, or between servers and external control units (CU) and devices, or networks. Communication between servers is implemented by using InterSystem Channel-3 (ISC-3), coupling using InfiniBand (IFB), or channel-to-channel connections (CTC).
146
8049ch04.fm
Communication to local area networks (LANs) is provided by the OSA-Express3 and OSA-Express4S features. Connectivity to I/O subsystems to exchange data is provided by FICON channels.
4.9.1 I/O feature support and configuration rules

Table 4-10 lists the I/O features supported. The table shows number of ports per card, port increments, and the maximum number of feature cards and the maximum of channels for each feature type. Also, the CHPID definitions used in the IOCDS are listed. each feature type. Also, the CHPID definitions used in the IOCDS are listed.
Table 4-10 zEC12 Supported I/O features I/O feature Number of Ports per card FICON Express4a LX/SX FICON Express8 LX/SX FICON Express8S LX/SX OSA- Express3 10 GbE LR/SR OSA-Express3 GbE LX/SX OSA-Express3 1000BASE-T OSA-Express4S GbE LX/SX OSA- Express4S 10 GbE LR/SR OSA-Express4S 1000BASE-T ISC-3 2 Gbps (10 km) ISC-3 1 Gbps (20 km) HCA2-O for 12x IFB HCA3-O for 12x IFB and 12x IFB3 HCA2-O LR for 1x IFB HCA3-O LR for 1x IFB 4 4 2 2 4 4 Port increments 4 4 2 2 4 4 Max. number of Ports 176 176 320 48 96 96 I/O slots 44 44 160 24b 24b 24b Yes Yes Yes Yes Yes Yes PCHID CHPID definition
FC, FCP FC, FCP FC, FCP OSD, OSX OSD, OSN OSE, OSD, OSC, OSN, OSM OSD OSD, OSX OSE, OSD, OSC, OSN, OSM CFP CFP CIB CIB CIB CIB
2 1 2
2 1 2
96 48 96
48b 48b 48b
Yes Yes YES
2 / ISC-D 2 / ISC-D 2 2 2 4
1 2 2 2 2 4
48 48 32 32 32 64
12 12 16 16 16 16
Yes Yes No No No No
a. FICON Express4 4km LX feature (FC3324) is not supported on zEC12
147
8049ch04.fm
b. Each OSA-Express3 feature installed in an I/O cage/drawer reduces by two (2) the number of OSA-Express4S features allowed
At least one I/O feature (FICON) or one coupling link feature (IFB or ISC-3) must be present in the minimum configuration. A maximum of 256 channels is configurable per channel subsystem and per operating system image. The following channels can be shared and spanned: FICON channels defined as FC or FCP OSA-Express3 defined as OSC, OSD, OSE, OSM, OSN or OSX OSA-Express4S defined as OSC, OSD, OSE, OSM, OSN or OSX Coupling links defined as CFP, ICP, or CIB HiperSockets defined as IQD Each Crypto Express feature occupies an I/O slot but does not have a CHPID type, but logical partitions in all CSSs have access to the features. Each Crypto Express adapter can be defined to up to 16 logical partitions. Each Flash Express feature occupies two I/O slots but does not have a CHPID type, but logical partitions in all CSSs have access to the features. The Flash Express feature can be defined to up to 60 logical partitions.
I/O feature cables and connectors

Cables: All fiber optic cables, cable planning, labeling, and installation are customer responsibilities for new zEC12 installations and upgrades. Fiber optic conversion kits and mode conditioning patch (MCP) cables are not orderable as features on zEC12 servers. All other cables have to be sourced separately. IBM Facilities Cabling Services - fiber transport system offers a total cable solution service to help with cable ordering requirements, and is highly desirable. These services consider the requirements for all of the protocols and media types supported (for example, FICON, Coupling Links, and OSA), whether the focus is the data center, the storage area network (SAN), local area network (LAN), or the end-to-end enterprise. The Enterprise Fiber Cabling Services make use of a proven modular cabling system, the Fiber Transport System (FTS), which includes trunk cables, zone cabinets, and panels for servers, directors, and storage devices. FTS supports Fiber Quick Connect (FQC), a fiber harness integrated in the frame of a zEC12 for quick connection, which is offered as a feature on zEC12 servers for connection to FICON LX channels. Whether you choose a packaged service or a custom service, high quality components are used to facilitate moves, additions, and changes in the enterprise to prevent having to extend the maintenance window. Table 4-11 lists the required connector and cable type for each I/O feature on the zEC12.
Table 4-11 I/O features connector and cable types Feature code 0163 0168 Feature name InfiniBand coupling (IFB) InfiniBand coupling (IFB LR) Connector type MPO LC Duplex Cable type 50 m MMa OM3 (2000 MHz-km) 9 m SMb
148
8049ch04.fm
Feature code 0219 3321 3322 3325 3326 0409 0410 3370 3371 3362 3363 3367 0404 0405 0406 0407 0408
Feature name ISC-3 FICON Express4 LX 10 km FICON Express4 SX FICON Express8 LX 10 km FICON Express8 SX FICON Express8S LX 10 km FICON Express8S SX OSA-Express3 10 GbE LR OSA-Express3 10 GbE SR OSA-Express3 GbE LX OSA-Express3 GbE SX OSA-Express3 1000BASE-T OSA-Express4S GbE LX OSA-Express4S GbE SX OSA-Express4S 10 GbE LR OSA-Express4S 10 GbE SR OSA-Express4S 1000BASE-T
Connector type LC Duplex LC Duplex LC Duplex LC Duplex LC Duplex LC Duplex LC Duplex LC Duplex LC Duplex LC Duplex LC Duplex RJ-45 LC Duplex LC Duplex LC Duplex LC Duplex RJ-45
Cable type 9 m SM 9 m SM 50, 62.5 m MM 9 m SM 50, 62.5 m MM 9 m SM 50, 62.5 m MM 9 m SM 50, 62.5 m MM 9 m SM 50, 62.5 m MM Category 5 UTPc 9 m SM 50, 62.5 m MM 9 m SM 50, 62.5 m MM Category 5 UTP
a. MM is multimode fiber. b. SM is single mode fiber. c. UTP is unshielded twisted pair. Consider using Category 6 UTP for 1000 Mbps connections.
4.9.2 FICON channels

The FICON Express8S, FICON Express8 and FICON Express42 features conform to the Fibre Connection (FICON) architecture, the High Performance FICON on System z (zHPF) architecture, and the Fibre Channel Protocol (FCP) architecture, providing connectivity between any combination of servers, directors, switches, and devices (control units, disks, tapes, printers) in a Storage Area Network (SAN). Attention: FICON Express and FICON Express2 features installed in previous systems are not supported on a zEC12 and cannot be carried forward on an upgrade. Each FICON Express8 or FICON Express4 feature occupies one I/O slot in the I/O cage or I/O drawer. Each feature has four ports, each supporting an LC Duplex connector, with one PCHID and one CHPID associated with each port. Each FICON Express8S feature occupies one I/O slot in the PCIe I/O drawer. Each feature has two ports, each supporting an LC Duplex connector, with one PCHID and one CHPID associated with each port.
FICON Express4 4 km LX (FC3324) is not supported on zEC12
149
8049ch04.fm
All FICON Express8S, FICON Express8 and FICON Express4 features use small form-factor pluggable (SFP) optics that allow for concurrent repair or replacement for each SFP. The data flow on the unaffected channels on the same feature can continue. A problem with one FICON port no longer requires replacement of a complete feature. All FICON Express8S, FICON Express8 and FICON Express4 features also support cascading (the connection of two FICON Directors in succession) to minimize the number of cross-site connections and help reduce implementation costs for disaster recovery applications, GDPS, and remote copy. Each FICON Express8S, FICON Express8 and FICON Express4 channel can be defined independently, for connectivity to servers, switches, directors, disks, tapes, and printers as: CHPID type FC - FICON, High Performance FICON for System z (zHPF), and FICON Channel-to-Channel (FCTC). FICON, FCTC, and zHPF protocols are supported simultaneously. CHPID type FCP - Fibre Channel Protocol which supports attachment to SCSI devices directly or through Fibre Channel switches or directors. FICON channels (CHPID type FC or FCP) can be shared among logical partitions and can be defined as spanned. All ports on a FICON feature must be of the same type, either LX or SX. The features are connected to a FICON-capable control unit, either point-to-point or switched point-to-point, through a Fibre Channel switch.
FICON Express8S
The FICON Express8S feature resides exclusively in the PCIe I/O drawer. Each of the two independent ports is capable of 2 gigabits per second (Gbps), 4 Gbps, or 8 Gbps depending upon the capability of the attached switch or device. The link speed is auto-negotiated, point-to-point, and is transparent to users and applications. The two types of FICON Express8S optical transceivers supported are the long wavelength (LX) and the short wavelength (SX): FICON Express8S 10km LX feature FC 0409, with two ports per feature, supporting LC Duplex connectors FICON Express8S SX feature FC 0410, with two ports per feature, supporting LC Duplex connectors Each port of the FICON Express8S 10 km LX feature uses a 1300 nanometer (nm) optical transceiver, supports an unrepeated distance of 10 km using 9 m single-mode fiber. Each port of the FICON Express8S SX feature uses an 850 nanometer (nm) optical transceiver. supports varying distances depending on the fiber used (50 or 62.5 m multimode fiber). Auto-negotiation: FICON Express8S features do not support auto-negotiation to a data link rate of 1 Gbps. Statement of direction: FICON Express8S feature will be the last FICON feature supporting auto-negotiation to a data link rate of 2 Gbps.
FICON Express8
The FICON Express8S feature resides in an I/O cage or I/O drawer. Each of the four independent ports is capable of 2 gigabits per second (Gbps), 4 Gbps, or 8 Gbps depending
150
8049ch04.fm
upon the capability of the attached switch or device. The link speed is auto-negotiated, point-to-point, and is transparent to users and applications. The two types of FICON Express8 optical transceivers supported are the long wavelength (LX) and the short wavelength (SX): FICON Express8 10km LX feature FC 3325, with four ports per feature, supporting LC Duplex connectors FICON Express8 SX feature FC 3326, with four ports per feature, supporting LC Duplex connectors Each port of FICON Express8 10 km LX feature uses a 1300 nanometer (nm) fiber bandwidth transceiver, supports an unrepeated distance of 10 km using 9 m single-mode fiber. Each port of FICON Express8 SX feature uses an 850 nanometer (nm) optical transceiver, supports varying distances depending on the fiber used (50 or 62.5 m multimode fiber). Auto-negotiation: FICON Express8 features do not support auto-negotiation to a data link rate of 1 Gbps.
FICON Express4
The FICON Express4 feature resides in an I/O cage or I/O drawer. Each of the four independent ports is capable of 1 gigabits per second (Gbps), 2Gbps, or 4 Gbps depending upon the capability of the attached switch or device. The link speed is auto-negotiated, point-to-point, and is transparent to users and applications. Statement of Direction: The IBM zEnterprise EC12 is planned to be the last high-end System z server to offer support of the FICON Express4 features (#3321, #3322). FICON Express4 will not be supported on future high-end System z servers as carry forward on an upgrade. Enterprises should continue migrating from the FICON Express4 features to the FICON Express8S features (#0409, #0410). The two types of FICON Express4 optical transceivers supported are the two long wavelength (LX) and one short wavelength (SX): FICON Express4 10km LX feature FC 3321, with four ports per feature, supporting LC Duplex connectors FICON Express4 SX feature FC 3322, with four ports per feature, supporting LC Duplex connectors FICON Express4 4 km LX (FC3324) is not supported on zEC12 The FICON Express4 LX features use 1300 nanometer (nm) optical transceivers. One supports an unrepeated distance of 10 km, and the other an unrepeated distance of 4 km, using 9 m single-mode fiber. Use of MCP cables limits the link speed to 1 Gbps and the unrepeated distance to 550 meters. The FICON Express4 SX feature use 850 nanometer (nm) optical transceivers. supports varying distances depending on the fiber used (50 or 62.5 m multimode fiber). Link speed: FICON Express4 is the last FICON family able to negotiate link speed down to 1 Gbps.
151
8049ch04.fm
FICON feature summary

Table 4-12 shows the FICON card feature codes, cable type, maximum unrepeated distance, and the link data rate on a zEC12. All FICON features use LC Duplex connectors. For long wave FICON features that can utilize a data rate of 1 Gbps, mode conditioning patch (MCP) cables (50 or 62.5 MM) can be used.The maximum distance for this connections is reduced to 550 m at a link data rate of 1Gbps. Details for each feature are in the following table.
Table 4-12 EC12 channel feature support Channel feature Feature Bit rate codes 3321 3322 1, 2, or 4 Gbps 4 Gbps Cable type Maximum unrepeated distancea 10 km/20 km 70 m (200) 150 m (500) 380 m (2000) 150 m (200) 300 m (500) 500 m (2000) 300 m (200) 500 m (500) 860 m (2000) 10 km 21 m (200) 50 m (500) 150 m (2000) 70 m (200) 150 m (500) 380 m (2000) 150 m (200) 300 m (500) 500 m (2000) 10 km 21 m (200) 50 m (500) 150 m (2000) 70 m (200) 150 m (500) 380 m (2000) 150 m (200) 300 m (500) 500 m (2000)
FICON Express4 10KM LX FICON Express4 SX
SM 9 m MM 62.5 m MM 50 m MM 62.5 m MM 50 m MM 62.5 m MM 50 m SM 9 m MM 62.5 m MM 50 m MM 62.5 m MM 50 m MM 62.5 m MM 50 m SM 9 m MM 62.5 m MM 50 m MM 62.5 m MM 50 m MM 62.5 m MM 50 m
2 Gbps
1 Gbps
FICON Express8 10KM LX FICON Express8 SX
3325 3326
2, 4, or 8 Gbps 8 Gbps
4 Gbps
2 Gbps
FICON Express8S 10KM LX FICON Express8S SX
0409 0410
2, 4, or 8 Gbps 8 Gbps
4 Gbps
2 Gbps
a. Minimum fiber bandwidths in MHz/km for multimode fiber optic links are included in parentheses were applicable.
152
8049ch04.fm
4.9.3 OSA-Express4S features

This section will discuss all the characteristics of all OSA-Express4S feature supported on zEC12 servers. The OSA-Express4S feature resides exclusively in the PCIe I/O drawer. The following OSA-Express4S features can be installed on zEC12 servers: OSA-Express4S Gigabit Ethernet LX, feature code 0404 OSA-Express4S Gigabit Ethernet SX, feature code 0405 OSA-Express4S 10 Gigabit Ethernet LR, feature code 0406 OSA-Express4S 10 Gigabit Ethernet SR, feature code 0407 OSA-Express4S 1000BASE-T Ethernet, feature code 0408 Table 4-13 lists the characteristics of the OSA-Express4S features.
Table 4-13 OSA-Express4S features I/O feature Feature code Number of ports per feature 1 1 2 2 2 Port increment Maximum number of ports (CHPIDs) 48 48 96b 96b 96b Maximum number of features 48a 48b 48b 48b 48b CHPID type
OSA-Express4S 10 GbE LR OSA-Express4S 10 GbE SR OSA-Express4S GbE LX OSA-Express4S GbE SX OSA-Express4S 1000BASE-T
0406 0407 0404 0405 0408
1 1 2 2 2
OSD, OSX OSD, OSX OSD OSD OSC, OSD, OSE, OSM, OSN
a. Each OSA-Express3 feature installed in an I/O cage/drawer reduces by two (2) the number of OSA-Express4S features allowed b. Both ports on each feature share one PCHID/CHPID
OSA-Express4S Gigabit Ethernet LX (FC 0404)

The OSA-Express4S Gigabit Ethernet (GbE) long wavelength (LX) feature has one PCIe adapter and two ports. The two ports share a channel path identifier (CHPID type OSD exclusively). The ports support attachment to a one Gigabit per second (Gbps) Ethernet Local Area Network (LAN). Each port can be defined as a spanned channel and can be shared among logical partitions and across logical channel subsystems. The OSA-Express4S GbE LX feature supports use of an LC Duplex connector. Ensure the attaching or downstream device has a long wavelength (LX) transceiver. The sending and receiving transceivers must be the same (LX to LX). A 9 m single mode fiber optic cable terminated with an LC Duplex connector is required for connecting each port on this feature to the selected device. If multimode fiber optic cables are
153
8049ch04.fm
being reused, a pair of Mode Conditioning Patch cables are required, one for each end of the link.
OSA-Express4S Gigabit Ethernet SX (FC 0405)

The OSA-Express4S Gigabit Ethernet (GbE) short wavelength (SX) feature has one PCIe adapter and two ports. The two ports share a channel path identifier (CHPID type OSD exclusively). The ports support attachment to a one Gigabit per second (Gbps) Ethernet Local Area Network (LAN). Each port can be defined as a spanned channel and can be shared among logical partitions and across logical channel subsystems. The OSA-Express4S GbE SX feature supports use of an LC Duplex connector. Ensure the attaching or downstream device has a short wavelength (SX) transceiver. The sending and receiving transceivers must be the same (SX to SX). A 50 or a 62.5 m multimode fiber optic cable terminated with an LC Duplex connector is required for connecting each port on this feature to the selected device.
OSA-Express4S 10 Gigabit Ethernet LR (FC 0406)

The OSA-Express4S 10 Gigabit Ethernet (GbE) long reach (LR) feature has one PCIe adapter and one port per feature. The port supports channel path identifier (CHPID) types OSD and OSX. When defined as CHPID type OSX, the 10 GbE port provides connectivity and access control to the intraensemble data network (IEDN) from zEC12 to zEnterprise BladeCenter Extension (zBX) The 10 GbE feature is designed to support attachment to a single mode fiber 10 Gigabits per second (Gbps) Ethernet Local Area Network (LAN) or Ethernet switch capable of 10 Gbps. The port can be defined as a spanned channel and can be shared among logical partitions within and across logical channel subsystems. The OSA-Express4S 10 GbE LR feature supports use of an industry standard small form factor LC Duplex connector. Ensure the attaching or downstream device has a long reach (LR) transceiver. The sending and receiving transceivers must be the same (LR to LR, which can also be referred to as LW or LX). A 9 m single mode fiber optic cable terminated with an LC Duplex connector is required for connecting this feature to the selected device.
OSA-Express4S 10 Gigabit Ethernet SR (FC 0407)

The OSA-Express4S 10 Gigabit Ethernet (GbE) Short Reach (SR) feature has one PCIe adapter and one port per feature. The port supports channel path identifier (CHPID) types OSD and OSX. When defined as CHPID type OSX, the 10 GbE port provides connectivity and access control to the intraensemble data network (IEDN) from zEC12 to zEnterprise BladeCenter Extension (zBX). The 10 GbE feature is designed to support attachment to a multimode fiber 10 Gigabits per second (Gbps) Ethernet Local Area Network (LAN) or Ethernet switch capable of 10 Gbps. The port can be defined as a spanned channel and can be shared among logical partitions within and across logical channel subsystems. The OSA-Express4S 10 GbE SR feature supports use of an industry standard small form factor LC Duplex connector. Ensure the attaching or downstream device has a Short Reach (SR) transceiver. The sending and receiving transceivers must be the same (SR to SR). A 50 or a 62.5 m multimode fiber optic cable terminated with an LC Duplex connector is required for connecting each port on this feature to the selected device.
OSA-Express4S 1000BASE-T Ethernet feature (FC 0408)

Feature code 0408 occupies one slot in the PCIe drawer. It has two ports that connect to a 1000 Mbps (1 Gbps), 100 Mbps, or 10 Mbps Ethernet LAN. Each port has an RJ-45 154
8049ch04.fm
receptacle for cabling to an Ethernet switch. The RJ-45 receptacle is required to be attached using EIA/TIA Category 5 or Category 6 unshielded twisted pair (UTP) cable with a maximum length of 100 meters. The OSA-Express4S 1000BASE-T Ethernet feature supports auto-negotiation when attached to an Ethernet router or switch. If you allow the LAN speed and duplex mode to default to auto-negotiation, the OSA-Express port and the attached router or switch auto-negotiate the LAN speed and duplex mode settings between them and connect at the highest common performance speed and duplex mode of interoperation. If the attached Ethernet router or switch does not support auto-negotiation, the OSA-Express port examines the signal it is receiving and connects at the speed and duplex mode of the device at the other end of the cable. The OSA-Express4S 1000BASE-T Ethernet feature can be configured as CHPID type OSC, OSD, OSE, OSN or OSM. Non-QDIO operation mode requires CHPID type OSE. When defined as CHPID type OSM, the port provides connectivity to the intranode management network (INMN). The following settings are supported on the OSA-Express3 1000BASE-T Ethernet feature port: Auto-negotiate 10 Mbps half-duplex or full-duplex 100 Mbps half-duplex or full-duplex 1000 Mbps full-duplex If you are not using auto-negotiate, the OSA-Express port will attempt to join the LAN at the specified speed and duplex mode. If this does not match the speed and duplex mode of the signal on the cable, the OSA-Express port will not connect. Statement of Direction: The OSA-Express4S 1000BASE-T Ethernet feature is planned to be the last copper Ethernet feature to support half-duplex operation and a 10 Mbps link data rate. The zEnterprise EC12 servers are planned to be the last IBM System z servers to support half-duplex operation and a 10 Mbps link data rate for copper Ethernet environments. Any future 1000BASE-T Ethernet feature will support full-duplex operation and auto-negotiation to 100 or 1000 Mbps exclusively.
4.9.4 OSA-Express3 features

This section discusses the connectivity options offered by the OSA-Express3 features. Statement of Direction: The IBM zEnterprise EC12 is planned to be the last high-end System z server to offer support of the Open System Adapter-Express3 (OSA-Express3 #3362, #3363, #3367, #3370, #3371) family of features. OSA-Express3 will not be supported on future high-end System z servers as carry forward on an upgrade. Enterprises should continue migrating from the OSA-Express3 features to the OSA-Express4S features (#0404, #0405, #0406, #0407, #0408). The OSA-Express3 feature resides exclusively in an I/O cage or I/O drawer. The following OSA-Express3 features can be installed on zEC12 servers: The following OSA-Express3 features can be installed on zEC12 servers: OSA-Express3 10 Gigabit Ethernet (GbE) Long Range (LR), feature code 3370
155
8049ch04.fm
OSA-Express3 10 Gigabit Ethernet (GbE) Short Reach (SR), feature code 3371 OSA-Express3 Gigabit Ethernet (GbE) Long wavelength (LX), feature code 3362 OSA-Express3 Gigabit Ethernet (GbE) Short wavelength (SX), feature code 3363 OSA-Express3 1000BASE-T Ethernet, feature code 3367 Table 4-14 lists the characteristics of OSA-Express3 features.
Table 4-14 OSA-Express3 features I/O feature Feature code Number of ports per feature 2 2 4 4 4 Port increment Maximum number of ports 48 48 96 96 96 Maximum number of features 24 24 24 24 24 CHPID type
OSA-Express3 10 GbE LR OSA-Express3 10 GbE SR OSA-Express3 GbE LX OSA-Express3 GbE SX OSA-Express3 1000BASE-T
3370 3371 3362 3363 3367
2 2 4 4 4
OSD,OSX OSD,OSX OSD, OSN OSD, OSN OSC,OSD, OSE,OSN, OSM
OSA-Express3 10 GbE LR (FC 3370)

Note: Each OSA-Express3 feature installed in an I/O cage/drawer reduces by two (2) the number of OSA-Express4S features allowed The OSA-Express3 10 GbE LR feature occupies one slot in the I/O cage or I/O drawer and has two ports that connect to a 10 Gbps Ethernet LAN through a 9 m single mode fiber optic cable terminated with an LC Duplex connector. Each port on the card has a PCHID assigned. The feature supports an unrepeated maximum distance of 10 km. Compared to the OSA-Express2 10 GbE LR feature, the OSA-Express3 10 GbE LR feature has double port density (two ports for each feature) and improved performance for standard and jumbo frames. The OSA-Express3 10 GbE LR feature does not support auto-negotiation to any other speed and runs in full-duplex mode only. It supports 64B/66B encoding (whereas GbE supports 8B/10B encoding). Therefore, auto-negotiation to any other speed is not possible. The OSA-Express3 10 GbE LR feature has two CHPIDs, with each CHPID having one port, and supports CHPID types OSD (QDIO mode) and OSX. CHPID type OSD is supported by z/OS, z/VM, z/VSE, TPF, and Linux on System z to provide customer managed external network connections. CHPID type OSX is dedicated for connecting the zEC12 to an intraensemble data network (IEDN), providing a private data exchange path across ensemble nodes.
156
8049ch04.fm
OSA-Express3 10 GbE SR (FC 3371)

The OSA-Express3 10 GbE SR feature (FC 3371) occupies one slot in the I/O cage or I/O drawer and has two CHPIDs, with each CHPID having one port. External connection to a 10 Gbps Ethernet LAN is done through a 62.5 m or 50 m multimode fiber optic cable terminated with an LC Duplex connector. The maximum supported unrepeated distance is 33 meters on a 62.5 m multimode (200 MHz) fiber optic cable, 82 meters on a 50 m multi mode (500 MHz) fiber optic cable, and 300 meters on a 50 m multimode (2000 MHz) fiber optic cable. The OSA-Express3 10 GbE SR feature does not support auto-negotiation to any other speed and runs in full-duplex mode only. OSA-Express3 10 GbE SR supports 64B/66B encoding, whereas GbE supports 8B/10 encoding, making auto-negotiation to any other speed impossible. The OSA-Express3 10 GbE SR feature supports CHPID types OSD (QDIO mode) and OSX. CHPID type OSD is supported by z/OS, z/VM, z/VSE, TPF, and Linux on System z to provide customer managed external network connections. CHPID type OSX is dedicated for connecting the zEC12 to an intraensemble data network (IEDN), providing a private data exchange path across ensemble nodes.
OSA-Express3 GbE LX (FC 3362)

Feature code 3362 occupies one slot in the I/O cage or I/O drawer. It has four ports that connect to a 1 Gbps Ethernet LAN through a 9 m single mode fiber optic cable terminated with an LC Duplex connector, supporting an unrepeated maximum distance of 5 km. Multimode (62.5 or 50 m) fiber optic cable can be used with this features. MCP: The use of these multimode cable types requires a mode conditioning patch (MCP) cable at each end of the fiber optic link. Use of the single mode to multimode MCP cables reduces the supported distance of the link to a maximum of 550 meters. The OSA-Express3 GbE LX feature does not support auto-negotiation to any other speed and runs in full-duplex mode only. The OSA-Express3 GbE LX feature has two CHPIDs, with each CHPID (OSD or OSN) having two ports for a total of four ports per feature. Exploitation of all four ports requires operating system support.
OSA-Express3 GbE SX (FC 3363)

Feature code 3363 occupies one slot in the I/O cage or I/O drawer. It has four ports that connect to a 1 Gbps Ethernet LAN through a 50 m or 62.5 m multimode fiber optic cable terminated with an LC Duplex connector over an unrepeated distance of 550 meters (for 50 m fiber) or 220 meters (for 62.5 m fiber). The OSA-Express3 GbE SX feature does not support auto-negotiation to any other speed and runs in full-duplex mode only. The OSA-Express3 GbE SX feature has two CHPIDs (OSD or OSN) with each CHPID having two ports for a total of four ports per feature. Exploitation of all four ports requires operating system support.
157
8049ch04.fm
OSA-Express3 1000BASE-T Ethernet feature (FC 3367)

Feature code 3367 occupies one slot in the I/O cage or I/O drawer. It has four ports that connect to a 1000 Mbps (1 Gbps), 100 Mbps, or 10 Mbps Ethernet LAN. Each port has an RJ-45 receptacle for cabling to an Ethernet switch. The RJ-45 receptacle is required to be attached using EIA/TIA Category 5 or Category 6 unshielded twisted pair (UTP) cable with a maximum length of 100 meters. The OSA-Express3 1000BASE-T Ethernet feature supports auto-negotiation when attached to an Ethernet router or switch. If you allow the LAN speed and duplex mode to default to auto-negotiation, the OSA-Express port and the attached router or switch auto-negotiate the LAN speed and duplex mode settings between them and connect at the highest common performance speed and duplex mode of interoperation. If the attached Ethernet router or switch does not support auto-negotiation, the OSA-Express port examines the signal it is receiving and connects at the speed and duplex mode of the device at the other end of the cable. The OSA-Express3 1000BASE-T Ethernet feature can be configured as CHPID type OSC, OSD, OSE, OSN or OSM. Non-QDIO operation mode requires CHPID type OSE. When defined as CHPID type OSM, the port provides connectivity to the intranode management network (INMN). The following settings are supported on the OSA-Express3 1000BASE-T Ethernet feature port: Auto-negotiate 10 Mbps half-duplex or full-duplex 100 Mbps half-duplex or full-duplex 1000 Mbps full-duplex If you are not using auto-negotiate, the OSA-Express port will attempt to join the LAN at the specified speed and duplex mode. If this does not match the speed and duplex mode of the signal on the cable, the OSA-Express port will not connect.
4.9.5 OSA-Express for ensemble connectivity

The following OSA-Express features are used to connect the zEC12 to its attached IBM zEnterprise BladeCenter Extension (zBX) Model 003 and other ensemble nodes: OSA-Express3 1000BASE-T Ethernet, feature code 3367 OSA-Express3 10 Gigabit Ethernet (GbE) Long Range (LR), feature code 3370 OSA-Express3 10 Gigabit Ethernet (GbE) Short Reach (SR), feature code 3371 OSA-Express4S 1000BASE-T Ethernet, feature code 0408 OSA-Express4S 10 Gigabit Ethernet (GbE) Long Range (LR), feature code 0406 OSA-Express4S 10 Gigabit Ethernet (GbE) Short Reach (SR), feature code 0407
Intraensemble data network (IEDN)

The IEDN is a private and secure 10 Gbps Ethernet network that connects all elements of an ensemble and is access-controlled using integrated virtual LAN (VLAN) provisioning. No customer-managed switches or routers are required. The IEDN is managed by a primary HMC. IEDN requires two OSA-Express3 10 GbE ports (one port from two OSA-Express3 10 GbE features) or preferably two OSA-Express4S 10 GbE configured as CHPID type OSX. The 158
8049ch04.fm
connection is from the zEC12 to the IEDN top of rack (TOR) switches on zBX Model 003. Or with a stand-alone zEC12 node (no-zBX) interconnect pairs of OSX ports through LC DUPLEX directly connected cables and not wrap cables as has previously been specified. For detailed information about OSA-Express3 an OSA Express4S in an ensemble network, see zBX connectivity on page 226
Intranode management network (INMN)

The INMN is a private and physically isolated 1000BASE-T Ethernet internal management network, operating at 1 Gbps. It connects all resources (zEC12 and zBX Model 003 components) of an ensemble node for management purposes. It is prewired, internally switched, configured, and managed with full redundancy for high availability. The INMN requires two ports (CHPID port 0 from two OSA-Express3 1000BASE-T or OSA-Express4S 1000BASE-T features, (CHPID port 1 is not used at all in this case) configured as CHPID type OSM. The connection is through port J07 of the bulk power hubs (BPHs) in the zEC12. The INMN top of rack (TOR) switches on zBX Model 003 also connect to the BPHs. For detailed information about OSA-Express3 an OSA Express4S in an ensemble network, see zBX connectivity on page 226
Ensemble HMC management functions

An HMC can manage multiple System z servers and can be located at a local or a remote site. If the zEC12 is defined as a member of an ensemble, a pair of HMCs (a primary and an alternate) are required and certain restrictions apply. The primary HMC is required to manage ensemble network connectivity, the INMN and the IEDN network. See 12.6, HMC in an ensemble on page 422 and 10.5, RAS capability for the HMC and SE on page 369 for a description and prerequisites.
4.9.6 HiperSockets
The HiperSockets function of IBM zEnterprise EC12 provides up to 32 high-speed virtual LAN attachments like IBM zEnterprise 196 and IBM zEnterprise 114 servers. Previous servers provided 16 attachments. HiperSockets can be customized to accommodate varying traffic sizes. Because HiperSockets does not use an external network, it can free up system and network resources, which can help eliminate attachment costs, and improve availability and performance. HiperSockets eliminates having to use I/O subsystem operations and having to traverse an external network connection to communicate between logical partitions in the same zEC12 server. HiperSockets offers significant value in server consolidation connecting many virtual servers, and can be used instead of certain coupling link configurations in a Parallel Sysplex. HiperSockets internal networks in the support two transport modes: Layer 2 (link layer) Layer 3 (network or IP layer) Traffic can be IPv4 or IPv6, or non-IP such as AppleTalk, DECnet, IPX, NetBIOS, or SNA. HiperSockets devices are protocol and Layer 3-independent. Each HiperSockets device (Layer 2 and Layer 3 mode) has its own MAC address designed to allow the use of applications that depend on the existence of Layer 2 addresses, such as DHCP servers and
159
8049ch04.fm
firewalls. Layer 2 support helps facilitate server consolidation, can reduce complexity, can simplify network configuration, and allows LAN administrators to maintain the mainframe network environment similarly as for non-mainframe environments. Packet forwarding decisions are based on Layer 2 information instead of Layer 3. The HiperSockets device can perform automatic MAC address generation to create uniqueness within and across logical partitions and servers. The use of Group MAC addresses for multicast is supported as well as broadcasts to all other Layer 2 devices on the same HiperSockets networks. Datagrams are delivered only between HiperSockets devices that use the same transport mode. A Layer 2 device cannot communicate directly to a Layer 3 device in another logical partition network. A HiperSockets device can filter inbound datagrams by VLAN identification, the destination MAC address, or both. Analogous to the Layer 3 functions, HiperSockets Layer 2 devices can be configured as primary or secondary connectors or multicast routers. This enables the creation of high-performance and high-availability link layer switches between the internal HiperSockets network and an external Ethernet network or to connect to the HiperSockets Layer 2 networks of different servers. HiperSockets Layer 2 in the zEC12 is supported by Linux on System z, and by z/VM for Linux guest exploitation. zEC12, and zEnterprise CPCs (z196 and z114 servers) support the HiperSockets Completion Queue function that is designed to allow HiperSockets to transfer data synchronously if possible and asynchronously if necessary. This feature combines ultra-low latency with more tolerance for traffic peaks. With the asynchronous support, during high volume situations, data can be temporarily held until the receiver has buffers available in its inbound queue. HiperSockets Completion Queue function is requires at a minimum: z/OS V1.13 Linux on System z distributions: Red Hat Enterprise Linux (RHEL) 6.2 SUSE Linux Enterprise Server (SLES) 11 SP2 z/VSE 5.1.1 The zEC12 and the zEnterprise servers provide the capability to integrate HiperSockets connectivity to the intraensemble data network (IEDN). This extends the reach of the HiperSockets network outside the CPC to the entire ensemble, appearing as a single, Layer 2. Because HiperSockets and IEDN are both internal System z networks, the combination allows System z virtual servers to use the optimal path for communications. In z/VM 6.2, the virtual switch function is enhanced to transparently bridge a guest virtual machine network connection on a HiperSockets LAN segment. This bridge allows a single HiperSockets guest virtual machine network connection to also directly communicate with the following: Other guest virtual machines on the virtual switch External network hosts through the virtual switch OSA UPLINK port. Statements of Direction: HiperSockets Completion Queue: IBM plans to support in a future deliverable of z/VM, transferring HiperSockets messages asynchronously, in addition to the current synchronous manner, on zEC12, z196 and z114.
160
8049ch04.fm
4.10 Parallel Sysplex connectivity

Coupling links are required in a Parallel Sysplex configuration to provide connectivity from the z/OS images to the coupling facility. A properly configured Parallel Sysplex provides a highly reliable, redundant, and robust System z technology solution to achieve near-continuous availability. A Parallel Sysplex comprises one or more z/OS operating system images coupled through one or more coupling facilities.
4.10.1 Coupling links

The type of coupling link that is used to connect a coupling facility (CF) to an operating system logical partition is important, because of the effect of the link performance on response times and coupling overheads. For configurations covering large distances, the time spent on the link can be the largest part of the response time. The following types of links are available to connect an operating system logical partition to a coupling facility: ISC-3: The InterSystem Channel-3 (ISC-3) type is available in peer mode only. ISC-3 links can be used to connect to zEC12, z196, z114, or System z10. They are optic fiber links that support a maximum distance of 10 km, 20 km with RPQ 8P2197, and 100 km with a System z qualified Dense Wave Division Multiplexer (DWDM). ISC-3s support 9 m single mode fiber optic cabling. The link data rate is 2 Gbps at distances up to 10 km, and 1 Gbps when RPQ 8P2197 is installed. Each port operates at 2 Gbps. Ports are ordered in increments of one. The maximum number of ISC-3 links per zEC12 is 48. ISC-3 supports transmission of Server Time Protocol (STP) messages. ISC-3: It is intended that the zEC12 is the last server supporting ISC-3 coupling links. Clients need to review the usage of their installed ISC-3 coupling links and where possible migrate to IFB (FC 0163 and FC 0171) or IFB LR (FC 0168 and FC 0170) coupling links. IFB: Parallel Sysplex using Infiniband (IFB) connects to a zEC12, z196, z114, or System z10 servers. 12x InfiniBand coupling links are fiber optic connections that support a maximum distance of up to 150 meters. IFB coupling links are defined as CHPID type CIB. IFB supports transmission of STP messages. zEC12 supports two types of 12x InfiniBand coupling links: FC 0171 HCA3-O (12xIFB) fanout FC 0163 HCA2-O (12xIFB) fanout). IFB LR: IFB LR (Long Reach) connects to zEC12, z196, z114, or System z10 servers. 1x InfiniBand coupling links are fiber optic connections that support a maximum unrepeated distance of up to 10 km and up to 100 km with a System z qualified Dense Wave Division Multiplexor (DWDM). IFB LR coupling links are defined as CHPID type CIB, supporting 7 or 32 subchannels per CHPID. IFB LR supports transmission of STP messages. zEC12 supports two types of 1x InfiniBand coupling links: FC 0170 HCA3-O LR (1xIFB) fanout. FC 0168 HCA2-O LR (1xIFB) fanout.
161
8049ch04.fm
IC: CHPIDs (type ICP) defined for internal coupling can connect a CF to a z/OS logical partition in the same zEC12. IC connections require two CHPIDs to be defined, which can only be defined in peer mode. The bandwidth is greater than 2 GBps. A maximum of 32 IC CHPIDs (16 connections) can be defined. Table 4-15 shows the coupling link options.
Table 4-15 Coupling link options Type Description Use Link rate 2 Gbps Distance zEC12H20 maximum 48 zEC12H43->HA1 maximum 48
ISC-3
InterSystem Channel-3
zEC12 to zEC12, z196 z114, z10 zEC12 to zEC12 z196, z114, z10 zEC12 to zEC12 z196, z114, z10 zEC12 to zEC12 z196, z114, z10 zEC12 to zEC12 z196, z114, z10 Internal communication
10 km unrepeated (6.2 miles) 100 km repeated
IFB
12x InfiniBand (HCA3-O)a
6 GBps
150 meters (492 feet)
16b
32
12x InfiniBand (HCA2-O)
6 GBps
150 meters (492 feet)
16b
32
IFB LR
1x IFB (HCA3-O LR)
2.5 Gbps 5.0 Gbps
10 km unrepeated (6.2.miles) 100 km repeated
32b
64
1x IFB (HCA2-O LR)
2.5 Gbps 5.0 Gbps
10 km unrepeated (6.2.miles) 100 km repeated
16b
32
IC
Internal coupling channel
Internal speeds
N/A
32
32
a. 12x IFB3 protocol: max 4 CHPIDs and connect to other HCA3-O port, else 12x IFB protocol. Auto configured when conditions are met for IFB3. 4.7.5, HCA3-O (12xIFB) fanout (FC0171) on page 140 b. Uses all available fanout slots. Allows no other I/O or coupling
The maximum for IFB links is 64. The maximum number of external coupling links combined (active ISC-3 links and IFB LR) cannot exceed 112 per server. There is a maximum of 128 coupling CHPIDs limitation, including ICP for IC, CIB (for IFB and IFB LR), and CFP (for ISC-3). The zEC12 supports various connectivity options depending on the connected zEC12, z196, z114, or System z10 server. Figure 4-13 on page 163 shows zEC12 coupling link support for zEC12, z196, z114, and System z10 servers.
162
8049ch04.fm
When defining IFB coupling links (CHPID type CIB), HCD now defaults to 7 subchannels. 32 Subchannels are only supported on HCA2-O LR (1xIFB) and HCA3-O LR (1xIFB) on zEC12 and when both sides of the connection use 32 subchannels. Otherwise you should change the default value from 7 to 32 subchannel on each CIB definition.
z10 EC and z10 BC

12x IFB, 1 x IFB & ISC-3
IS C-3*, 2 Gbp s 10/100 km HCA2-O LR 1x IFB, 5 Gbps 10/ 100 km 12x IFB, 6 GBps Up to 150 m
EC12
HCA3-O LR OR HCA2-O LR* HCA3-O L R OR HCA2-O L R* HCA 3-O OR HCA2-O* HCA2-O* OR HCA3-O
12 x IF B 3 or IFB 6 GBps 150 m
z19 6 and z114

12x IFB, 1 2x IFB3, 1x IFB , & ISC-3
1x IFB, 5 Gbps 10/ 100 km HCA3-O LR OR HCA2-O LR*
HCA2-O
HCA3-O OR HCA2-O*
IS C-3*, 2 Gbps 10/100 km 12x IFB3 or IFB 6 GBps 150 m
HCA2-O OR HCA3-O
I SC-3* 10/ 100 km
HCA2-O LR* OR HCA3-O L R 1x IF 5 Gb B, ps 10/100 km HCA3-O LR OR HCA2-O LR*
z9 EC and z9 BC z89 0, z990

Not supported in same Parallel Sysplex or STP CTN with zNext
HCA3-O OR HCA2-O*
*HC A2-O, HC A2-O L R, & ISC-3 carry fo rward o nly on zN ext
zNext
Figure 4-13 zEC12 Parallel Sysplex Coupling Connectivity
z/OS and coupling facility images can be running on the same or on separate servers. There must be at least one CF connected to all z/OS images, although there can be other CFs that are connected only to selected z/OS images. Two coupling facility images are required for system-managed CF structure duplexing and, in this case, each z/OS image must be connected to both duplexed CFs. To eliminate any single-points of failure in a Parallel Sysplex configuration, have at least the following components: Two coupling links between the z/OS and coupling facility images. Two coupling facility images not running on the same server. One stand-alone coupling facility. If using system-managed CF structure duplexing or running with resource sharing only, then a stand-alone coupling facility is not mandatory.
Coupling link features

The zEC12 supports five types of coupling link options: InterSystem Channel-3 (ISC-3) FC 0217, FC 0218, and FC 0219 HCA2-O fanout for 12x InfiniBand, FC 0163 HCA2-O LR fanout for 1x InfiniBand, FC 0168 HCA3-O fanout for 12x InfiniBand, FC 0171 HCA3-O LR fanout for 1x InfiniBand, FC 0170 The coupling link features available on the zEC12 connect the zEC12 servers to the identified System z servers by various link options: ISC-3 at 2 Gbps to zEC12, z196, z114, and System z10
163
8049ch04.fm
12x InfiniBand using HCA3-O and HCA2-O (12xIFB) fanout card at 6 GBps to zEC12, z196, z114 and System z10. 1x InfiniBand using both HCA3-O LR and HCA2-O LR at 5.0 or 2.5 Gbps to zEC12, z196, z114, and System z10 servers
ISC-3 coupling links

Three feature codes are available to implement ISC-3 coupling links: FC 0217, ISC-3 mother card FC 0218, ISC-3 daughter card FC 0219, ISC-3 port The ISC mother card (FC 0217) occupies one slot in the I/O cage or I/O drawer and supports up to two daughter cards. The ISC daughter card (FC 0218) provides two independent ports with one CHPID associated with each enabled port. The ISC-3 ports are enabled and activated individually (one port at a time) by Licensed Internal Code. When the quantity of ISC links (FC 0219) is selected, the quantity of ISC-3 port features selected determines the appropriate number of ISC-3 mother and daughter cards to be included in the configuration, up to a maximum of 12 ISC-M cards. Each active ISC-3 port in peer mode supports a 2 Gbps (200 MBps) connection through 9 m single mode fiber optic cables terminated with an LC Duplex connector. The maximum unrepeated distance for an ISC-3 link is 10 km. With repeaters the maximum distance extends to 100 km. ISC-3 links can be defined as timing-only links when STP is enabled. Timing-only links are coupling links that allow two servers to be synchronized using STP messages when a CF does not exist at either end of the link.
RPQ 8P2197 extended distance option

The RPQ 8P2197 daughter card provides two ports that are active and enabled when installed and do not require activation by LIC. This RPQ allows the ISC-3 link to operate at 1 Gbps (100 MBps) instead of 2 Gbps (200 MBps). This lower speed allows an extended unrepeated distance of 20 km. One RPQ daughter is required on both ends of the link to establish connectivity to other servers. This RPQ supports STP if defined as either a coupling link or timing-only.
HCA2-O fanout for 12x InfiniBand (FC 0163)

For detailed information, see 4.7.3, HCA2-O (12xIFB) fanout (FC0163) on page 138.
HCA2-O LR fanout for 1x InfiniBand (FC 0168)

For detailed information, see 4.7.4, HCA2-O LR (1xIFB) fanout (FC0168) on page 139.
HCA3-O fanout for 12x InfiniBand (FC 0171)

For detailed information, see 4.7.5, HCA3-O (12xIFB) fanout (FC0171) on page 140.
HCA3-O LR fanout for 1x InfiniBand (FC 0170)

For detailed information, see 4.7.6, HCA3-O LR (1xIFB) fanout (FC0170) on page 140.
Internal coupling links

IC links are Licensed Internal Code-defined links to connect a CF to a z/OS logical partition in the same server. These links are available on all System z servers. The IC link is a System z server coupling connectivity option that enables high-speed, efficient communication between a CF partition and one or more z/OS logical partitions running on the same server. The IC is a
164
8049ch04.fm
linkless connection (implemented in Licensed Internal Code) and so does not require any hardware or cabling. An IC link is a fast coupling link, using memory-to-memory data transfers. IC links do not have PCHID numbers, but do require CHPIDs. IC links require an ICP channel path definition at the z/OS and the CF end of a channel connection to operate in peer mode. They are always defined and connected in pairs. The IC link operates in peer mode and its existence is defined in HCD/IOCP. IC links have the following attributes: On System z servers, operate in peer mode (channel type ICP). Provide the fastest connectivity, significantly faster than any external link alternatives. Result in better coupling efficiency than with external links, effectively reducing the server cost associated with Parallel Sysplex technology. Can be used in test or production configurations, and reduce the cost of moving into Parallel Sysplex technology while enhancing performance and reliability. Can be defined as spanned channels across multiple CSSs. Are free of charge (no feature code). Employing ICFs with IC channels will result in considerable cost savings when configuring a cluster. IC links are enabled by defining channel type ICP. A maximum of 32 IC channels can be defined on a System z server.
Coupling link migration considerations

For a more specific explanation about migrating to InfiniBand coupling links, see the Coupling Facility Configuration Options white paper, available at: http://www.ibm.com/systems/z/advantages/pso/whitepaper.html
Coupling links and Server Time Protocol

All external coupling links can be used to pass time synchronization signals by using Server Time Protocol (STP). Server Time Protocol is a message-based protocol in which STP messages are passed over data links between servers. The same coupling links can be used to exchange time and coupling facility messages in a Parallel Sysplex. Using the coupling links to exchange STP messages has the following advantages: By using the same links to exchange STP messages and coupling facility messages in a Parallel Sysplex, STP can scale with distance. Servers exchanging messages over short distances, such as IFB links, can meet more stringent synchronization requirements than servers exchanging messages over long ISC-3 links (distances up to 100 km). This advantage is an enhancement over the IBM Sysplex Timer implementation, which does not scale with distance. Coupling links also provide the connectivity necessary in a Parallel Sysplex. Therefore, there is a potential benefit of minimizing the number of cross-site links required in a multi-site Parallel Sysplex. Between any two servers that are intended to exchange STP messages, it is best that each server be configured so that at least two coupling links exist for communication between the servers. This configuration prevents the loss of one link, causing the loss of STP communication between the servers. If a server does not have a CF logical partition, timing-only links can be used to provide STP connectivity.
165
8049ch04.fm
The zEC12 server does not support attachment to the IBM Sysplex Timer. A zEC12 can be added into a Mixed CTN only when there is a system z10 attached to the Sysplex Timer operating as Stratum 1 server. Connections to two Stratum 1 servers are preferable to provide redundancy and avoid a single point of failure. Important: A Parallel Sysplex in an ETR network must migrate to Mixed CTN or STP-only CTN before introducing a zEC12.
STP recovery enhancement

The new generation of host channel adapters (HCA3-O (12xIFB) or HCA3-O LR (1xIFB)), introduced for coupling, has been designed to send a reliable unambiguous going away signal to indicate that the server on which the HCA3 is running is about to enter a failed (check stopped) state. When the going away signal sent by the Current Time Server (CTS) in an STP-only Coordinated Timing Network (CTN) is received by the Backup Time Server (BTS), the BTS can safely take over as the CTS without relying on the previous Offline Signal (OLS) in a two-server CTN or the Arbiter in a CTN with three or more servers. This enhancement is exclusive to zEnterprise CPCs and zEC12 and is available only if you have an HCA3-O (12xIFB) or HCA3-O LR (1xIFB) on the CTS communicating with an HCA3-O (12xIFB) or HCA3-O LR (1xIFB) on the BTS. Note that the already available STP recovery design is still available for the cases when a going away signal is not received or for other failures besides a server failure. Important: See the following white paper if you have configured a Server Time Protocol (STP) Coordinated Timing Network (CTN) with three or more servers: http://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP101833 If the guidelines are not followed, this MIGHT result in all the servers in the CTN becoming unsynchronized, a condition that results in a sysplex wide outage. For STP configuration information, see: Server Time Protocol Planning Guide, SG24-7280, Server Time Protocol Implementation Guide, SG24-7281 and Server Time Protocol Recovery Guide, SG24-7380
4.10.2 External clock facility

The external clock facility (ECF) card located in CPC cage provides a Pulse Per Second (PPS) input. This PPS signal can be received from an Network Time Protocol (NTP) server acts as an external time sources (ETS). Two ECF cards are installed in the card slots above the books, to provide redundancy for continued operation and concurrent maintenance when a single ECF card fails. Each ECF card has a BNC connector for PPS connection support, attaching to two different ETS. Two PPS connections from two different NTP servers are preferable for redundancy. The time accuracy of an STP-only CTN is improved by adding an NTP server with the PPS output signa as the ETS device. STP tracks the highly stable accurate PPS signal from the NTP server and maintains accuracy of 10 s as measured at the PPS input of the zEC12 server. If STP uses an NTP server without PPS, a time accuracy of 100 ms to the ETS is maintained. NTP servers with PPS output are available from various vendors that offer network timing solutions.
166
8049ch04.fm
4.11 Cryptographic functions

Cryptographic functions are provided by the CP Assist for Cryptographic Function (CPACF) and the PCI Express cryptographic adapters. zEC12 supports the Crypto Express4S feature and in a carry forward only basis the Crypto Express3 cards when upgrading from earlier generations.
4.11.1 CPACF functions (FC 3863)

Feature code (FC) 3863 is required to enable CPACF functions.
4.11.2 Crypto Express4S feature (FC 0865)

Crypto Express4S is an optional and zEC12 exclusive feature. On the initial order, the minimum of two features are installed, thereafter the number of features increase by one at a time up to a maximum of sixteen features. Each Crypto Express4S feature holds one PCI Express cryptographic adapter. Each adapter can be configured by the installation as a Secure IBM CCA coprocessor, as a Secure IBM Enterprise PKCS #11 (EP11) coprocessor or as an accelerator. Each Crypto Express4S feature occupies one I/O slot in the PCIe I/O drawer, and it has no CHPID assigned, but uses one PCHID.
4.11.3 Crypto Express3 feature (FC 0864)

Crypto Express3 is an optional feature and it is available only in a carry forward basis when upgrading from earlier generations to zEC12. The minimum number of carry forward features are two and the maximum supported are eight features. Each Crypto Express3 feature holds two PCI Express cryptographic adapters. Either of the adapters can be configured by the installation as a Secure IBM CCA coprocessor or as an accelerator. Statement of Direction: The IBM zEnterprise EC12 is planned to be the last high-end System z server to offer support of the Crypto Express3 feature (#0864). Crypto Express3 will not be supported on future high-end System z servers as carry forward on an upgrade. Enterprises should begin migrating from the Crypto Express3 feature to the Crypto Express4S feature (#0865). Each Crypto Express3 feature occupies one I/O slot in the I/O cage or in the I/O drawer, and it has no CHPIDs assigned, but uses two PCHIDS. Cryptographic functions are described in Chapter 6, Cryptography on page 183.
4.12 Flash Express

The Flash Express cards are supported in PCIe I/O drawer with other PCIe I/O cards. They are plug into PCIe I/O drawers in pairs for availability. Like the Crypto Express4S cards, each card takes up a CHPID and no HCD/IOCP definition is required. Flash Express sub-channels are pre-defined and are allocated from the .25K reserved in sub-channel set 0.
167
8049ch04.fm
Flash Express cards are internal to the CPC and are accessible using the new System z architected Extended Asynchronous Data Mover (EADM) Facility. EADM is an extension of the ADM architecture used in the past with expanded storage. EADM access is initiated with a Start Subchannel instruction. zEC12 support a maximum of 4 pairs of Flash Express cards. Only one Flash Express card is allowed per Domain. The PCIe drawer has 4 I/O Domains and can install 2 pairs of Flash Express cards. Each pair is installed either in the front of PCIe I/O drawers at slots 1 and 14 or in the rear at slots 25 and 33. The Flash Express cards are first plug into the fronts slot of installed or being installed PCIe I/O drawer before plugging in rear of drawer. These four slots are reserved for Flash Express and not fill by other types of I/O cards until there is no spare slot. Figure 4-14 shows a PCIe I/O cage fully populated with Flash Express cards.
Top View
Card Slot 01 Card Slot 02 Card Slot 03 Card Slot 04 PCI-IN 0 Card Slot 06 Card Slot 07 Card Slot 08 Card Slot 09 FSP Card Slot 11 Card Slot 12 Card Slot 13 Card Slot 14 PCI-IN 2 D o m a i n 2 D o m a i n 3 D o m a i n 0 Card Slot 38 D Card Slot 37 o m Card Slot 36 a Card Slot 35 i PCI-IN 1 n Card Slot 33 1 Card Slot 32 Card Slot 31 Card Slot 30 FSP Card Slot 28 Card Slot 27 Card Slot 26 Card Slot 25 PCI-IN 3 Card Slot 23 Card Slot 22 Card Slot 21 Card Slot 20
Front
Rear
2 interconnect Card Slot 16 Card Slot 17 cables

Card Slot 18 Card Slot 19
2 interconnect cables
Figure 4-14 PCIe I/O cage fully populated with Flash Express cards.
168
8049ch05.fm
Chapter 5.
CPC Channel Subsystem

This chapter describes the concepts of the zEC12 channel subsystem, including multiple channel subsystems. Also discussed are the technology, terminology, and implementation aspects of the channel subsystem. This chapter discusses the following topics: Channel subsystem on page 170 I/O configuration management on page 178 Channel subsystem summary on page 178 System-initiated CHPID reconfiguration on page 180 Multipath initial program load on page 181
169
8049ch05.fm
5.1 Channel subsystem

The role of the channel subsystem (CSS) is to control communication of internal and external channels to control units and devices. The CSS configuration defines the operating environment for the correct execution of all system I/O operations. The CSS provides the server communications to external devices through channel connections. The channels execute transfer of data between main storage and I/O devices or other servers under the control of a channel program. The CSS allows channel I/O operations to continue independently of other operations within the central processors (CPs) and IFLs. The building blocks that make up a channel subsystem are shown in Figure 5-1.
EC12
Partitions
Partitions: Support the running of a System Control Program (SCP), such as z/OS and allow CPs, IFLs, memory, and Subchannels access to channels. Subchannels: A subchannel represents an I/O device to the hardware, and is used by the SCP to pass an I/O request from the SCP to the channel subsystem. Channel Subsystem: Controls queuing, de-queuing, and priority management, and I/O identification of all I/O operations performed by any logical partitions in a CSS. Channels: The communication path from the channel subsystem to the I/O network and the connected control units / devices.
Subchannels
Channel Subsystem
Channels
Figure 5-1 Channel subsystem overview
5.1.1 Multiple CSSs concept

The design of System z systems offers considerable processing power, memory size, and I/O connectivity. In support of the larger I/O capability, the CSS concept has been scaled up correspondingly to provide relief for the number of supported logical partitions, channels, and devices available to the system. A single channel subsystem allows the definition of up to 256 channel paths. To overcome this limit, the multiple channel subsystems concept was introduced. The architecture provides for up to four channel subsystems. The structure of the multiple CSSs provides channel connectivity to the defined logical partitions in a manner that is transparent to subsystems and application programs, enabling the definition of a balanced configuration for the processor and I/O capabilities. Each CSS can have from 1 to 256 channels and be configured to 1 to 15 logical partitions. Therefore, four CSSs support a maximum of 60 logical partitions. CSSs are numbered from 0 to 3 and are sometimes referred to as the CSS image ID (CSSID 0, 1, 2 or 3). These CSSs are also referred to as logical channel subsystems (LCSSs).
170
8049ch05.fm
5.1.2 CSS elements

The elements that encompass the CSS are described in this section.
Subchannels
A subchannel provides the logical representation of a device to a program and contains the information required for sustaining a single I/O operation. A subchannel is assigned for each device defined to the logical partition. Multiple subchannel sets, described in 5.1.3, Multiple subchannel sets on page 171, are available to increase addressability. Three subchannel sets per CSS are supported on EC12. Subchannel set 0 can have up to 63.75 K subchannels, and subchannel sets 1 and 2 can have up to 64 K subchannels each.
Channel paths
Each CSS can have up to 256 channel paths. A channel path is a single interface between a server and one or more control units. Commands and data are sent across a channel path to perform I/O requests.
Channel path identifier

Each channel path in the system is assigned a unique identifier value known as a channel path identifier (CHPID). A total of 256 CHPIDs are supported by the CSS, and a maximum of 1024 are supported per system (CPC). The channel subsystem communicates with I/O devices by means of channel paths between the channel subsystem and control units. On System z, a CHPID number is assigned to a physical location (slot/port) by the customer, through the hardware configuration definition (HCD) tool or IOCP.
Control units
A control unit provides the logical capabilities necessary to operate and control an I/O device and adapts the characteristics of each device so that it can respond to the standard form of control provided by the CSS. A control unit can be housed separately, or it can be physically and logically integrated with the I/O device, the channel subsystem, or within the system itself.
I/O devices
An I/O device provides external storage, a means of communication between data-processing systems, or a means of communication between a system and its environment. In the simplest case, an I/O device is attached to one control unit and is accessible through one channel path.
5.1.3 Multiple subchannel sets

Do not confuse the multiple subchannel set (MSS) functionality with multiple channel subsystems. In most cases, a subchannel represents an addressable device. For example, a disk control unit with 30 drives uses 30 subchannels (for base addresses), and so forth. An addressable device is associated with a device number and the device number is commonly (but incorrectly) known as the device address.
Chapter 5. CPC Channel Subsystem
171
8049ch05.fm
Subchannel numbers
Subchannel numbers (including their implied path information to a device) are limited to four hexadecimal digits by the architecture (0x0000 to 0xFFFF). Four hexadecimal digits provide 64 K addresses, known as a set. IBM has reserved 256 subchannels, leaving over 63 K subchannels for general use1. Again, addresses, device numbers, and subchannels are often used as synonyms, although this is not technically accurate. We might hear that there is a maximum of 63.75 K addresses or a
maximum of 63.75 K device numbers.

The processor architecture allows for sets of subchannels (addresses), with a current implementation of three sets. Each set provides 64 K addresses. Subchannel set 0, the first set, still reserves 256 subchannels for IBM use. Each of subchannel sets 1 and 2 provides the full range of 64 K subchannels. In principle, subchannels in either set can be used for any device-addressing purpose. These are referred to as special devices in the following topics. Figure 5-2 summarizes the multiple channel subsystems and multiple subchannel sets.
z196 EC12
Logical Channel Subsystem 0 Logical Channel Subsystem 1 Logical Channel Subsystem 2 Logical Channel Subsystem 3
Up to 15 Logical Partitions
Partitions
Partitions
Partitions
Partitions
CSS-0 Subchannels
CSS-1 Subchannels
CSS-2 Subchannels
CSS-3 Subchannels
SS-0
SS-1
SS-2
64K
SS-0
SS-1
SS-2
64K
SS-0
SS-1
SS-2
64K
SS-0
SS-1
SS-2
64K
63.75K Channels 64K
63.75K Channels 64K
63.75K Channels 64K
63.75K Channels 64K
Channels
Channels
Channels
Channels
Four Channel Subsystems
Multiple Subchannel Sets (MSS)
Figure 5-2 Multiple channel subsystems and multiple subchannel sets
The additional subchannel sets, in effect, add an extra high-order digit (either 0, 1 or 2) to existing device numbers. For example, we might think of an address as 08000 (subchannel set 0), or 18000 (subchannel set 1) or 28000 (subchannel set 2). Adding a digit is not done in system code or in messages because of the architectural requirement for four-digit addresses (device numbers or subchannels). However, certain messages do contain the subchannel set number, and you can mentally use that as a high-order digit for device numbers. Only a few
The number of reserved subchannel is 256. We abbreviate this to 63.75 K in this discussion to easily differentiate it from the 64 K subchannels available in subchannel sets 1 and 2.The informal name, 63.75 K subchannel, represents the following equation: (63 x 1024) + (0.75 x 1024) = 65280
172
8049ch05.fm
requirements refer to the subchannel sets 1 and 2, because they are only used for these special devices. JCL, messages, and programs rarely refer directly these special devices. Moving these special devices into an alternate subchannel set creates additional space for device number growth. The appropriate subchannel set number must be included in IOCP definitions or in the HCD definitions that produce the IOCDS. The subchannel set number defaults to zero.
IPL from an alternate subchannel set

zEC12 supports IPL from subchannel set 1 (SS1) or subchannel set 2 (SS2), in addition to subchannel set 0. Devices used early during IPL processing can now be accessed using subchannel set 1 or subchannel set 2. This allows the users of Metro Mirror (PPRC) secondary devices defined using the same device number and a new device type in an alternate subchannel set to be used for IPL, IODF, and stand-alone dump volumes when needed. IPL from an alternate subchannel set is supported by z/OS V1.13 or higher, as well as V1.12 and V1.11 with PTFs and applies to the FICON and zHPF protocols.
The display ios,config command

The z/OS display ios,config(all) command, shown in Figure 5-3, includes information about the MSSs. D IOS,CONFIG(ALL) IOS506I 18.21.37 I/O CONFIG DATA 610 ACTIVE IODF DATA SET = SYS6.IODF45 CONFIGURATION ID = TESTxxxx EDT ID = 01 TOKEN: PROCESSOR DATE TIME DESCRIPTION SOURCE: SCZP201 12-03-04 09:20:58 SYS6 IODF45 ACTIVE CSS: 0 SUBCHANNEL SETS CONFIGURED: 0, 1, 2 CHANNEL MEASUREMENT BLOCK FACILITY IS ACTIVE HARDWARE SYSTEM AREA AVAILABLE FOR CONFIGURATION CHANGES PHYSICAL CONTROL UNITS 8131 CSS 0 - LOGICAL CONTROL UNITS 4037 SS 0 SUBCHANNELS 62790 SS 1 SUBCHANNELS 61117 SS 2 SUBCHANNELS 60244 CSS 1 - LOGICAL CONTROL UNITS 4033 SS 0 SUBCHANNELS 62774 SS 1 SUBCHANNELS 61117 SS 2 SUBCHANNELS 60244 CSS 2 - LOGICAL CONTROL UNITS 4088 SS 0 SUBCHANNELS 65280 SS 1 SUBCHANNELS 65535 SS 2 SUBCHANNELS 62422 CSS 3 - LOGICAL CONTROL UNITS 4088 SS 0 SUBCHANNELS 65280 SS 1 SUBCHANNELS 65535 SS 2 SUBCHANNELS 62422 ELIGIBLE DEVICE TABLE LATCH COUNTS 0 OUTSTANDING BINDS ON PRIMARY EDT
Figure 5-3 Display ios,config(all) with MSS
173
8049ch05.fm
5.1.4 Parallel access volumes and extended address volumes

Parallel access volume (PAV) support enables a single System z system to simultaneously process multiple I/O operations to the same logical volume, which can help to significantly reduce device queue delays. Dynamic PAV allows the dynamic assignment of aliases to volumes to be under WLM control. With the availability of HyperPAV, the requirement for PAV devices is greatly reduced. HyperPAV allows an alias address to be used to access any base on the same control unit image per I/O base. It also allows different HyperPAV hosts to use one alias to access different bases, which reduces the number of alias addresses required. HyperPAV is designed to enable applications to achieve equal or better performance than possible with the original PAV feature alone, while also using the same or fewer z/OS resources. HyperPAV is an optional feature on the IBM DS8000 series. To further reduce the complexity of managing large I/O configurations System z introduced Extended Address Volumes (EAV). EAV is designed to build very large disk volumes using virtualization technology. By being able to extend the disk volume size, a client might potentially need fewer volumes to hold the data, therefore making systems management and data management less complex.
5.1.5 Logical partition name and identification

No logical partitions can exist without at least one defined CSS. Logical partitions are defined to a CSS, not to a system. A logical partition is associated with one CSS only. A logical partition is identified through its name, its identifier, and its multiple image facility (MIF) image ID (MIF ID). The logical partition name is user defined through HCD or the IOCP and is the partition name in the RESOURCE statement in the configuration definitions. Each name must be unique across the CPC. The logical partition identifier is a number in the range of 00 - 3F assigned by the user on the image profile through the support element (SE) or the hardware management console (HMC). It is unique across the CPC and might also be referred to as the user logical partition ID (UPID). The MIF ID is a number that is defined through the HCD tool or directly through the IOCP. It is specified in the RESOURCE statement in the configuration definitions. It is in the range of 1 F and is unique within a CSS. However, because of the multiple CSSs, the MIF ID is not unique within the CPC. The multiple image facility enables resource sharing across logical partitions within a single CSS or across the multiple CSSs. When a channel resource is shared across logical partitions in multiple CSSs, this is known as spanning. Multiple CSSs can specify the same MIF image ID. However, the combination CSSID.MIFID is unique across the CPC.
Dynamic addition or deletion of a logical partition name

All undefined logical partitions are reserved partitions. They are automatically predefined in the HSA with a name placeholder and a MIF ID.
Summary of identifiers
It is good practice to establish a naming convention for the logical partition identifiers. As shown in Figure 5-4 on page 175, which summarizes the identifiers and how they are defined, you can use the CSS number concatenated to the MIF ID, which means that logical partition
174
8049ch05.fm
ID 3A is in CSS 3 with MIF ID A. This fits within the allowed range of logical partition IDs and conveys helpful information to the user.
CSS0
CSS1
CSS2
CSS3
Specified in HCD / IOCP
Logical Partition Name

TST1 PROD1 PROD2
Logical Partition Name

TST2 PROD3 PROD4
Logical Log Part Partition Name Name

TST3 TST4 PROD5
Logical Partition ID
02 04 0A
14 16 1D
Log Part ID
22
35 3A
Specified in HMC Image Profile
MIF ID 2
MIF ID 4
MIF ID A
MIF ID 4
MIF ID 6
MIF ID D
MIF ID 2
MIF ID 5
MIF ID A
Figure 5-4 CSS, logical partition, and identifiers example
5.1.6 Physical channel ID

A physical channel ID (PCHID) reflects the physical identifier of a channel-type interface. A PCHID number is based on the I/O drawer or I/O cage location, the channel feature slot number, and the port number of the channel feature. A hardware channel is identified by a PCHID. In other words, the physical channel, which uniquely identifies a connector jack on a channel feature, is known by its PCHID number. Do not confuse PCHIDs with CHPIDs. A CHPID does not directly correspond to a hardware channel port, and can be arbitrarily assigned. Within a single channel subsystem, 256 CHPIDs can be addressed. That gives a maximum of 1,024 CHPIDs when four CSSs are defined. Each CHPID number is associated with a single channel. CHPIDs are not pre-assigned. The installation is responsible to assign the CHPID numbers through the use of the CHPID mapping tool (CMT) or HCD/IOCP. Assigning CHPIDs means that a CHPID number is associated with a physical channel/port location and a CSS. The CHPID number range is still from 00 - FF and must be unique within a CSS. Any non-internal CHPID that is not defined with a PCHID can fail validation when attempting to build a production IODF or an IOCDS.
5.1.7 Channel spanning

Channel spanning extends the MIF concept of sharing channels across logical partitions to sharing physical channels across logical partitions and channel subsystems. Spanning is the ability for a physical channel (PCHID) to be mapped to CHPIDs defined in multiple channel subsystems. When defined that way, the channels can be transparently shared by any or all of the configured logical partitions, regardless of the channel subsystem to which the logical partition is configured.
175
8049ch05.fm
A channel is considered a spanned channel if the same CHPID number in different CSSs is assigned to the same PCHID in IOCP, or is defined as spanned in HCD. In the case of internal channels (for example, IC links and HiperSockets), the same applies, but with no PCHID association. They are defined with the same CHPID number in multiple CSSs. In Figure 5-5, CHPID 04 is spanned to CSS0 and CSS1. Because it is an internal channel link, no PCHID is assigned. CHPID 06 is an external spanned channel and has a PCHID assigned.
Partition Partition 2 1
...
Partition Partition Partition Partition Partition 15 14 18 16 17
...
Partition 60
CSS0
CSS1
MIF-1
MIF-2
...
MIF-F
MIF-1
MIF-2
MIF-3
...
MIF-F
CHPID 00
CHPID 01
CHPID 02 PCHID 10D
CHPID 03 Share PCHID 20E
...
CHPID FF PCHID 20A
CHPID 04 SPAN
CHPID 06 SPAN
...
CHPID 00
CHPID 01
CHPID CHPID 05 22 Share PCHID 147 PCHID 158
CHPID FF PCHID 159
...
PCHID PCHID 10C 10B
PCHID 120
PCHID PCHID 145 146
Figure 5-5 zEC12 CSS: Channel subsystems with channel spanning
CHPIDs that span CSSs reduce the total number of channels available. The total is reduced, because no CSS can have more than 256 CHPIDs. For a zEC12 with two CSSs defined, a total of 512 CHPIDs is supported. If all CHPIDs are spanned across the two CSSs, then only 256 channels are supported. For a zEC12 with four CSSs defined, a total of 1024 CHPIDs is supported. If all CHPIDs are spanned across the four CSSs, then only 256 channels are supported. Channel spanning is supported for internal links (HiperSockets and Internal Coupling (IC) links) and for certain external links (FICON Express8S, and FICON Express8 channels, OSA-Express4S, OSA-Express3, and Coupling Links).
5.1.8 Multiple CSS construct

A pictorial view of a zEC12 with multiple CSSs defined is shown in Figure 5-6 on page 177. In this example, two channel subsystems are defined (CSS0 and CSS1). Each CSS has three logical partitions with their associated MIF image identifiers.
176
8049ch05.fm
zEC12
CSS LPAR name LPAR ID MIF ID MSS CHPID PCHID
80 140
CSS 0 LP1 01 1
SS 0 81 150
CSS 1 LP3 05 5
SS 2 90 1E0 91 1F0 80 141
LP2 03 3
SS 1
LP14 12 2
SS 0 81 151
LP15 13 3
SS 1 90 1E1
LP16 15 5
SS 2
91 1F1
Directors
61
62
Control Units and Devices

Figure 5-6 zEC12 CSS connectivity
Disk LCUs
Disk LCUs
In each CSS, the CHPIDs are shared across all logical partitions. The CHPIDs in each CSS can be mapped to their designated PCHIDs using the CHPID Mapping Tool (CMT) or manually using HCD or IOCP. The output of the CMT is used as input to HCD or the IOCP to establish the CHPID to PCHID assignments.
5.1.9 Adapter ID
When using HCD or IOCP to assign a CHPID to a Parallel Sysplex over an InfiniBand (IFB) coupling link port, an adapter ID (AID) number is required. The AID is bound to the serial number of the fanout. If the fanout is moved, the AID moves with it. No IOCDS update is required if adapters are moved to a new physical location.
5.1.10 Channel subsystem enhancement for I/O resilience

The zEC12 channel subsystem has been enhanced to provide improved throughput and I/O service times when abnormal conditions occur such as multi-system work load spikes, multi-system resource contention in the storage area network (SAN) or at the control unit ports, SAN congestion, improperly defined SAN configurations, dynamic changes in fabric routing, and destination port congestion. It could also apply to Licensed Internal Code (LIC) failures in the SAN, channel extenders, Wavelength Division Multiplexers, and control units. When abnormal conditions occur which can cause an imbalance in I/O performance characteristics (such as latency and throughput) across a set of channel paths to the control unit, the channel subsystem is designed to intelligently utilize the channels that provide
177
8049ch05.fm
optimal performance. This enhancement is accomplished by exploiting the in-band I/O instrumentation and metrics of the System z FICON and zHPF protocols. This channel subsystem enhancement is exclusive to zEC12 and is supported on all FICON channels when configured as CHPID type FC. This enhancement is transparent to operating systems.
5.2 I/O configuration management

For ease of management, it is preferable to use HCD to build and control the I/O configuration definitions. HCD support for multiple channel subsystems is available with z/VM and z/OS. HCD provides the capability to make both dynamic hardware and software I/O configuration changes. Tools are provided to help maintain and optimize the I/O configuration: IBM Configurator for e-business (eConfig): The eConfig tool is available to your IBM representative. It is used to create new configurations or upgrades of an existing configuration, and maintains installed features of those configurations. Reports produced by eConfig are helpful in understanding the changes being made for a system upgrade and what the final configuration will look like. Hardware configuration definition (HCD): HCD supplies an interactive dialog to generate the I/O definition file (IODF) and subsequently the input/output configuration data set (IOCDS). It is good practice to use HCD or HCM to generate the I/O configuration, as opposed to writing IOCP statements. The validation checking that HCD performs as data is entered helps minimize the risk of errors before the I/O configuration is implemented. Hardware configuration management (HCM): HCM is a priced optional feature that supplies a graphical interface to HCD. It is installed on a PC and allows managing both the physical and the logical aspects of a mainframe systems hardware configuration. CHPID mapping tool (CMT) The CHPID Mapping Tool provides a mechanism to map CHPIDs onto PCHIDs as required. Additional enhancements have been built into the CMT to cater to the requirements of the zEC12. It provides the best availability choices for the installed features and defined configuration. CMT is a workstation-based tool available for download from the IBM Resource Link site: http://www.ibm.com/servers/resourcelink The health checker function in z/OS V1.10 introduces a health check in the I/O Supervisor that can help system administrators identify single points of failure in the I/O configuration.
5.3 Channel subsystem summary

The zEC12 provides support for the full architecture. Table 5-1 shows CSS-related information in terms of maximum values for devices, subchannels, logical partitions, and CHPIDs.
178
8049ch05.fm
Table 5-1 zEC12 CSS overview Setting zEC12
Maximum number of CSSs Maximum number of CHPIDs Maximum number of LPARs supported per CSS Maximum number of LPARs supported per system Maximum number of HSA subchannels Maximum number of devices Maximum number of CHPIDs per CSS Maximum number of CHPIDs per logical partition Maximum number of subchannel per logical partition
4 1024 15 60 11505 K (191.75 K per partition x 60 partitions) 255 K (4 CSSs x 63.75 K devices) 256 256 191.75 K (63.75 K + 2 x 64 K)
All channel subsystem images (CSS images) are defined within a single I/O configuration data set (IOCDS). The IOCDS is loaded and initialized into the hardware system area (HSA) during system power-on reset.The HSA is pre-allocated in memory with a fixed size of 32 GB. This eliminates planning for HSA and pre-planning for HSA expansion, because HCD/IOCP always reserves the following items by the IOCDS process: Four CSSs 15 LPARs in each CSS Subchannel set 0 with 63.75 K devices in each CSS Subchannel set 1 with 64 K devices in each CSS Subchannel set 2 with 64 K devices in each CSS All these are designed to be activated and used with dynamic I/O changes. Figure 5-7 shows a logical view of the relationships. Note that each CSS supports up to 15 logical partitions. System-wide, a total of up to 60 logical partitions are supported.
179
8049ch05.fm
Driver
H89 HA1
1-101 PUs
Single LIC (CPC-wide)
H20
1-20 PUs 32-704 GB
H43
1-43 PUs
H66
1-66 PUs
Up to four books per server Available Processing Units (min 1, max 101) Memory (min 32 GB, max 3040 GB) Up to 15 Logical Partitions per LCSS Up to 60 Logical Partitions per CEC Up to 4 Logical Channel Subsystems Up to 3 Subchannel Sets per CSS Single HSA with one active IOCDS IFB-C, and IFB-O cables
32-1392 GB 32-2272 GB 32-3040 GB
Logical Partitions
CSS 0 up to 256 CHPIDs SS 0 SS 1 SS 2 CSS 1 CSS 2 up to 256 up to 256 CHPIDs CHPIDs SS 0 SS 1 SS 2 SS 0 SS 1 SS 2 CSS 3 up to 256 CHPIDs SS 0 SS 1 SS 2
HSA (32 GB)

IOCDS
1-16 IFB C/O 1-32 IFB C/O 1-40 IFB C/O 1-48 IFB C/O
Physical Channels (PCHIDs)
I/O Cage with features
Figure 5-7 Logical view of zEC12 models, CSSs, IOCDS, and HSA
Book repair: The HSA can be moved from one book to a different book in an enhanced availability configuration as part of a concurrent book repair action. The channel definitions of a CSS are not bound to a single book. A CSS can define resources that are physically connected to any InfiniBand cable of any book in a multibook CPC.
5.4 System-initiated CHPID reconfiguration

The system-initiated CHPID reconfiguration function is designed to reduce the duration of a repair action and minimize operator interaction when a FICON channel, an OSA port, or an ISC-3 link is shared across logical partitions on a zEC12 server. When an I/O card is to be replaced (for a repair), it usually has a few failed channels and others that are still functioning. To remove the card, all channels must be configured offline from all logical partitions sharing those channels. Without system-initiated CHPID reconfiguration, this means that the CE must contact the operators of each affected logical partition and have them set the channels offline, and then, after the repair, contact them again to configure the channels back online. With system-initiated CHPID reconfiguration support, the support element sends a signal to the channel subsystem that a channel needs to be configured offline. The channel subsystem determines all the logical partitions sharing that channel and sends an alert to the operating systems in those logical partitions. The operating system then configures the channel offline without any operator intervention. This cycle is repeated for each channel on the card. When the card is replaced, the Support Element sends another signal to the channel subsystem for each channel. This time, the channel subsystem alerts the operating system that the channel has to be configured back online. This process minimizes operator interaction to configure channels offline and online.
180
8049ch05.fm
System-initiated CHPID reconfiguration is supported by z/OS.
5.5 Multipath initial program load

Multipath initial program load (IPL) helps increase availability and helps eliminate manual problem determination during IPL execution. This allows IPL to complete, if possible, using alternate paths when executing an IPL from a device connected through FICON channels. If an error occurs, an alternate path is selected. Multipath IPL is applicable to FICON channels (CHPID type FC). z/OS supports multipath IPL.
181
8049ch05.fm
182
8049ch06.fm
Chapter 6.
Cryptography
This chapter describes the hardware cryptographic functions available on the IBM zEnterprise EC12. The CP Assist for Cryptographic Function (CPACF) along with the PCIe Cryptographic Coprocessors offer a balanced use of resources and unmatched scalability. The zEC12 include both standard cryptographic hardware and optional cryptographic features for flexibility and growth capability. IBM has a long history of providing hardware cryptographic solutions, from the development of Data Encryption Standard (DES) in the 1970s to have the Crypto Express tamper-sensing and tamper-responding programmable features designed to meet the U.S. Government's highest security rating FIPS 140-2 Level 41. The cryptographic functions include the full range of cryptographic operations necessary for e-business, e-commerce, and financial institution applications. User-Defined Extensions (UDX) permits to add custom cryptographic functions to the set of functions that the zEC12 offers. Secure Sockets Layer/Transport Layer Security (SSL/TLS) is a key technology for conducting secure e-commerce using Web servers, and it has being adopted by a rapidly increasing number of applications, demanding new levels of security, performance, and scalability. This chapter discusses the following topics: Cryptographic synchronous functions on page 184 Cryptographic asynchronous functions on page 184 CPACF protected key on page 187 PKCS #11 Overview on page 189 Cryptographic feature codes on page 193 CP Assist for Cryptographic Function on page 194 Crypto Express4S on page 194 Crypto Express3 on page 197 Tasks performed by PCIe Crypto Express on page 199 TKE workstation feature on page 203 Cryptographic functions comparison on page 208
183
8049ch06.fm
6.1 Cryptographic synchronous functions

Cryptographic synchronous functions are provided by the CP Assist for Cryptographic Function (CPACF). The CPACF must be explicitly enabled using a no-charge enablement feature (Feature Code #3863). For IBM and customer written programs, CPACF functions can be invoked by instructions described in the z/Architecture Principles of Operation, SA22-7832. As a group, these instructions are known as the Message-Security Assist (MSA). z/OS Integrated Cryptographic Service Facility (ICSF) callable services on z/OS as well as in-kernel crypto APIs and libica cryptographic functions library running at Linux on System z can also invoke CPACF synchronous functions. The zEC12 hardware includes the implementation of algorithms as hardware synchronous operations, which means holding the PU processing of the instruction flow until the operation has completed. The synchronous functions are as follows: Data encryption and decryption algorithms for data privacy and confidentially Data Encryption Standard (DES), which includes: Single-length key DES Double-length key DES Triple-length key DES (also known as Triple-DES) Advanced Encryption Standard (AES) for 128-bit, 192-bit, and 256-bit keys Hashing algorithms for data integrity, such as SHA-1, and SHA-2 support for SHA-224, SHA-256, SHA-384, and SHA-512 Message authentication code (MAC): Single-length key MAC Double-length key MAC Pseudo Random Number Generation (PRNG) for cryptographic key generation Keys: The keys must be provided in clear form only. SHA-1, and SHA-2 support for SHA-224, SHA-256, SHA-384, and SHA-512 are shipped enabled on all servers and do not require the CPACF enablement feature. The CPACF functions are supported by z/OS, z/VM, z/VSE, zTPF and Linux on System z.
6.2 Cryptographic asynchronous functions

Cryptographic asynchronous functions are provided by the optional Peripheral Component Interconnect Express (PCIe) cryptographic coprocessors Crypto Express4S and/or Crypto Express3...
6.2.1 Secure key functions

The following secure key functions are provided as cryptographic asynchronous functions. System internal messages are passed to the cryptographic coprocessors to initiate the operation, then messages are passed back from the coprocessors to signal completion of the operation: Data encryption and decryption algorithms for data protection: Data Encryption Standard (DES), which includes:
184
8049ch06.fm
Single-length key DES Double-length key DES Triple-length key DES (Triple-DES) DES key generation and distribution PIN generation, verification, and translation functions Random number generator PKCS #11 functions2 ICSF has implemented callable services in support of PKCS #11 standard. Secure IBM Enterprise PKCS #11 (EP11) coprocessor mode implements secure keys for PKCS #11 functions. Public key algorithm (PKA) functions Supported callable services intended for application programs that use PKA include: Importing RSA public-private key pairs in clear and encrypted forms Rivest-Shamir-Adelman (RSA), which can provide: Key generation, up to 4,096-bit Signature generation and verification, up to 4,096-bit Import and export of DES keys under an RSA key, up to 4,096-bit
Public key encryption (PKE) / Public key decryption (PKD): The PKE and PKD callable services are provided for assisting the SSL/TLS handshake. They are used to offload compute-intensive portions of the protocol onto the cryptographic adapters. Europay Mastercard VISA (EMV) standard: Applications can be written to comply with the EMV standard for financial transactions between heterogeneous hardware and software. EMV standards have been updated to exploit improved security properties of EMV contact and contactless cards. ICSF HRC770A improved support of EMV card applications that support American Express cards.
6.2.2 Additional functions

Other key functions of the Crypto Express features serve to enhance the security of the cryptographic processing: Remote loading of initial ATM keys: This function provides the ability to remotely load the initial keys for capable Automated Teller Machines (ATM) and Point of Sale (POS) systems. Remote key loading refers to the process of loading DES keys to ATM from a central administrative site without requiring someone to manually load the DES keys on each machine. The standard ANSI X9.24-2 defines the acceptable methods of doing this using public key cryptographic techniques. The process uses ICSF callable services along with the Crypto Express4S or Crypto Express3 features to perform the remote load. Trusted Block Create (CSNDTBC) is a callable service that is used to create a trusted block containing a public key and certain processing rules. The rules define the ways and formats in which keys are generated and exported. Remote Key Export (CSNDRKX) is a callable service that uses the trusted block to generate or export DES keys for local use and for distribution to an ATM or other remote device. The PKA Key Import (CSNDPKI),
2
Requires Crypto Express4S and TKE workstation
Chapter 6. Cryptography
185
8049ch06.fm
PKA Key Token Change (CSNDKTC), and Digital Signature Verify (CSFNDFV) callable services support the remote key loading process. Key exchange with non-CCA cryptographic systems: This function allows the exchange of operational keys between the Crypto Express4S or Crypto Express3 coprocessors and non-CCA systems, such as the Automated Teller Machines (ATM). IBM Common Cryptographic Architecture (CCA) employs control vectors to control the usage of cryptographic keys. Non-CCA systems use other mechanisms, or can use keys that have no associated control information. Enhancements to key exchange functions added to the CCA the ability to exchange keys between CCA systems and systems that do not use control vectors.It allows the CCA system owner to define permitted types of key to be imported and exported while preventing uncontrolled key exchange that can open the system to an increased threat of attack. Elliptic Curve Cryptography (ECC) Digital Signature Algorithm support: Elliptic Curve Cryptography is an emerging public-key algorithm intended to eventually replace RSA cryptography in many applications. ECC is capable of providing digital signature functions and key agreement functions. The CCA functions provide ECC key generation and key management and provide digital signature generation and verification functions compliant with the ECDSA method described in ANSI X9.62 Public Key Cryptography for the Financial Services Industry: The Elliptic Curve Digital Signature Algorithm (ECDSA). ECC uses keys that are shorter than RSA keys for equivalent strength-per-key-bit. So the strength-per-key-bit is substantially greater in an algorithm that uses elliptic curves. This Crypto function is supported by z/OS, z/VM and Linux on System z. Elliptic Curve Diffie-Hellman (ECDH) algorithm support: The Common Cryptographic Architecture has been extended to include the Elliptic Curve Diffie-Hellman (ECDH) algorithm. Elliptic Curve Diffie-Hellman (ECDH) is a key agreement protocol that allows two parties, each having an elliptic curve public-private key pair, to establish a shared secret over an insecure channel. This shared secret can be used directly as a key, or to derive another key which can then be used to encrypt subsequent communications using a symmetric key cipher such as AES KEK. Enhancements include: Key management function to support AES KEK Generating an ECC private key wrapped with an AES KEK Importing and exporting an ECC private key wrapped with an AES KEK Support for ECDH with a new service
User-Defined Extensions (UDX) support: UDX allows the user to add customized operations to a cryptographic coprocessor. User-Defined Extensions to the Common Cryptographic Architecture (CCA) support customized operations that execute within the Crypto Express features when defined as coprocessor. UDX is supported under a special contract through an IBM or approved third-party service offering. The CryptoCards website directs your request to an IBM Global Services location appropriate for your geographic location. A special contract is negotiated between you and IBM Global Services. The contract is for development of the UDX code by IBM Global Services according to your specifications and an agreed-upon level of the UDX. An UDX toolkit for System z is tied to specific versions of the CCA card code and the related host code. UDX is available for the Crypto Express4S (Secure IBM CCA coprocessor mode only) and Crypto Express3 feature. An UDX migration is no more disruptive than a normal MCL or ICSF release migration.
186
8049ch06.fm
In zEC12 it is allowed to import up to 4 UDX files and they can be imported only from a DVD-ROM. The UDX configuration panel was updated to include a Reset to IBM Default bottom. Note: CCA will have a new code level at zEC12 and the UDX customers will require a new UDX. For more information, see the IBM CryptoCards website: http://www.ibm.com/security/cryptocards
6.3 CPACF protected key

The zEC12 supports the protected key implementation. Since PCIXCC deployment, secure keys are processed on the PCI-X and PCIe cards, requiring an asynchronous operation to move the data and keys from the general purpose CP to the crypto cards. Clear keys process faster than secure keys because the process is done synchronously on the CPACF. Protected keys blend the security of Crypto Express4s or Crypto Express3 coprocessors and the performance characteristics of the CPACF, running closer to the speed of clear keys. An enhancement to CPACF facilitates the continued privacy of cryptographic key material when used for data encryption. In Crypto Express4S and Crypto Express3 coprocessors, a secure key is encrypted under a master key, whereas a protected key is encrypted under a wrapping key that is unique to each LPAR. As the wrapping key is unique to each LPAR, a protected key cannot be shared with another LPAR. CPACF, using key wrapping, ensures that key material is not visible to applications or operating systems during encryption operations. CPACF code generates the wrapping key and stores it in the protected area of hardware system area (HSA). The wrapping key is accessible only by firmware. It cannot be accessed by operating systems or applications. DES/T-DES and AES algorithms were implemented in CPACF code with support of hardware assist functions. Two variations of wrapping key are generated, one for DES/T-DES keys and another for AES keys. Wrapping keys are generated during the clear reset each time an LPAR is activated or reset. There is no customizable option available at SE or HMC that permits or avoids the wrapping key generation. Figure 6-1 on page 188 shows this function flow.
187
8049ch06.fm
CPACF Wrapped key stored in ICSF address space in fetch protected storage
Source key is stored in CKDS as a CCA MK wrapped key
ICSF ICSF
CPACF Wrapped Key
Secure Key Default Data Key
CKDS
Secure Key
Software Hardware Secure Key
Crypto Express Feature

CCA
System zz System Millicode Millicode

CPACF Wrapped Key
Cleartext Key CCA Master Key
Cleartext Key
Wrapping Key
CPACF CPACF CPACF
CPACF Wrapping key resides in protected HSA storage and is not visible for operating system
Figure 6-1 CPACF key wrapping
If a CEX4C or a CEX3C is available, a protected key can begin its life as a secure key. Otherwise, an application is responsible for creating or loading a clear key value and then using the PCKMO instruction to wrap the key. ICSF is not called by application if the Crypto Express4S or the Crypto Express3 is not available. A new segment in the profiles at the CSFKEYS class in RACF restricts which secure keys can be used as protected keys. By default, all secure keys are considered not eligible to be used as protected keys. The process described in Figure 6-1 considers a secure key as being the source of a protected key. The source key in this case was already stored in CKDS as a secure key (encrypted under the master key). This secure key is sent to Crypto Express4S or to the Crypto Express3 to be deciphered and sent to CPACF in clear text. At CPACF, the key is wrapped under the LPAR wrapping key and then it is returned to ICSF. After the key is wrapped, ICSF can keep the protected value in memory, passing it to the CPACF, where the key will be unwrapped for each encryption/decryption operation. The protected key is designed to provide substantial throughput improvements for a large volume of data encryption as well as low latency for encryption of small blocks of data. A high performance secure key solution, also known as a protected key solution, requires HCR7770 as a minimum release.
188
8049ch06.fm
6.4 PKCS #11 Overview

The PKCS #11 is one of the industry-accepted standards called Public Key Cryptographic Standards (PKCS) provided by RSA Laboratories of RSA Security Inc. The PKCS #11 specifies an application programming interface (API) to devices, referred to as tokens, that hold cryptographic information and perform cryptographic functions. PKCS #11 provides an alternative to IBMs Common Cryptographic Architecture (CCA). The PKCS #11 describes the cryptographic token interface standard and its API is also know as Cryptoki (Cryptographic Token Interface). It is a de facto industry standard on many computing platforms today. It is a higher level API when compared to CCA and it is easier to use by language C based applications. The persistent storage/retrieval of objects is part of the standard. The objects are certificates, keys, and even application specific data objects.
6.4.1 The PKCS #11 model

On most single-user systems, a token is a smart card or other plug-installed cryptographic device, accessed through a card reader or slot. Cryptoki provides a logical view of slots and tokens making each cryptographic device looks logically like every other device regardless of the technology being used. The PKCS #11 specification assigns numbers to slots, known as slot IDs. An application identifies the token that it wants to access by specifying the appropriate slot ID. On systems that have multiple slots, the application determines which slot to access. The PKCS #11 logical view of a token is a device that stores objects and can perform cryptographic functions. PKCS #11 defines three types of objects: A data object that is defined by an application. A certificate object that stores a digital certificate. A key object that stores a cryptographic key. The key can be a public key, a private key, or a secret key. Objects are also classified according to their lifetime and visibility: Token objects are visible to all applications connected to the token that have sufficient permission, and they remain on the token even after the sessions, which are connections between an application and the token, are closed, and the token is removed from its slot. Session objects are more temporary, and when a session is closed by any means, all session objects created by that session are automatically destroyed. Furthermore, session objects are visible only to the application that created them. Attributes are characteristics that distinguish an instance of an object. General attributes in PKCS #11, distinguish, for example, whether the object is public or private. Other attributes are specific to a particular type of object, such as a Modulus or exponent for RSA keys. The PKCS #11 standard was designed for systems that grant access to token information based on a PIN. The standard recognizes two types of token user: Security officer (SO) Standard user (USER) The role of the SO is to initialize a token (zeroize the content) and set the Users PIN. The SO can also access public objects on the token but not private ones. The User can access private objects on the token. Access is granted only after the User has been authenticated. Users can also change their own PINs. Users cannot, however, re-initialize a token.
189
8049ch06.fm
Security Officer PIN for Token Initialization APPLICATION
TOKEN
Public Objects
PKCS#11 API (CRYPTOKI)
SLOT
Processor Private Objects
Protected by the Users PIN

Figure 6-2 The PKCS #11 General Model
The PKCS #11 general model components are represented in Figure 6-2: Token - Logical view of a cryptographic device (e.g. smart card, HSM) Slot - Logical view of a smart card reader Objects - Items stored in a token (e.g. digital certificate, cryptographic key) User - The owner of the private data on the token by knowing the access Personal Identification Number (PIN) Security Officer - Person who initializes the token and the User PIN.
6.4.2 z/OS PKCS #11 implementation

ICSF supports PKCS #11 standard broadening the scope of cryptographic applications that make use of System z cryptography. The PKCS #11 support was introduced in ICSF FMID HCR7740 within z/OS V1R9. In zEC12, along with Crypto Express4S and FMID HCR77A0, ICSF expanded the support and introduced PKCS #11 secure keys. On z/OS, PKCS #11 tokens are not physical cryptographic devices but rather virtual smart cards. New tokens can be created at any time. The tokens can be application specific or system-wide, depending on the access control definitions, which are used instead of PINs. The tokens and their contents are stored in a new ICSF VSAM data set, the Token Key Data Set (TKDS). TKDS serves as the repository for cryptographic keys and certificates used by PKCS #11 applications. z/OS provides several facilities to manage tokens:
190
8049ch06.fm
A C language application program interface (API) that implements a subset of the PKCS #11 specification. Token management ICSF callable services, which are also used by the C API. The ICSF ISPF panel, called Token Browser, which provides the capability to see a formatted view of TKDS objects and make minor, limited updates to them. The RACF RACDCERT command supports the certificate, public key, and private key objects and provides subfunctions to manage these objects together with tokens. The gskkyman command supports management of certificates and keys similar to the way RACFDCERT does. ICSF supports PKCS #11 session objects and token objects. ICSF supports PKCS#11 session objects and token objects. Session objects exist in memory only. They are not maintained on the direct access storage device (DASD).I An application has only one session objects database, even if the application spawns multiple PKCS #11 sessions. Token objects are stored in the TKDS with one record per object. They are visible to all applications having sufficient permission to the token. The objects are persistent and remain associated with the token even after a session is closed. The PKCS #11 standard was designed for systems that grant access to token information based on a PIN. z/OS does not use PINs; instead, profiles in the SAF CRYPTOZ class control access to tokens. Each token has two resources in the CRYPTOZ class: The resource USER.token-name controls the access of the user role to the token. The resource SO.token-name controls the access of the SO role to the token. A users access level to each of these resources (read, update, and control) determines the users access level to the token.
TOKEN Public Objects SLOT Processor Private Objects
APPLICATION
PKCS#11 Implementation Model PKCS#11 z/OS Implementation
Protected by the Users PIN
Protected by SAF CRYPTOZ CLASS

APPLICATION
Token Label Token Label
Public Objects Private Objects
RACF
Tokens In TKDS
ICSF
TKDS
Figure 6-3 Mapping the PKCS #11 model to the z/OS PKCS #11 implementation
191
8049ch06.fm
.In Figure 6-3 on page 191 we map the concepts introduced by PKCS #11 model to the z/OS PKCS #11 implementation.
Tokens
The PKCS #11 tokens on z/OS are virtual, conceptually similar to RACF (SAF) keyrings. An application can have one or more z/OS PKCS #11 tokens, depending on its need. z/OS PKCS #11 tokens are created using system software such as RACF, the gskkyman utility or by applications using the C API. Also ICSF panels can be used for token management with limited usability. Each token has a unique token name or label that is specified by the user or application when the token is created. Similar to the way that z/OS PKCS #11 supports token creation, the PKCS #11 tokens can be deleted using the same system software tools used when created.
Token Key Data Set (TKDS)

The Token Key Data Set (TKDS) is a VSAM data set that serves as the repository for cryptographic keys and certificates used by z/OS PKCS #11 applications. Before an installation can run PKCS #11 applications, the TKDS must be created, and the ICSF installation options data set updated to identify the name of the TKDS data set. To optimize performance, ICSF creates a data space containing an in-storage copy of the TKDS. Important: Up to ICSF FMID HCR7790 keys in the token key data set were not encrypted. Therefore it is important that the RACF profile be created to protect the token key data set from unauthorized access.
6.4.3 Secure IBM Enterprise PKCS #11 (EP11) Coprocessor

The IBM Enterprise PKCS #11 Licensed Internal Code (LIC) implements the industry standardized set of services that adhere to the PKCS #11 specification V2.20 and more recent amendments. It is designed to meet the Common Criteria (EAL 4+) and FIPS 140-2 Level 4 certifications. It conforms to the Qualified Digital Signature (QDS) Technical Standards that is becoming mandate by the European Union. The PKCS #11 secure key support is provided by the Crypto Express4S card configured in Secure IBM Enterprise PKCS #11 (EP11) coprocessor mode. Prior to EP11, ICSF PKCS #11 implementation only supported clear keys and the key protection was accomplished only by RACF CRYPTOZ class protection. In EP11, keys now can be generated and securely wrapped under the EP11 Master Key. The secure keys never leave the secure coprocessor boundary unencrypted. The Crypto Express4S firmware has an unique code for EP11 separated from the CCA code. Crypto Express4S with EP11 configuration is known as CEX4P. There is no change in the Cryptos domain configuration in the LPAR activation profiles. The configuration selection is performed in the Cryptographic Configuration panel in Support Element. A coprocessor in EP11 mode is configured off after being zeroized. Attention: The Trusted Key Entry (TKE) workstation is required for management of the Crypto Express4S when defined as an EP11 coprocessor.
192
8049ch06.fm
6.5 Cryptographic feature codes

Table 6-1 lists the cryptographic features available.
Table 6-1 Cryptographic features for zEnterprise CPC
Feature code
3863
Description
CP Assist for Cryptographic Function (CPACF) enablement: This feature is a prerequisite to use CPACF (except for SHA-1, SHA-224, SHA-256, SHA-384, and SHA-512) and Crypto Express features. Crypto Express3 feature: A maximum of eight features can be carry forwarded. It is an optional feature and each feature contains two PCI Express cryptographic adapters (adjunct processors). This feature is not supported as a new build, it is available only in a carry forward basis when upgrading from earlier generations to zEC12. Crypto Express4S feature: A maximum of sixteen features can be ordered. It is an optional feature and each feature contains one PCI Express cryptographic adapters (adjunct processor). Trusted Key Entry (TKE) workstation: This feature is optional. TKE provides a basic key management (key identification, exchange, separation, update, backup), as well as security administration. The TKE workstation has one Ethernet port and supports connectivity to an Ethernet Local Area Network (LAN) operating at 10, 100, or 1000 Mbps. Up to ten (10) features per zEC12 can be installed TKE 7.2 Licensed Internal Code (TKE 7.2 LIC): The 7.2 LIC requires Trusted Key Entry workstation feature code 0841. It is required to support CEX4P. The 7.2 LIC can also be used to control z196, z114, z10 EC, z10 BC, z9 EC, z9 BC, z990, and z890 servers. TKE Smart Card Reader: Access to information in the smart card is protected by a personal identification number (PIN). One (1) feature code includes two Smart Card Readers, two cables to connect to the TKE workstation, and 20 smart cards. Smart card part 74Y0551 is required to support CEX4P. TKE additional smart cards: When one feature code is ordered a quantity of 10 smart cards are shipped. Order increment is one up to 99 (990 blank smart cards). Smart card part 74Y0551 is required to support CEX4P.
0864
0865
0841
0850
0885
0884
TKE includes support for the AES encryption algorithm with 256-bit master keys and key management functions to load or generate master keys to the cryptographic coprocessor. If the TKE workstation is chosen to operate the Crypto Express features in a zEC12, a TKE workstation with the TKE 7.2 LIC or later is required. See 6.10, TKE workstation feature on page 203 for a more detailed description. Important: Products that include any of the cryptographic feature codes contain cryptographic functions that are subject to special export licensing requirements by the United States Department of Commerce. It is the customers responsibility to understand and adhere to these regulations when moving, selling, or transferring these products.
193
8049ch06.fm
6.6 CP Assist for Cryptographic Function

The CP Assist for Cryptographic Function (CPACF) offers a set of symmetric cryptographic functions that enhance the encryption and decryption performance of clear key operations for SSL, VPN, and data-storing applications that do not require FIPS 140-2 level 4 security3. CPACF is designed to facilitate the privacy of cryptographic key material when used for data encryption through key wrapping implementation. It ensures that key material is not visible to applications or operating systems during encryption operations. For additional information see 6.3, CPACF protected key on page 187 The CPACF feature provides hardware acceleration for DES, Triple-DES, MAC, AES-128, AES-192, AES-256, SHA-1, SHA-224, SHA-256, SHA-384, and SHA-512 cryptographic services. It provides high-performance hardware encryption, decryption, and hashing support. The following instructions support the cryptographic assist function: KMAC KM KMC KMF KMCTR KMO KIMD KLMD PCKMO Compute Message Authentic Code Cipher Message Cipher Message with Chaining Cipher Message with CFB Cipher Message with Counter Cipher Message with OFB Compute Intermediate Message Digest Compute Last Message Digest Provide Cryptographic Key Management Operation
These functions are provided as problem-state z/Architecture instructions, directly available to application programs. These instructions are known as Message-Security Assist (MSA). When enabled, the CPACF runs at processor speed for every CP, IFL, zIIP, and zAAP. Details on MSA instructions can be found in the z/Architecture Principles of Operation, SA22-7832. The functions of the CPACF must be explicitly enabled using FC 3863 by the manufacturing process or at the customer site as a MES installation, except for SHA-1, and SHA-2 support for SHA-224, SHA-256, SHA-384, and SHA-512, which are always enabled.
6.7 Crypto Express4S

The Crypto Express4S feature (FC 0865) is an optional and zEC12 exclusive feature. Each feature has one Peripheral Component Interconnect Express (PCIe) cryptographic adapter. The Crypto Express4S feature occupies one I/O slot in a zEC12 PCIe I/O drawer. This feature provides a secure programming and hardware environment wherein crypto processes are performed. Each cryptographic coprocessor includes a general-purpose processor, non-volatile storage, and specialized cryptographic electronics. The Crypto Express4S feature provides tamper-sensing and tamper-responding, high-performance cryptographic operations. Each Crypto Express4S PCI Express adapter can be configured as one of the following:
Secure IBM CCA coprocessor (CEX4C) for Federal Information Processing Standard
(FIPS) 140-2 Level 4 certification. This includes secure key functions and it is optionally
Federal Information Processing Standard
194
8049ch06.fm
programmable to deploy additional functions and algorithms using User Defined Extension (UDX).
Secure IBM Enterprise PKCS #11 (EP11) coprocessor (CEX4P) implements industry
standardized set of services that adhere to the PKCS #11 specification v2.20 and more recent amendments. It was designed for extended FIPS and Common Criteria evaluations to meet public sector requirements. This new cryptographic coprocessor mode introduced the PKCS #11 secure key function. Trusted Key Entry (TKE) workstation is required to support the administration of the Crypto Express4S when configured as EP11 mode.
Accelerator (CEX4A) for acceleration of public key and private key cryptographic
operations used with Secure Sockets Layer / Transport Layer Security (SSL/TLS) processing. These modes can be configured via the Support Element, and the PCIe adapter must be configured offline to change the mode. Note: Switching between configuration modes will erase all card secrets. Exception occurs when switching from Secure CCA to accelerator and vice versa. The Crypto Express4S uses the 4765 PCIe Coprocessor. It holds a secured subsystem module. The card layout can be seen at Figure 6-4.
Figure 6-4 Crypto Express4S card layout
The Crypto Express4S feature does not have external ports and does not use fiber optic or other cables. It does not use CHPIDs, but requires one slot in PCIe I/O drawer. and one PCHID for each PCIe cryptographic adapter. Removal of the feature or card zeroizes its content. The zEC12 supports a maximum of sixteen Crypto Express4S features. Access to the PCIe cryptographic adapter is controlled through the setup in the image profiles on the SE. Adapter: Though PCIe cryptographic adapters have no CHPID type and are not identified as external channels, all logical partitions in all channel subsystems have access to the adapter (up to 16 logical partitions per adapter). Having access to the adapter requires setup in the image profile for each partition. The adapter must be in the candidate list.
195
8049ch06.fm
6.7.1 Configuration rules

Each zEC12 supports up to sixteen Crypto Express4S features. Table 6-2 shows configuration information for Crypto Express4S.
Table 6-2 Crypto Express4S feature
Minimum number of orderable features for each servera Order increment above two features Maximum number of features for each server Number of PCIe cryptographic adapters for each feature (coprocessor or accelerator) Maximum number of PCIe adapters for each server Number of cryptographic domains for each PCIe adapter
b
2 1 16 1 16 16
a. The minimum initial order of Crypto Express4S features is two. After the initial order, additional Crypto Express4S can be ordered one feature at a time up to a maximum of sixteen. b. More than one partition, defined to the same CSS or to different CSSs, can use the same domain number when assigned to different PCIe cryptographic adapters.
The concept of dedicated processor does not apply to the PCIe cryptographic adapter. Whether configured as coprocessor or accelerator, the PCIe cryptographic adapter is made available to a logical partition as directed by the domain assignment and the candidate list in the logical partition image profile, regardless of the shared or dedicated status given to the CPs in the partition. When installed non-concurrently, Crypto Express4S features are assigned PCIe cryptographic adapter numbers sequentially during the power-on reset following the installation. When a Crypto Express4S feature is installed concurrently, the installation can select an out-of-sequence number from the unused range. When a Crypto Express4S feature is removed concurrently, the PCIe adapter numbers are automatically freed. The definition of domain indexes and PCIe cryptographic adapter numbers in the candidate list for each logical partition needs to be planned ahead to allow for nondisruptive changes: Operational changes can be made by using the Change LPAR Cryptographic Controls task from the Support Element, which reflects the cryptographic definitions in the image profile for the partition. With this function, adding and removing the cryptographic feature without stopping a running operating system can be done dynamically. The same usage domain index can be defined more than once across multiple logical partitions. However, the PCIe cryptographic adapter number coupled with the usage domain index specified must be unique across all active logical partitions. The same PCIe cryptographic adapter number and usage domain index combination can be defined for more than one logical partition, for example to define a configuration for backup situations. Note that only one of the logical partitions can be active at any one time. The zEC12 allows for up to 60 logical partitions to be active concurrently. Each PCI Express supports 16 domains, whether it is configured as a Crypto Express4S accelerator or a Crypto Express4S coprocessor. The server configuration must include at least four Crypto Express4S features (four PCIe adapters and 16 domains per PCIe adapter) when all 60 logical partitions require concurrent access to cryptographic functions. More Crypto Express4S features might be needed to satisfy application performance and availability requirements.
196
8049ch06.fm
6.8 Crypto Express3

The Crypto Express3 feature (FC 0864) is an optional feature and it is available only in a carry forward basis when upgrading from earlier generations to zEC12. Each feature has two Peripheral Component Interconnect Express (PCIe) cryptographic adapters. The Crypto Express3 feature occupies one I/O slot in an I/O cage or in an I/O drawer. Statement of Direction: The IBM zEnterprise EC12 is planned to be the last high-end System z server to offer support of the Crypto Express3 feature (#0864). Crypto Express3 will not be supported on future high-end System z servers as carry forward on an upgrade. Enterprises should begin migrating from the Crypto Express3 feature to the Crypto Express4S feature (#0865). Each Crypto Express3 PCI Express adapter can be configured as one of the following:
Secure coprocessor (CEX3C) for Federal Information Processing Standard (FIPS) 140-2
Level 4 certification. This includes secure key functions and it is optionally programmable to deploy additional functions and algorithms using User Defined Extension (UDX).
Accelerator (CEX3A) for acceleration of public key and private key cryptographic
operations used with Secure Sockets Layer / Transport Layer Security (SSL/TLS) processing. These modes can be configured via the Support Element, and the PCIe adapter must be configured offline to change the mode. The Crypto Express3 feature like its predecessors is designed to complement the functions of CPACF. This feature is tamper-sensing and tamper-responding. Unauthorized removal of the adapter or feature zeroizes its content. It provides dual processors operating in parallel supporting cryptographic operations with high reliability. The CEX3 uses the 4765 PCIe Coprocessor. It holds a secured subsystem module, batteries for backup power and a full-speed USB 2.0 host port available through a mini-A connector. On System z these USB ports are not used. The securely encapsulated subsystem contains two 32-bit PowerPC 405D5 RISC processors running in lock-step with cross-checking to detect malfunctions. The subsystem also includes a separate service processor used to manage self-test and firmware updates, RAM, flash memory, and battery-powered memory, cryptographic-quality random number generator, AES, DES, TDES, SHA-1, SHA-224, SHA-256, SHA-384, SHA-512 and modular-exponentiation (for example, RSA, DSA) hardware, and full-duplex DMA communications. Figure 6-5 brings the Crypto Express3 feature physical layout.
197
8049ch06.fm
Integrated/Duplicate Processors Reliably runs Common Crypto Arch (CCA)

FLASH FLASH
CPU CPU
2 boards
Separate Service ProcessorConcurrent user code update
SP DRAM DRAM CPU CPU

DRAM DRAM BBRAM BBRAM
CPU CPU CPU
SP
Tamper Tamper Detection Detection +AES
Core Functions Core Functions RTC RTC
+RSA
I/F Logic Logic
Secure Secure Boundary Boundary
Core +SHA
New Interface
USB USB Serial Serial
PCI express PCI express PCI x I/F x4 x4
Interface change to PCI-e
Figure 6-5 Crypto Express3 feature layout
The Crypto Express3 feature does not have external ports and does not use fiber optic or other cables. It does not use CHPIDs, but requires one slot in the I/O cage and one PCHID for each PCIe cryptographic adapter. Removal of the feature or card zeroizes the content. The zEC12 supports a maximum of eight Crypto Express3 features in a carry forward basis, offering a combination of up to 16 coprocessor and accelerators. Access to the PCIe cryptographic adapter is controlled through the setup in the image profiles on the SE. Adapter: Though PCIe cryptographic adapters have no CHPID type and are not identified as external channels, all logical partitions in all channel subsystems have access to the adapter (up to 16 logical partitions per adapter). Having access to the adapter requires setup in the image profile for each partition. The adapter must be in the candidate list.
6.8.1 Configuration rules

Each zEC12 supports up to eight Crypto Express3 features, which equals up to a maximum of 16 PCIe cryptographic adapters. Table 6-3 shows configuration information for Crypto Express3.
Table 6-3 Crypto Express3 feature
Minimum number of carry forward features for each server Maximum number of features for each server Number of PCIe cryptographic adapters for each feature (coprocessor or accelerator) Maximum number of PCIe adapters for each server Number of cryptographic domains for each PCIe adaptera
2 8 2 16 16
a. More than one partition, defined to the same CSS or to different CSSs, can use the same domain number when assigned to different PCIe cryptographic adapters.
198
8049ch06.fm
The concept of dedicated processor does not apply to the PCIe cryptographic adapter. Whether configured as coprocessor or accelerator, the PCIe cryptographic adapter is made available to a logical partition as directed by the domain assignment and the candidate list in the logical partition image profile, regardless of the shared or dedicated status given to the CPs in the partition. When installed non-concurrently, Crypto Express3 features are assigned PCIe cryptographic adapter numbers sequentially during the power-on reset following the installation. When a Crypto Express3 feature is installed concurrently, the installation can select an out-of-sequence number from the unused range. When a Crypto Express3 feature is removed concurrently, the PCIe adapter numbers are automatically freed. The definition of domain indexes and PCIe cryptographic adapter numbers in the candidate list for each logical partition needs to be planned ahead to allow for nondisruptive changes: Operational changes can be made by using the Change LPAR Cryptographic Controls task from the Support Element, which reflects the cryptographic definitions in the image profile for the partition. With this function, adding and removing the cryptographic feature without stopping a running operating system can be done dynamically. The same usage domain index can be defined more than once across multiple logical partitions. However, the PCIe cryptographic adapter number coupled with the usage domain index specified must be unique across all active logical partitions. The same PCIe cryptographic adapter number and usage domain index combination can be defined for more than one logical partition, for example to define a configuration for backup situations. Note that only one of the logical partitions can be active at any one time. The zEC12 allows for up to 60 logical partitions to be active concurrently. Each PCI Express supports 16 domains, whether it is configured as a Crypto Express3 accelerator or a Crypto Express3 coprocessor. The server configuration must include at least two Crypto Express3 features (four PCIe adapters and 16 domains per PCIe adapter) when all 60 logical partitions require concurrent access to cryptographic functions. More Crypto Express3 features might be needed to satisfy application performance and availability requirements.
6.9 Tasks performed by PCIe Crypto Express

The Crypto Express features running at zEC12, supports all cryptographic functions introduced on zEnterprise CPC: Expanded key support for AES algorithm: CCA supports the Advanced Encryption Standard (AES) algorithm to allow the use of AES keys to encrypt data. Expanded key support for AES adds a framework to support a much broader range of application areas, and lays the groundwork for future use of AES in areas where standards and customer applications are expected to evolve. As stronger algorithms and longer keys become increasingly common, security requirements dictate that these keys must be wrapped using key encrypting keys (KEKs) of sufficient strength. This feature adds support for AES key encrypting keys. These AES wrapping keys have adequate strength to protect other AES keys for transport or storage. This support introduced AES key types that use the variable length key token. The supported key types are EXPORTER, IMPORTER, and for use in the encryption and decryption services, CIPHER. Enhanced ANSI TR-31 interoperable secure key exchange:
199
8049ch06.fm
ANSI TR-31 defines a method of cryptographically protecting Triple Data Encryption Standard (TDES) cryptographic keys and their associated usage attributes. The TR-31 method complies with the security requirements of the ANSI X9.24 Part 1 standard, although use of TR-31 is not required in order to comply with that standard. CCA has added functions that can be used to import and export CCA TDES keys in TR-31 formats. These functions are designed primarily as a secure method of wrapping TDES keys for improved and more secure key interchange between CCA and non-CCA devices and systems. PIN block decimalization table protection: To help avoid a decimalization table attack to learn a personal identification number (PIN), a solution is now available in the CCA to thwart this attack by protecting the decimalization table from manipulation. PINs are most often used for automated teller machines (ATMs) but are increasingly used at point-of sale, for debit and credit cards. ANSI X9.8 PIN security: This function facilitates compliance with the processing requirements defined in the new version of the ANSI X9.8 and ISO 9564 PIN Security Standards and provides added security for transactions that require Personal Identification Numbers (PINs). Enhanced CCA key wrapping to comply with ANSI X9.24-1 key bundling requirements: This support permits that Common Cryptographic Architecture (CCA) key token wrapping method uses Cipher Block Chaining (CBC) mode in combination with other techniques to satisfy the key bundle compliance requirements in standards including ANSI X9.24-1 and the recently published Payment Card Industry Hardware Security Module (PCI HSM) standard. Secure key HMAC (Keyed-Hash Message Authentication Code): HMAC is a method for computing a message authentication code using a secret key and a secure hash function. It is defined in the standard FIPS (Federal Information Processing Standard) 198, The Keyed-Hash Message Authentication Code (HMAC). The CCA function supports HMAC using SHA-1, SHA-224, SHA-256, SHA-384, and SHA-512 hash algorithms. The HMAC keys are variable-length and are securely encrypted so that their values are protected. This Crypto function is supported by z/OS, z/VM and Linux on System z.
6.9.1 PCIe Crypto Express as a CCA coprocessor

The PCIe Crypto Express coprocessors enable the user to perform the following tasks: Encrypt and decrypt data by using secret-key algorithms; triple-length key DES and double-length key DES as well as AES algorithms are supported Generate, install, and distribute cryptographic keys securely by using both public and secret-key cryptographic methods generate, verify, and translate personal identification numbers (PINs) Crypto Express coprocessors support 13 through 19-digit personal account numbers (PANs) Ensure the integrity of data by using message authentication codes (MACs), hashing algorithms, and Rivest-Shamir-Adelman (RSA) public key algorithm (PKA) digital signatures, as well as Elliptic Curve Cryptography (ECC) digital signatures The Crypto Express coprocessors also provide the functions listed for the Crypto Express accelerator, however, with a lower performance than the Crypto Express accelerator can provide.
200
8049ch06.fm
Three methods of master key entry are provided by Integrated Cryptographic Service Facility (ICSF) for the Crypto Express feature coprocessors: A pass-phrase initialization method, which generates and enters all master keys that are necessary to fully enable the cryptographic system in a minimal number of steps A simplified master key entry procedure provided through a series of Clear Master Key Entry panels from a TSO terminal A Trusted Key Entry (TKE) workstation, which is available as an optional feature in enterprises that require enhanced key-entry security Linux on System z also permits the master key entry through panels or through the TKE workstation. The security-relevant portion of the cryptographic functions is performed inside the secure physical boundary of a tamper-resistant card. Master keys and other security-relevant information are also maintained inside this secure boundary. The Processor Resource/Systems Manager (PR/SM) fully supports the Crypto Express coprocessor features to establish a logically partitioned environment on which multiple logical partitions can use the cryptographic functions. A 128-bit data-protection symmetric master key, a 256-bit AES master key, a 256-bit ECC master key, and one 192-bit public key algorithm (PKA) master key are provided for each of 16 cryptographic domains that a cryptographic adapter can serve. Use the dynamic addition or deletion of a logical partition name to rename a logical partition. Its name can be changed from NAME1 to * (single asterisk) and then changed again from * to NAME2. The logical partition number and MIF ID are retained across the logical partition name change. The master keys in the Crypto Express feature coprocessor that were associated with the old logical partition NAME1 are retained. No explicit action is taken against a cryptographic component for this dynamic change. Coprocessors: Cryptographic coprocessors are not tied to logical partition numbers or MIF IDs. They are set up with PCIe adapter numbers and domain indices that are defined in the partition image profile. The customer can dynamically configure them to a partition and change or clear them when needed.
6.9.2 PCIe Crypto Express as an EP11 coprocessor

The Crypto Express4S card configured in Secure IBM Enterprise PKCS #11 (EP11) coprocessor mode provides the PKCS #11 secure key support. Prior to EP11, ICSF PKCS #11 implementation only supported clear keys. In EP11, keys now can be generated and securely wrapped under the EP11 Master Key. The secure keys never leave the secure coprocessor boundary unencrypted. Secure IBM Enterprise PKCS #11 (EP11) coprocessor perform the following tasks: Encrypt and decrypt (AES, DES, TDES, RSA) Sign and verify (DSA, RSA, ECDSA) Generate key and key pairs (DES, AES, DSA, ECC, RSA) HMAC (SHA1, SHA224, SHA256, SHA384, SHA512) Digest (SHA1, SHA224, SHA256, SHA384, SHA512) Wrap and unwrap keys Random Number Generation Get mechanism list and info Attribute values
201
8049ch06.fm
The function extension capability through UDX is not available to the EP11. When Crypto Express4S is defined in EP11 mode, the Trusted Key Entry (TKE) workstation is required to manage the Crypto Express4S feature.
6.9.3 PCIe Crypto Express as an Accelerator

The Crypto Express accelerator is a coprocessor that is reconfigured by the installation process so that it uses only a subset of the coprocessor functions at a higher speed. Note the following information about the reconfiguration: It is done through the Support Element. It is done at the PCIe cryptographic adapter level. A Crypto Express3 feature can host a coprocessor and an accelerator, two coprocessors, or two accelerators. It works both ways, from coprocessor to accelerator and from accelerator to coprocessor. Master keys in the coprocessor domain can be optionally preserved when it is reconfigured to be an accelerator. Reconfiguration is disruptive to coprocessor and accelerator operations. The coprocessor or accelerator must be deactivated before engaging the reconfiguration. FIPS 140-2 certification is not relevant to the accelerator because it operates with clear keys only. The function extension capability through UDX is not available to the accelerator. The functions that remain available when Crypto Express feature is configured as an accelerator are used for the acceleration of modular arithmetic operations (that is, the RSA cryptographic operations used with the SSL/TLS protocol), as follows: PKA Decrypt (CSNDPKD), with PKCS-1.2 formatting PKA Encrypt (CSNDPKE), with zero-pad formatting Digital Signature Verify The RSA encryption and decryption functions support key lengths of 512 bit to 4,096 bit, in the Modulus Exponent (ME) and Chinese Remainder Theorem (CRT) formats.
6.9.4 IBM Common Cryptographic Architecture (CCA) Enhancements

A new set of cryptographic functions and callable services are provided by IBM CCA Licensed Internal Code (LIC) to enhance the functions designed specifically to secure financial transactions and keys. These functions require ICSF FMID HCR77A0 and Secure IBM CCA coprocessor mode. Improved wrapping key strength In order to comply with cryptographic standards, including ANSI X9.24 Part 1 and PCI-HSM, a key must not be wrapped with a key weaker than itself. Many CCA verbs allow the customer to select the key wrapping key. With this release, CCA will allow the customer to configure the coprocessor to ensure their system meets these key wrapping requirements. It can be configured to respond in one of three ways when a key is wrapped with a weaker key: Ignore weak wrapping Complete the requested operation but return a warning message Prohibit weak wrapping altogether. DUKPT for Message Authentication Code (MAC) and encryption keys
202
8049ch06.fm
Derived Unique Key Per Transaction (DUKPT) is defined in the ANSI X9.24 Part 1 standard. It provides a method in which a separate key is used for each transaction or other message sent from a device, so that an attacker who is able to discover the value of a key would only be able to gain information about a single transaction and not about any that preceded it or which follow it. The keys are derived from a base key that is initially loaded into the device, but then erased as soon as the first keys are derived from it. Those keys, in turn, are erased as subsequent keys are derived. The original definition of DUKPT only allowed derivation of keys to be used in encryption of personal identification number (PIN) blocks. The purpose was to protect PINs that were entered at a point of sale (POS) device and then sent to a host system for verification. Recent versions of X9.24 Part 1 expanded this so that DUKPT can also be used to derive keys for MAC generation and verification, and for data encryption and decryption. Three separate variations of the DUKPT key derivation process are used so that there is key separation between the keys derived for PIN, MAC, and encryption purposes. Secure Cipher Text Translate2 (CTT2) CTT2 is a new data encryption service which will take as input data that is encrypted with one key and return the same data encrypted under a different key. This verb has the advantage that it provides the ability to securely change the encryption key for cipher text without exposing the intermediate plain text. The decryption of data and reencryption of data happens entirely inside the secure module on the Crypto Express feature. Compliance with new random number generation standards The standards defining acceptable methods for generating random numbers have been enhanced to include improved security properties. The Crypto Express coprocessor function has been updated to support methods compliant with these new standards. In this release, the random number generation in the Crypto Express feature when defined as a coprocessor conforms to the Deterministic Random Bit Generator (DRBG) requirements defined in NIST Special Publication 800-90/90A, using the SHA-256 based DBRG mechanism. The methods in these NIST standards supersede those previously defined in NIST FIPS 186-2, ANSI X9.31 and ANSI X9.62. With these improvements, customer applications can help to meet the timeline outlined in Chapter 4 of NIST SP800-131 for switching to the new methods and ceasing use of the old methods. EMV enhancements for applications supporting American Express cards Two changes have been made to the CCA application programming interfaces (APIs) to help improve support of EMV card applications that support American Express cards. The Transaction_Validation verb is used to generate and verify American Express card security codes (CSCs). This release adds support for the American Express CSC version 2.0 algorithm used by contact and contactless cards. The PIN_Change/Unblock verb is used for PIN maintenance. It prepares an encrypted message portion for communicating an original or replacement PIN for an EMV smart card. The verb embeds the PINs in an encrypted PIN block from information that is supplied. With this CCA enhancement, PIN_Change/Unblock adds support for the message format used to change or unblock the PIN on American Express EMV cards.
6.10 TKE workstation feature

The TKE, Trusted Key Entry, workstation is an optional feature that offers key management functions. The TKE workstation, feature code 0841, contains a combination of hardware and software. Included with the system unit are a mouse, keyboard, flat panel display, PCIe adapter and a writable USB media to install TKE Licensed Internal Code (LIC). The TKE
203
8049ch06.fm
workstation feature code 0841 was the first to have the 4765 crypto card installed. TKE LIC V7.2 requires CEX4 or CEX3 and TKE workstation feature code 0841. Adapters: The TKE workstation supports Ethernet adapters only to connect to a LAN. A TKE workstation is part of a customized solution for using the Integrated Cryptographic Service Facility for z/OS program product (ICSF for z/OS) or the Linux for System z that provide a basic key management system for cryptographic keys of a zEC12 that has Crypto Express features installed and that is configured for using DES, AES, ECC and PKA cryptographic keys. The TKE provides a secure, remote and flexible method of providing Master Key Part Entry, and to remotely manage PCIe Cryptographic Coprocessor(s). The cryptographic functions on the TKE are performed by one PCIe Cryptographic Coprocessor. The TKE workstation communicates with the System z server using a TCP/IP connection. The TKE workstation is available with Ethernet LAN connectivity only. Up to ten TKE workstations can be ordered. TKE feature number 0841 can be used to control the zEC12 and it can also be used to control z196, z114, z10 EC, z10 BC, z9 EC, z9 BC, z990, and z890 servers.
TKE 7.0 Licensed Internal Code (LIC)

The TKE workstation feature code 0841 along with LIC 7.0 offers a significant number of enhancements: ECC Master Key Support: ECC keys will be protected using a new ECC master key (256-bit AES key). From the TKE, administrators can generate key material, load or clear the new ECC master key register, or clear the old ECC master key register. The ECC key material can be stored on the TKE or on a smart card. CBC Default Settings Support: The TKE provides function that allows the TKE user to set the default key wrapping method used by the host crypto module. TKE Audit Record Upload Configuration Utility Support: The TKE Audit Record Upload Configuration Utility allows Trusted Key Entry (TKE) workstation audit records to be sent to a System z host and saved on the host as z/OS System Management Facilities (SMF) records. The SMF records have a record type of 82 (ICSF) and a subtype of 29. TKE workstation audit records are sent to the same TKE Host Transaction Program that is used for Trusted Key Entry operations. USB Flash Memory Drive Support: The TKE workstation now supports a USB flash memory drive as a removable media device. When a TKE application displays media choices, the application allows you to choose a USB flash memory drive if the IBM supported drive is plugged into a USB port on the TKE and it has been formatted for the specified operation. Stronger Pin Strength Support: TKE smart cards created on TKE 7.0 require a 6-digit pin rather than a 4-digit pin. TKE smart cards that were created prior to TKE 7.0 will continue to use 4-digit pins and will work on TKE 7.0 without changes. You can take advantage of the stronger pin strength by initializing new TKE smart cards and copying the data from the old TKE smart cards to the new TKE smart cards. Stronger password requirements for TKE Passphrase User Profile Support: New rules are required for the passphrase used for passphrase logon to the TKE workstation crypto adapter. The passphrase must meet the following requirements: Be 8 to 64 characters long
204
8049ch06.fm
Contain at least 2 numeric and 2 non-numeric characters Not contain the user ID These rules are enforced when you define a new user profile for passphrase logon, or when you change the passphrase for an existing profile. Your current passphrases will continue to work. Simplified TKE usability with Crypto Express migration wizard: A wizard is now available to allow users to collect data, including key material, from a Crypto Express coprocessor and migrate the data to a different Crypto Express coprocessor. The target Crypto Express coprocessor must have the same or greater capabilities. This wizard is an aid to help facilitate migration from Crypto Express2 to Crypto Express3. Crypto Express2 is not supported on zEC12 neither on z196 and z114. The following benefits are obtained when using this wizard: Reduces migration steps, thereby minimizing user errors Minimizes the number of user clicks Significantly reduces migration task duration

The TKE workstation feature code 0841 along with LIC 7.1 offers additional enhancements: New access control support for all TKE applications: Every TKE application and the ability to create and manage crypto module and domain groups now require the TKE local cryptographic adapter profile to have explicit access to the TKE application or function that the user wants to run. This was done to provide more control of what functions the TKE users are allowed to perform. New migration utility: During a migration from a lower release of TKE to TKE 7.1 LIC, it will be necessary to add access control points to the existing roles. The new access control points can be added through the new Migrate Roles Utility or by manually updating each role through the Cryptographic Node Management Utility. The IBM-supplied roles created for TKE 7.1 LIC have all of the access control points needed to perform the functions they were permitted to use in TKE releases prior to TKE 7.1 LIC. Single process for loading an entire key: The TKE now has a wizard-like feature that takes users through the entire key loading procedure for a master or operational key. The feature preserves all of the existing separation of duties and authority requirements for clearing, loading key parts, and completing a key. The procedure saves time, by walking users through the key loading procedure. However, this feature does not reduce the number of people it takes to perform the key load procedure. Single process for generating multiple key parts of the same type: The TKE now has a wizard-like feature that allows a user to generate more than one key part at a time. The procedure saves time because the user has to start the process only one time, and the TKE efficiently generates the desired number of key parts. AES operational key support: CCA V4.2 for the Crypto Express feature includes three new AES operational key types. From the TKE, users can load and manage the new AES EXPORTER, IMPORTER, and CIPHER operational keys from the TKE workstation crypto module notebook. Decimalization table support:
205
8049ch06.fm
CCA V4.2 for the Crypto Express feature includes support for 100 decimalization tables for each domain on a Crypto Express feature. From the TKE, users can manage the decimalization tables on the Crypto Express feature from the TKE workstation crypto module notebook. Users can manage the tables for a specific domain or manage the tables of a set of domains if they are using the TKE workstation Domain Grouping function. Host cryptographic module status support: From the TKE workstation crypto module notebook, users will be able to display the current status of the host cryptographic module that is being managed. If they view the Crypto Express feature module information from a crypto module group or a domain group, they will see only the status of the group's master module. Display of active IDs on the TKE console: A user can be logged onto the TKE workstation in privileged access mode. In addition, the user can be signed onto the TKE workstation's local cryptographic adapter. If a user is signed on in privileged access mode, that ID is shown on the TKE console. With this new support, both the privileged access mode ID and the TKE local cryptographic adapter ID will be displayed on the TKE console. Increased number of key parts on smart card: If a TKE smart card is initialized on a TKE workstation with a 7.1 level of LIC, it will be able to hold up to 50 key parts. Previously, TKE smart cards could hold only 10 key parts. Use of ECDH to derive shared secret: When the TKE workstation with a 7.1 level of LIC exchanges encrypted material with a Crypto Express card at CCA level V4.2, Elliptic Curve Diffie-Hellman (ECDH) is used to derive the shared secret. This increases the strength of the transport key used to encrypt the material.

The TKE workstation feature code 0841 along with LIC 7.2 offers even additional enhancements: Support for the Crypto Express4S feature when configured as an EP11 coprocessor: The TKE workstation is required to manage a Crypto Express4S feature that is configured as an EP11 coprocessor. Allow domain grouping between Crypto Express4S features that are defined only as EP11. The TKE smart card reader (#0885) is mandatory. EP11 requires the use of the new smart card part 74Y0551 (#0884, #0885). The new smart card can be used for any of the 6 types of smart cards used on TKE. Two items must be placed on the new smart cards: Master Key Material: The Crypto Express4S feature has master keys for each domain. The key material must be placed on a smart card before the key material can be loaded. Administrator Signature Keys: When commands are sent to the Crypto Express4S feature, they must be signed by administrators. Administrator signature keys must be on smart cards. Support for the Crypto Express4S feature when configured as a CCA coprocessor: Crypto Express4S (defined as a CCA coprocessor) is managed the same way as any other CCA-configured coprocessors. A Crypto Express4S can be in the same crypto module group or domain group as a Crypto Express4S, Crypto Express3, and Crypto Express2 feature. Support for 24-byte Data Encryption Standard (DES) master keys:
206
8049ch06.fm
CCA supports both 16-byte and 24-byte DES master keys. The DES master key length for a domain is determined by a new domain control bit that can be managed using the TKE. Two Access Control Points (ACP) allow the user to choose between warning or prohibiting the loading of a weak Master Key. The latest CCA version is required. Protect generated RSA keys with AES importer keys TKE generated RSA keys are encrypted by AES keys before being sent to System z. It allows the generation for 2046-bit and 4096-bit RSA keys for target crypto card use. New Data Encryption Standard (DES) operational keys: Four new DES operational keys can be managed from the TKE workstation (#0841). The key types are: CIPHERXI CIPHERXL CIPHERXO DUKPT-KEYGENKY
The new keys are managed the same way as any other DES operational key. New Advanced Encryption Standard (AES) CIPHER key attribute: A new attribute, key can be used for data translate only can now be specified when creating an AES CIPHER operational key part. Allow creation of corresponding keys: There are some cases where operational keys need to be loaded to different host systems to serve an opposite purpose. For example, one host system needs an exporter key encrypting key; another system needs a corresponding importer key encrypting key with the same value. The TKE workstation now allows nine types of key material to be used for creating a corresponding key. Support for four smart card readers: The TKE workstation supports two, three, or four smart card readers when smart cards are being used. The additional readers were added to help reduce the number of smart card swaps needed while managing EP11 configured coprocessors. EP11 can be managed with only two smart card readers. CCA configured coprocessors can be managed with three or four smart card readers.
6.10.1 Logical partition, TKE host, and TKE target

If one or more logical partitions are configured to use Crypto Express coprocessors, the TKE workstation can be used to manage DES, AES, ECC, and PKA master keys for all cryptographic domains of each Crypto Express coprocessor feature assigned to the logical partitions defined to the TKE workstation. Each logical partition in the same system using a domain managed through a TKE workstation connection is either a TKE host or a TKE target. A logical partition with a TCP/IP connection to the TKE is referred to as TKE host. All other partitions are TKE targets. The cryptographic controls as set for a logical partition through the Support Element determine whether the workstation is a TKE host or TKE target.
6.10.2 Optional smart card reader

Adding an optional smart card reader (FC 0885) to the TKE workstation is possible. One (1) feature code 0885 includes two Smart Card Readers, two cables to connect to the TKE
207
8049ch06.fm
workstation, and 20 smart cards. The reader supports the use of smart cards that contain an embedded microprocessor and associated memory for data storage that can contain the keys to be loaded into the Crypto Express features. Access to and use of confidential data on the smart card is protected by a user-defined personal identification number (PIN). Up to 990 additional smart cards can be ordered for backup. The additional smart card feature code is FC 0884, and when one feature code is ordered, a quantity of ten smart cards are shipped. The order increment is one up to 99 (990 blank smart cards).
6.11 Cryptographic functions comparison

Table 6-4 lists functions or attributes on zEC12 of the three cryptographic hardware features. In the table, X indicates that the function or attribute is supported.
Table 6-4 Cryptographic functions on zEC12 Functions or attributes
CPACF
a
CEX4Ca X X X X X
CEX4Pa X X X
CEX4Aa X X X X X -
CEX3Cab X X X X X
CEX3Aab X X X X X -
Supports z/OS applications using ICSF Supports Linux on System z CCA applications Encryption and decryption using secret-key algorithm Provides highest SSL/TLS handshake performance Supports SSL/TLS functions Provides highest symmetric (clear key) encryption performance Provides highest asymmetric (clear key) encryption performance Provides highest asymmetric (encrypted key) encryption performance Disruptive process to enable Requires IOCDS definition Uses CHPID numbers Uses PCHIDs Requires CPACF enablement (FC 3863) Requires ICSF to be active Offers user programming function (UDX) Usable for data privacy: encryption and decryption processing
X X X X -
Note c Xd
Note c Xd Xe X X
Note c Xd Xe X -
Note c Xd Xe X X X
Note c Xd Xe X -
Xe X
Xe X X X
208
8049ch06.fm
Functions or attributes
CPACF
a
CEX4Ca X X X X X X X X X X X
CEX4Pa X X X X X X X X X X
CEX4Aa X Xf X -
CEX3Cab X X X X X X X X X X X
CEX3Aab X Xf X X -
Usable for data integrity: hashing and message authentication Usable for financial processes and key management operations Crypto performance RMF monitoring Requires system master keys to be loaded System (master) key storage Retained key storage Tamper-resistant hardware packaging Designed for FIPS 140-2 Level 4 certification Supports Linux applications doing SSL handshakes RSA functions High performance SHA-1 and SHA2 Clear key DES or triple DES Advanced Encryption Standard (AES) for 128-bit, 192-bit, and 256-bit keys Pseudorandom number generator (PRNG) Clear key RSA Europay Mastercard VISA (EMV) support Public Key Decrypt (PKD) support for Zero-Pad option for clear RSA private keys Public Key Encrypt (PKE) support for MRP function Remote loading of initial keys in ATM Improved key exchange with non CCA system ISO 16609 CBC mode triple DES MAC support
X X X X
X -
X X X
X -
X X
X X X
X X
X X X X
X -
X X X X
X -
a. This requires CPACF enablement feature code 3863. b. Available only in a carry forward basis when upgrading from earlier generations to zEC12. c. To make the addition of the Crypto Express features nondisruptive, the logical partition must be predefined with the appropriate PCI Express cryptographic adapter number selected in its candidate list in the partition image profile. d. One PCHID is required for each PCIe cryptographic adapter.
209
8049ch06.fm
e. This is not required for Linux if only RSA clear key operations are used. DES or triple DES encryption requires CPACF to be enabled. f. This is physically present but is not used when configured as an accelerator (clear key only).
6.12 Software support

The software support levels are listed in 8.4, Cryptographic support on page 288.
210
8049ch07.fm
Chapter 7.
zBX zEnterprise BladeCenter Extension Model 003

IBM has extended the mainframe ecosystem by bringing select IBM BladeCenter product lines under the same management umbrella. This is called the zEnterprise BladeCenter Extension (zBX) Model 003 and the common management umbrella is the IBM Unified Resource Manager. The zBX brings the computing capacity of systems in blade form-factor to the IBM zEnterprise System. It is designed to provide a redundant hardware infrastructure that supports the multi-platform environment of the zEC12 in a seamless integrated way. Also key to this hybrid environment is the Unified Resource Manager, which helps deliver end-to-end infrastructure virtualization and management, as well as the ability to optimize multi-platform technology deployment according to complex workload requirements. For more information about the Unified Resource Manager, see Chapter 12., Hardware Management Console and Support Element on page 397 and IBM zEnterprise Unified Resource Manager, SG24-7921. This chapter introduces the zEnterprise BladeCenter Extension (zBX) Model 003 and describes its hardware components. It also explains the basic concepts and building blocks for zBX connectivity. You can use the information in this chapter for planning purposes and to help define the configurations that best fit your requirements. This chapter discusses the following topics: zBX concepts on page 212 zBX hardware description on page 212 zBX entitlements, firmware, and upgrades on page 223 zBX connectivity on page 226 zBX connectivity examples on page 237
211
8049ch07.fm
7.1 zBX concepts

The integration of System z in a hybrid infrastructure represents a new height for mainframe functionality and qualities of service. It has been rightly portrayed as a cornerstone for the IT infrastructure, especially when flexibility for rapidly changing environments is called for. zEC12 characteristics make it especially valuable for mission critical workloads. Today, most of these workloads have multi-tiered architectures that span various hardware and software platforms. However, there are differences in the qualities of service offered by the platforms. There are also various configuration procedures for their hardware and software, operational management, software servicing, failure detection and correction, and so on. These in turn require personnel with distinct skill sets, various sets of operational procedures, and an integration effort that is not trivial and, therefore, not often achieved. Failure in achieving integration translates to lack of flexibility and agility, which can impact the bottom line. IBM mainframe systems have been providing specialized hardware and fit-for-purpose (tuned to the task) computing capabilities for a long time. In addition to the machine instruction assists, another early example is the vector facility of the IBM 3090. Other such specialty hardware includes the System Assist Processor for I/O handling (that implemented the 370-XA architecture), the Coupling Facility, and the Cryptographic processors. Furthermore, all the I/O cards are specialized dedicated hardware components, with sophisticated software, that offload processing from the System z processor units (PUs). The common theme with all of these specialized hardware components is their seamless integration within the mainframe. The zBX components are also configured, managed, and serviced the same way as the other components of the System z CPC. Despite the fact that the zBX processors are not z/Architecture PUs, the zBX is in fact, handled by System z management firmware called the IBM zEnterprise Unified Resource Manager. The zBX hardware features are integrated into the mainframe ecosystem, not add-ons. System z has long been an integrated heterogeneous platform. With zBX, that integration reaches a new level. zEnterprise with its zBX infrastructure offers the possibility of running an application that spans z/OS, z/VM, z/VSE, Linux on System z, AIX, Linux on System x, and Microsoft Windows, yet have it under a single management umbrella. Also, zBX can host and integrate special purpose workload optimizers such as the WebSphere DataPower Integration Appliance XI50 for zEnterprise (DataPower XI50z).
7.2 zBX hardware description

The zBX has a machine type of 2458-003 and is exclusive to the zEC12. It is capable of hosting integrated multi-platform systems and heterogeneous workloads, with integrated advanced virtualization management. The zBX Model 003 is configured with the following key components: One to four standard 19 inch IBM 42U zEnterprise racks with required network and power infrastructure One to eight BladeCenter chassis with a combination of up to 1121 different blades Redundant infrastructure for fault tolerance and higher availability Management support through the zEC12 Hardware Management Console (HMC) and Support Element (SE)
The number of chassis and blades varies depending on the type of the blades configured within zBX. See 7.2.4, zBX blades on page 218 for more information.
212
8049ch07.fm
You can read more about zBX reliability, availability, and serviceability (RAS) in 10.6, RAS capability for zBX on page 370. The zBX can be ordered with a new zEC12 or as an MES to an existing zEC12. If a z196 controlling a zBX is upgraded to a zEC12, the controlled zBX Model 002 must be upgraded to a Model 003. Either way, the zBX is treated as an extension to a zEC12 and cannot be ordered as a standalone feature. Figure 7-1 shows a zEC12 with a maximum zBX configuration. The first rack (Rack B) in the zBX is the primary rack where one or two BladeCenter chassis and four top of rack (TOR) switches reside. The other three racks (C, D, and E) are expansion racks with one or two BladeCenter chassis each.
zEnterprise EC12
zEnterprise BladeCenter Extension

Figure 7-1 zEC12 with a maximum zBX configuration
7.2.1 zBX racks

The zBX Model 003 (2458-003) hardware is housed in up to four IBM zEnterprise racks. Each rack is an industry-standard 19", 42U high rack, and has four sidewall compartments to support installation of power distribution units (PDUs) and switches, with additional space for cable management. Figure 7-2 shows the rear view of a two rack zBX configuration, including these components: Two top-of-rack (TOR) 1000BASE-T switches (Rack B only) for the Intranode management network (INMN) Two TOR 10 GbE switches (Rack B only) for the Intraensemble data network (IEDN) Up to two BladeCenter Chassis in each rack with:
2
14 blade server slots per chassis 1 Gbps Ethernet switch modules (ESM) 10 Gbps High speed switch Ethernet (HSS) modules 8 Gbps Fibre Channel switches for connectivity to the SAN environment2 Blower modules
Client supplied FC switches are required and must support N_Port ID Virtualization (NPIV). Some FC switch vendors also require interop mode. Check the interoperability matrix for the latest details, at this website: http://www-03.ibm.com/systems/support/storage/ssic/interoperability.wss
Chapter 7. zBX zEnterprise BladeCenter Extension Model 003
213
8049ch07.fm
Power Distribution Units (PDUs)

Rear View Rack C Rack B
TOR Intranode Management Network switches TOR Intraensemble Data Netw ork switches
Power Distribution Units
One or two BladeCenter chassis per rack
BladeCenters: Blades Power supplies 10 GbE switches 8 Gbps FC switches Blowers
Figure 7-2 zBX racks - rear view with BladeCenter chassis
A zBX rack supports a maximum of two BladeCenter chassis. Each rack is designed for enhanced air flow and is shipped loaded with the initial configuration. It can be upgraded on-site. The zBX racks are shipped with lockable standard non-acoustic doors and side panels. The following optional features are also available: IBM rear door heat eXchanger (FC 0540) reduces the heat load of the zBX emitted into ambient air. The rear door heat eXchanger is an air-to-water heat exchanger that diverts heat of the zBX to chilled water (customer supplied data center infrastructure). The rear door heat eXchanger requires external conditioning units for its use. IBM acoustic door (FC 0543) can be used to reduce the noise from the zBX. Height reduction (FC 0570) reduces the rack height to 36U high and accommodates doorway openings as low as 1832 mm (72.1 inches). Order this choice if you have doorways with openings less than 1941 mm (76.4 inches) high.
7.2.2 Top of rack (TOR) switches

The four top-of-rack (TOR) switches are installed in the first rack (Rack B). Adding expansion racks (Rack C, D, and E) do not require additional TOR switches. The TOR switches are located near the top of the rack and are mounted from the rear of the rack. From top down there are two 1000BASE-T switches for the Intranode management network (INMN) and two 10 GbE switches for the Intraensemble data network (IEDN).
214
8049ch07.fm
A zBX Model 003 can only be managed by one zEC12 through the INMN connections. Each VLAN-capable 1000BASE-T switch has 48 ports. The switch ports are reserved as follows: One port for each of the two bulk power hubs (BPH) on the controlling zEC12 One port for each of the advanced management modules (AMM) and Ethernet switch modules (ESM), in each zBX BladeCenter chassis One port for each of the two IEDN 10 GbE TOR switches Two ports each for interconnecting the two switches Both switches have the same connections to the corresponding redundant components (BPH, AMM, ESM, and IEDN TOR switches) to avoid any single point of failure. Table 7-5 on page 228 shows port assignments for the 1000BASE-T TOR switches. Important: Although IBM provides a 26m cable for the INMN connection, it is best to have the zBX installed next to or near the controlling zEC12, for easy access to the zBX for service related activities or tasks. Each VLAN-capable 10 GbE TOR switch has 40 ports dedicated to the IEDN. The switch ports have the following connections: Up to 16 ports are used for connections to an HSS module (SM07 and SM09) of each BladeCenter chassis in the same zBX (as part of IEDN), to provide data paths to blades. Up to eight ports are used for OSA-Express4S 10 GbE or OSA-Express3 10 GbE (LR or SR) connections to the ensemble CPCs (as part of IEDN), to provide data paths between the ensemble CPCs and the blades in a zBX. Up to seven ports are used for zBX to zBX connections within a same ensemble (as part of the IEDN). Up to seven ports are used for the customer managed data network. Customer network connections are not part of IEDN, and cannot be managed or provisioned by the Unified Resource Manager. The Unified Resource Manager will recognize them as migration connections and provide access control from the customer network to the 10 GbE TOR switches. The management port is connected to the INMN 1000BASE-T TOR switch. Two ports are used for interconnections between two switches (as a failover path), using two Direct Attached Cables (DAC) to interconnect both switches. Figure 7-3 shows the connections of TOR switches and only the first BladeCenter chassis in frame B. For more information about the connectivity options for the INMN and the IEDN, as well as the connectivity rules, see 7.4, zBX connectivity on page 226.
215
8049ch07.fm
P06 P07
OSM
B P H A
47 01
P03 P04
45 44 46
00
<- Port# B36P INMN <- Port# B35P INMN <- Port# B32P INMN <- Port# B30P INMN
03
47
01 45 44
46
00
P04 P07
OSM
B P H B
P03 P06
08 16 41
38 39
08 16 41 38 39
01 02 S M 0 1 SM07 01 10 09 M M 0 1
S M 01 0 2
10 09 SM09
01
02
M M 0 2
03
BladeCenter chassis B10B
Figure 7-3 Graphical illustration of zEnterprise network connections
7.2.3 zBX BladeCenter chassis

Each zBX BladeCenter chassis is designed with additional components installed for high levels of resiliency. The front of a zBX BladeCenter chassis has the following components: Blade server slots: There are 14 blade server slots (BS01 to BS14) available in a zBX BladeCenter chassis. Each slot is capable of housing any zBX supported blades, with the following restrictions: Slot 14 cannot hold a double-wide blade. The DataPower XI50z blades are double-wide. Each feature takes two adjacent BladeCenter slots, so the maximum number of DataPower blades per BladeCenter is seven. The maximum number of DataPower blades per zBX is 28. Power module: The power module includes a power supply and a three-pack of fans; two of three fans are needed for a power module operation. Power module 1 and 2 (PM01 and PM02) are installed as a pair to provide a power supply for the seven blade server slots from BS01 to BS07, and power module 3 and 4 (PM03 and PM04) support the BS08 to BS14. The two different power connectors (marked with 1 and 2 in Figure 7-4 on page 217) provide power connectivity for the power modules (PM) and blade slots. PM01 and PM04 are connected to power connector 1, while PM02 and PM03 are connected to power connector 2. Thus, each slot has fully redundant power from a different power module connected to a different power connector. Figure 7-4 on page 217 shows the rear view of a zBX BladeCenter chassis.
216
8049ch07.fm
2
SM01 (Bay 1)
SM07 (Bay 7)
1
SM03 (Bay 3) MM01
Blower Module 1
Figure 7-4 zBX BladeCenter chassis rear view
The rear of a zBX BladeCenter chassis has following components: Advanced management module: The advanced management module (AMM) provides systems-management functions and keyboard/video/mouse (KVM) multiplexing for all of the blade servers in the BladeCenter unit that support keyboard/video/mouse (KVM). It controls the external keyboard, mouse, and video connections, for use by a local console, and a 10/100 Mbps Ethernet remote management connection. Use of KVM is not supported on zBX. All the required management function are available on the controlling zEC12 support element (SE) or the HMC. The management module communicates with all components in the BladeCenter unit, detecting their presence, reporting their status, and sending alerts for error conditions when required. The service processor in the management module communicates with the service processor (iMM) in each blade server to support features such as blade server power-on requests, error and event reporting, keyboard/video/mouse (KVM) requests, and requests to use the BladeCenter shared media tray. The AMMs are connected to the INMN through the 1000BASE-T TOR switches. Thus, firmware and configuration for the AMM is controlled by the SE of the controlling zEC12, together with all service management and reporting function of AMMs. Two AMMs (MM01 and MM02) are installed in the zBX BladeCenter chassis. Only one AMM has primary control of the chassis (it is active); the second module is in passive (standby) mode. If the active module fails, the second module is automatically enabled with all of the configuration settings of the primary module. Ethernet switch module: Two 1000BASE-T (1 Gbps) Ethernet switch modules (SM01 and SM02) are installed in switch bays 1 and 2 in the chassis. Each ESM has 14 internal full-duplex Gigabit ports,
SM02 (Bay 2)
SM04 (Bay 4)
MM02
Blower Module 2
SM09 (Bay 9)
217
8049ch07.fm
one connected to each of the blade servers in the BladeCenter chassis, two internal full-duplex 10/100 Mbps ports connected to the AMM modules, and six 1000BASE-T copper RJ-45 connections for INMN connections to the TOR 1000BASE-T switches. The ESM port 01 is connected to one of the 1000BASE-T TOR switches. As part of the INMN, configuration and firmware of ESM is controlled by the controlling zEC12 Support Element (SE). High speed switch module: Two high-speed switch modules (SM07 and SM09) are installed to the switch bay 7 and 9. The HSS modules provide 10 GbE uplinks to the 10 GbE TOR switches and 10 GbE downlinks to the blades in the chassis. Port 01 and 02 are connected to one of the 10 GbE TOR switches. Port 09 and 10 are used to interconnect HSS in bay 7 and 9 as a redundant failover path. 8 Gbps Fibre Channel switch module: Two 8 Gbps Fibre Channel (FC) switches (SM03 and SM04) are installed in switch bays 3 and 4. Each switch has 14 internal ports reserved for the blade servers in the chassis, and six external Fibre Channel ports to provide connectivity to SAN environment. Blower module: There are two hot swap blower modules installed. The blower speeds vary depending on the ambient air temperature at the front of the BladeCenter unit and the temperature of internal BladeCenter components. If a blower fails, the remaining blowers run full speed. BladeCenter mid-plane fabric connections: The BladeCenter mid-plane provides redundant power, control, and data connections to a blade server by internally routed chassis components (power modules, AMMs, switch modules, media tray) to connectors in a blade server slot. There are six connectors in a blade server slot on the mid-plane, from top to bottom: Top 1X fabric connects blade to MM01, SM01, and SM03 Power connector from power module 1 (blade server slot 1 to 7) or power module 3 (blade server slot 8 to 14) Top 4X fabric connects blade to SM07 Bottom 4X fabric connects blade to SM09 Bottom 1X fabric connects blade to MM02, SM02 and SM04 Power connector from power module 2 (blade server slot 1 to 7) or power module 4 (blade server slot 8 to 14) Thus, each blade server has redundant power, data, and control links from separate components.
7.2.4 zBX blades

The zBX Model 003 supports the following blade types: POWER7 blades: Three configurations of POWER blades are supported, depending on their memory sizes (see Table 7-1 on page 219). The number of blades can be from 1 to 112. IBM WebSphere DataPower XI50 for zEnterprise blades: Up to 28 IBM WebSphere DataPower XI50 for zEnterprise (DataPower XI50z) blades are supported. These blades are double-wide (each one occupies two blade server slots). IBM HX5 System x blades:
218
8049ch07.fm
Up to 56 IBM System x HX5 blades are supported. All zBX blades are connected to AMMs and ESMs through the chassis mid-plane. The AMMs are connected to the INMN.
zBX blade expansion cards

Each zBX blade has two PCI Express connectors, combination input output vertical (CIOv), and combination form factor horizontal (CFFh). I/O expansion cards are attached to these connectors and connected to the mid-plane fabric connectors. Thus a zBX blade can expand its I/O connectivity through the mid-plane to the high speed switches and switch modules in the chassis. Depending on the blade type, 10 GbE CFFh expansion cards and 8 Gbps Fibre Channel CIOv expansion cards provide I/O connectivity to the IEDN, the INMN, or client supplied FC-attached storage.
POWER7 blade
The POWER7 blade (Table 7-1) is a single width blade, which includes a POWER7 processor, up to 16 DIMMs, and an HDD (Hard Disk Drive). The POWER7 blade supports 10 GbE connections to IEDN, and 8 Gbps FC connections to client provided Fibre Channel storage through the Fibre Channel (FC) switches (SM03 and SM04) in the chassis. The POWER7 blade is loosely integrated to a zBX, so that you can acquire supported blades through existing channels from IBM. The primary HMC and SE of the controlling zEC12 perform entitlement management for installed POWER7 blades on a one-blade basis. PowerVM Enterprise Edition must be ordered with each POWER blade. AIX 5.3, AIX 6.1 and AIX 7.1 and subsequent releases are supported.
Table 7-1 Supported configuration of POWER7 blades Feature FC Config 1 quantity Config 2 quantity Config 3 quantity
Processor (3.0GHz@150W) 8 GB Memory 16 GB Memory Internal HDD (300GB) CFFh 10 GbE expansion CIOv 8 Gb FC expansion
8412 8208 8209 8274 8275 8242
8 4 0 1 1 1
8 8 0 1 1 1
8 0 8 1 1 1
DataPower XI50z blades

The IBM WebSphere DataPower Integration Appliance XI50 for zEnterprise (DataPower XI50z), integrated into the zBX, is a high-performance hardware appliance that offers these benefits: Provides fast and flexible integration with any-to-any transformation between disparate message formats with integrated message-level security and superior performance. Provides web services enablement for core System z applications to enable web-based workloads. As a multifunctional appliance DataPower XI50z can help provide multiple levels of XML optimization, streamline and secure valuable service-oriented architecture (SOA) applications.
219
8049ch07.fm
Enables SOA and XML applications with System z web services for seamless integration of distributed and System z platforms. It can help to simplify, govern, and enhance the network security for XML and web services. Provides drop-in integration for heterogeneous environments by enabling core Enterprise Service Bus (ESB) functionality, including routing, bridging, transformation, and event handling. Offers standards-based, centralized System z governance, and extreme reliability through integrated operational controls, call home, and integration with RACF security through a secured private network. The zBX provides additional benefits to the DataPower appliance environment in these areas: Blade hardware management: Improved cooling and power management controls, includes cooling of the frame and energy monitoring and management of the DataPower blades Virtual network provisioning Call-home for current and expected problems. Hardware Management Console integration: Single view showing the System z environment together with the DataPower blades in an overall hardware operational perspective Group GUI operations for functions supported on HMC such as activate or deactivate blades Improved availability: Guided placement of blades to optimize built-in redundancy in all components at the rack, BladeCenter, and HMC levels, including top of rack switch, ESM switches, and physical network. Detection and reporting by the HMC/SE on appliance failures. The HMC/SE can also be used to re-cycle the DataPower appliance. Networking: Virtual network provisioning Enforced isolation of network traffic by VLAN support 10 Gbps end-to-end network infrastructure Built-in network redundancy Network protection vie IEDN, possibly obviating any perceived need for encryption of flows between DataPower and the target back-end System z server
Monitoring and reporting: Monitoring and reporting of DataPower hardware health and degraded operation by HMC Monitoring of all hardware, call-home, and auto-parts replacement Consolidation and integration of DataPower hardware problem reporting with other problems reported in zBX System z value: Simplified ordering of the DataPower appliance by System z allows the proper blade infrastructure to be transparently ordered. Simplified upgrades keep MES history so the upgrades flow based on what is installed. System z service on the zBX and DataPower blade with a single point of service. The DataPower appliance becomes part of the data center and comes under data center control.
220
8049ch07.fm
In addition, although not specific to the zBX environment, dynamic load balancing to DataPower appliances is available using the z/OS Communications Server Sysplex Distributor.
Configuration
The DataPower XI50z is a double-wide IBM HS22 blade. Each one takes two BladeCenter slots, so the maximum number of DataPower blades per BladeCenter is seven, and the maximum number of DataPower blades per zBX is 28. It can coexist with POWER7 blades and with IBM BladeCenter HX5 blades in the same zBX BladeCenter. DataPower XI50z blades are configured and ordered as zBX (machine type 2458-003) features, but have their own machine type (2462-4BX). The DataPower XI50z with DataPower expansion unit has the following specification: 2.13GHz 2x quad core processors 8M cache 3 x 4Gb DIMMs (12Gb Memory) 4Gb USB Flash Key that contains the DataPower XI50z firmware load 2 x 300GB HDDs used by the customer for logging, storing style sheets, and storing XML files. The hard disk array consists of two hard disk drives in a RAID-1 (mirrored) configuration. Broadcom BCM5709S x2 with TOE (integrated on planar) BPE4 Expansion Unit. It is sealed FRU with one-way tamper-proof screws (contains the crypto for secure SOA applications). XG5 accelerator PCIe card CN1620 Cavium crypto PCIe card Dual 10Gb Ethernet card
2462 Model 4BX (DataPower XI50z)

The 2462 Model 4BX is designed to work together with the 2458 Model 003 (zBX). It is functionally equivalent to an IBM 4195-4BX with similar feature codes. The IBM 2462 Model 4BX is ordered through certain feature codes for the 2458-003. Configuring the IBM 2458 Model 003 with feature code 0611 (DataPower XI50z), a machine type IBM 2462 Model 4BX is ordered for each configured feature code. It requires Software PID 5765-G84. Software Maintenance Agreement (SWMA) must be active for the IBM software that runs on the DataPower XI50z in order to obtain service or other support for the IBM software. Failure to maintain SWMA will result in the customer not being able to obtain service for the IBM software, even if the DataPower XI50z is under warranty or post-warranty IBM hardware maintenance service contract. DataPower XI50z license entitlements include these: DataPower Basic Enablement (feature code 0650) Tivoli Access Manager (TAM) (feature code 0651) TIBCO (feature code 0652). Database Connectivity (DTB) (feature code 0653) Application Optimization (AO) (feature code 0654) Month Indicator (feature code 0660) Day Indicator (feature code 0661) Hour Indicator (feature code 0662) Minute Indicator (feature code 0663) 5765-G84 IBM WebSphere DataPower Integration Blade XI50B feature code description:
221
8049ch07.fm
0001 License with 1 year SWMA 0002 Option for TIBCO 0003 Option for Application Optimization 0004 Option for Database Connectivity 0005 Option for Tivoli Access Manager Every IBM 2462 Model 4BX includes feature codes 0001, 0003, and 0005 (they are optional on DataPower XI50B). Optional Software feature codes 0002 and 0004 are required if FC 0652 TIBCO or FC 0653 Database Connectivity are ordered. The TIBCO option (FC 0002) lets you extend the DataPower XI50z so you can send and receive messages from TIBCO Enterprise Message Service (EMS). Option for Database Connectivity (FC 0004) lets you extend the DataPower XI50z to read and write data from relational databases such as IBM DB2, Oracle, Sybase, and Microsoft SQL Server. For software PID number 5765-G85 (registration and renewal), every IBM 2462 Model 4BX includes feature code 0001. Feature code 0003 is available at the end of the first year to renew SW maintenance for one more year. For software PID number 5765-G86 (maintenance reinstatement 12 months), feature code 0001 is available if SW PID 5765-G85 feature code 0003 was not ordered before the one year expired. For software PID number 5765-G87 (3-year registration) feature code 0001 can be ordered instead of SW PID 5765-G85 feature code 0003 to make the initial period three years, instead of one year. For software PID number 5765-G88 (3-year renewal), feature code 0001 can be used as alternative SW PID 5765-G85 feature code 0003 if a three year renewal is desired. The maximum duration is five years. For software PID number 5765-G89 (3-year after license), feature code 0001 is available if SW PID 5765-G85 feature code 0003 was not ordered before the one year expired if a three year renewal is desired.
IBM BladeCenter HX5 blades

The IBM BladeCenter HX5 is a scalable blade server designed to provide new levels of utilization, performance, and reliability for compute and memory intensive workloads such as database, virtualization, business intelligence, modeling and simulation, and other enterprise applications. Select System x blades running Linux on System x and Microsoft Windows on IBM System x servers are supported in the zBX, utilizing the zBX integrated hypervisor for IBM System x blades (Kernel Virtual Machine - based), providing logical device integration between System z and System x blades for multitiered applications. System x blades are licensed separately and are enabled and managed as part of the ensemble by Unified Resource Manager. The following operating systems are supported: Linux on System x (64/bit only) Red Hat RHEL 5.5, 5.6, 5.7, 6.0, and 6.1 SUSE SLES 10 (SP4), 11 SP1 Windows Server 2008 R2 and Windows Server 2008 SP2 (Datacenter Edition recommended), 64-bit only 222
8049ch07.fm
Support of select IBM System x blades in the zBX allows the zEnterprise to access a whole new application portfolio. Front-end applications that need access to centralized data serving would be a good fit for running on the blades, as well as applications that are a front end to core CICS or IMS transaction processing such as IBM WebSphere. BladeCenter HX5 blades are customer acquired through existing channels or through IBM. POWER7, DataPower XI50z, and System x blades can be in the same BladeCenter chassis. Supported configuration options are listed in Table 7-2 on page 223. IBM BladeCenter HX5 7873 is a dual-socket 16-core blade with the following features: Intel 8-core processor Two processor sockets 2.13 GHz 105W processor Max 14 A16Ms per BC-H Memory: up to 16 DIMM DDR-3 with 6.4 GTs 100 GB SSD Internal Disk
Table 7-2 Supported configuration of System x blades System x blades Part number Feature code Config 0 Config 1 Config 2 Config 3
Blades base - HX5 Processor 2.13 GHz 105W Intel Processors Blade width Total Cores Memory kits 8 GB 1333 MHz 16 GB 1333 MHz GB/core Speed Burst SSD Exp Card 50GB SSD MLC No Internal Raid CFFh 10GbE CIOv 8Gb FC
MT 7873 69Y3071 69Y3072
A16M A16S A179
1 1 1 2 Single 16
1 1 1 2 Single 16 16 0 8 1 1 2 1 1 1
1 1 1 2 Single 16 8 8 12 1 1 2 1 1 1
1 1 1 2 Single 16 0 16 16 1 1 2 1 1 1
46C0558 49Y1527
A17Q 2422
8 0 4
46M6843 46M6906 43W7727 46M6170 44X1946
1741 5765 5428 9012 0099 1462
1 1 2 1 1 1
7.2.5 Power distribution unit (PDU)

The power distribution units (PDUs) provide connection to the main power source for intranode management network and intraensemble data network top of rack switches, and the BladeCenter. The number of power connections needed is based on the zBX configuration. A rack contains two PDUs if one BladeCenter is installed and four PDUs if two BladeCenters are installed.
7.3 zBX entitlements, firmware, and upgrades

When ordering a zBX, the controlling zEC12 node will have the entitlements feature for the configured blades. The entitlements are similar to a high water mark or maximum purchased
223
8049ch07.fm
flag, and only a blade quantity equal to or less than installed in the zBX can communicate with the CPC. Also, Unified Resource Manager has two management suites, Manage suite (FC 0019) and Automate/Advanced Management Suite (FC 0020). If the controlling zEC12 has Manage suite (FC 0019), then the same quantity entered for any blade enablement feature code (FC 0611, FC 0612 or FC 0613) will be used for Manage Firmware (FC 0047, FC 0048, or FC 0049) of the corresponding blades. If the controlling zEC12 has Automate/Advanced Management Suite (FC 0020), then the same quantity entered for Blade Enablement feature codes (FC 0611, FC 0612, or FC 0613) will be used for the Manage Firmware (FC 0047, FC 0048, FC 0049) and Automate/Advanced Management Firmware (FC 0050, FC 0051, FC 0053) of the corresponding blades. Table 7-3 lists these features, while Table 7-4 shows maximum quantities for these feature codes. Minimum quantity to order depends on the number of corresponding blades configured and located in the zBX Model 003.
Table 7-3 Feature codes for blade enablements and Unified Resource Manager suites Blade Enablement Manage - per connection Automate/Advanced Management - per connection
z/OS only IFL DataPower XI50z POWER7 Blade IBM System x HX5 Blade
N/A N/A FC 0611 FC 0612 FC 0613
FC 0019 N/C FC 0047 FC 0048 FC 0049
FC 0020 FC 0054 FC 0050 FC 0051 FC 0053
Blades: If any attempt is made to install additional blades that exceed the FC 0611, FC 0612, or FC 0613 count, those blades will be not be powered on by the system. The blades will also be checked for minimum hardware requirements.
Table 7-4 Maximum quantities for Unified Resource Manager feature codes Feature code Maximum quantity Feature Description
FC 0047 FC 0050 FC 0048 FC 0051 FC 0049 FC 0053 FC 0054
28 28 112 112 56 56 101
Manage firmware Data Power Automate/Advanced Management firmware Data Power Manage firmware POWER blade Automate/Advanced Management firmware POWER blade Manage firmware System x blade Advanced Management firmware System x blade Automate/Advanced Management firmware IFL
Note that FC 0047, FC 0048, FC 0049, FC 0050, FC 0051, FC 0053, and FC 0054 are priced features. In order to get ensemble member management and cables, FC 0025 should also be ordered on the zEC12. 224
8049ch07.fm
Feature codes are available to detach a zBX from an existing CPC and attach a zBX to another CPC. FC 0030 indicates that the zBX will be detached, while FC 0031 indicates that the detached zBX is going to be attached to a CPC. Only zBX Model 003 (2458-003) is supported with zEC12. When upgrading a z196 with zBX Model 002 attachment to zEC12, the zBX Model 002 (2458-002) must be upgraded to a zBX Model 003 as well. Upgrades from zBX Model 002 to zBX Model 003 are disruptive. A zBX Model 003 has the following improvements compared the zBX Model 002: New AMM in BladeCenter chassis with enhanced functions Additional ethernet connectivity in IEDN network for redundancy and higher bandwidth between TOR switches and BladeCenter switch modules. New Firmware level with improved functionality
7.3.1 zBX management

One key feature of the zBX is its integration under the System z management umbrella. Thus, initial firmware installation as well as updates and patches follow the already familiar pattern of System z. The same reasoning applies to the configuration and definitions. Similar to channels and processors, the SE has a view for the zBX blades. This view shows icons for each of the zBX components objects including an overall status (power, operational, and so on). The following functions and actions are managed and controlled from the zEC12 HMC/SE: View firmware information for the BladeCenter and blades Retrieve firmware changes Change firmware level Backup/restore critical data: zBX configuration data is backed up as part of System zEC12 SE backup. It is restored on replacement of a blade. For more details, see IBM zEnterprise Unified Resource Manager, SG24-7921.
7.3.2 zBX firmware

The firmware for the zBX is managed, controlled, and delivered in the same way as for the zEC12. It is packaged and tested with System z microcode and changes are supplied and applied with MCL bundle releases. Here we summarize the benefits of the zBX firmware packaged with System z microcode: Tested together with System z driver code and MCL bundle releases Retrieve code as same integrated process of System z (IBM RETAIN or media) No need to use separate tools and connect to websites to obtain code Utilize new upcoming System z firmware features, such as Digitally Signed Firmware Infrastructure incorporates System z concurrency controls where possible zBX firmware update fully concurrent, blades similar to Config Off/On controls Audit trail of all code changes in security log Automatic back out of changes to previous working level on code apply failures Optimizer firmware zBX exploits the broadband RSF capability of HMC
225
8049ch07.fm
7.4 zBX connectivity

There are three types of LANs (each with redundant connections) that attach to the zBX: the INMN, the IEDN, and the customer managed data network. The INMN is fully isolated and only established between the controlling zEC12 and the zBX. The IEDN connects the zBX to a maximum of eight zEC12. Each zEC12 must have a minimum of two connections to the zBX. The IEDN is also used to connect a zBX to a maximum of seven other zBXs. The IEDN is a VLAN-capable network that allows enhanced security by isolating data traffic between virtual servers. Figure 7-5 on page 226 (as a high level diagram) shows the connectivity required for the zBX environment. The zEC12 connects through two OSA-Express4S 1000BASE-T or OSA-Express3 1000BASE-T3 features (CHPID type OSM) to the INMN TOR switches. The OSA-Express4S 10 GbE or OSA-Express3 10 GbE3 features (CHPID type OSX) connect to the two IEDN TOR switches. Depending on the requirements, any OSA-Express4S or OSA-Express3 features (CHPID type OSD) can connect to the customer managed data network. Terminology: If not specifically stated otherwise, the term OSA1000BASE-T applies to the OSA-Express4S 1000BASE-T and OSA-Express3 1000BASE-T features throughout this chapter
zEnterprise node
Bu lk Pow e r Hu b A B ulk Pow e r H ub B
intr anode ma nagem ent networ k
T OR Sw itch es T OR Switches
OSM
OSX
intr aense mble da ta ne twork
zBX
Figure 7-5 INMN, IEDN, customer managed local area network
The IEDN provides private and secure 10 GbE high speed data paths between all elements of a zEnterprise ensemble (up to eight zEC12 with optional zBXs). The zBX is managed by the HMC through the physically isolated INMN, which interconnects all resources of the zEC12 and zBX components.
Carry forward only for zEC12 when upgrading from earlier generations.
226
8049ch07.fm
7.4.1 Intranode management network

The scope of the intranode management network (INMN) is within an ensemble node. A node consists of a zEC12 and its optional zBX. INMNs in different nodes are not connected to each other. The INMN connects the Support Element (SE) of the zEC12 to the hypervisor, optimizer, and guest management agents within the node.
INMN communication
Communication across the INMN is exclusively for the purpose of enabling the Unified Resource Manager of the HMC to perform its various management disciplines (virtual server, performance, network virtualization, energy, storage management, and so on) for the node. The zEC12 connection to the INMN is achieved through the definition of a CHPID type OSM, which can be defined over an OSA1000BASE-T Ethernet feature. There is also a 1 GbE infrastructure within the zBX.
INMN configuration
The key points to consider for an INMN are as follows: Each zEC12 must have two OSA 1000BASE-T ports connected to the Bulk Power Hub in the same zEC12: The two ports provide a redundant configuration for failover purposes in case one link fails. For availability, each connection must be from two different OSA 1000BASE-T features within the same zEC12. Figure 7-6 shows both the OSA 1000BASE-T features and required cable type.
1 CHPID p er feature
2 ports per CHPID
2 C HPIDs p er feature
2 ports per CHPID
X
Port 0 Port 1 is not used with CHPID typ e=OSM Port 0 Port 1 is not used with CHPID typ e=OSM
X
OSA-Express4S 1000BASE-T Ethernet (F C 0408)
OSA-Express3 1000BASE-T Ethernet (FC 3367) Category 6 Ethernet RJ45 3.2m cab le (FC 0025)
Figure 7-6 OSA-Express4S 1000BASE-T and OSA-Express3 1000BASE-T features and cable type
OSA 1000BASE-T ports can be defined in the IOCDS as SPANNED, SHARED, or DEDICATED:
227
8049ch07.fm
DEDICATED restricts the OSA 1000BASE-T port to a single LPAR. SHARED allows the OSA 1000BASE-T port to be used by all or selected LPARs in the same zEC12. SPANNED allows the OSA 1000BASE-T port to be used by all or selected LPARs across multiple Channel Subsystems in the same zEC12. SPANNED and SHARED ports can be restricted by the PARTITION keyword in the CHPID statement to allow only a subset of LPARs in the zEC12 to use the OSA 1000BASE-T port. SPANNED, SHARED, and DEDICATED link pairs can be defined within the maximum of 16 links supported by the zBX. z/OS Communication server TCPIP stack must be enabled for IPv6. The CHPID type OSM related definitions will be dynamically created. No IPv4 address is needed, a IPv6 link local address will be dynamically applied z/VM virtual switch types provide INMN access: Up-link can be virtual machine NIC (Network Interface Card). Ensemble membership conveys UUID (Universally Unique IDentifier) and MAC (Media Access Control) prefix. Two 1000BASE-T top of rack switches in the zBX (Rack B) are used for the INMN; no additional 1000BASE-T Ethernet switches are required. Figure 7-7 shows the 1000BASE-T TOR switches.
J00 J02 J04 J06 J08 J10 J12 J14 J16 J18 J20 J22 J24 J26 J 28 J30 J32 J34 J36 J38 J40 J42 J44 J46 J01 J03 J05 J07 J09 J11 J13 J15 J17 J19 J21 J23 J25 J27 J 29 J31 J33 J35 J37 J39 J41 J43 J45 J47
J00 J02 J04 J06 J08 J10 J12 J14 J16 J18 J20 J22 J24 J26 J 28 J30 J32 J34 J36 J38 J40 J42 J44 J46 J01 J03 J05 J07 J09 J11 J13 J15 J17 J19 J21 J23 J25 J27 J 29 J31 J33 J35 J37 J39 J41 J43 J45 J47
Figure 7-7 Two 1000BASE-T TOR switches (INMN)
The port assignments for both 1000BASE-T TOR switches are listed in Table 7-5.
Table 7-5 Port assignments for the 1000BASE-T TOR switches Ports Description
J00-J03 J04-J07 J08-J11 J12-J15 J16-J43 J44-J45 J46 J47
Management for BladeCenters located in zBX Rack-B Management for BladeCenters located in zBX Rack-C Management for BladeCenters located in zBX Rack-D Management for BladeCenters located in zBX Rack-E Not used INMN switch B36P(Top) to INMN switch B35P(Bottom) INMN-A to IEDN-A port J41 / INMN-B to IEDN-B port J41 INMN-A to zEC12 BPH-A port J06 / INMN-B to zEC12 BPH-B port J06
1000BASE-T supported cable:
228
8049ch07.fm
3.2 meter Category 6 Ethernet cables are shipped with the zEC12 ensemble management flag feature (FC 0025). Those cables connect the OSA 1000BASE-T (OSM) ports to the Bulk Power Hubs (port 7). 26 meter Category 5 Ethernet cables are shipped with the zBX. Those cables are used to connect the zEC12 Bulk Power Hubs (port 6) and the zBX top of rack switches (port J47).
7.4.2 Primary and alternate HMCs

The zEnterprise System Hardware Management Console (HMC) that has management responsibility for a particular zEnterprise ensemble is called a primary HMC. Only one primary HMC is active for a given ensemble. This HMC has an alternate HMC to provide redundancy; it is not available for use until it becomes the primary HMC in a failover situation. To manage ensemble resources, the primary HMC for that ensemble must be used. A primary HMC can of course perform all HMC functions. For more information about the HMC network configuration, see Chapter 12, Hardware Management Console and Support Element on page 397. Figure 7-8 on page 229 shows the primary and alternate HMC configuration connecting into the two bulk power hubs (BPHs) in the zEC12 by a customer managed management network. The 1000BASE-T TOR switches in the zBX are also connected to the BPHs in the zEC12. Ports: All ports on the zEC12 BPH are reserved for specific connections. Any deviations or mis-cabling will affect the operation of the zEC12 system.
y
HMC HMC
Customer Network 1
J01 J05
Customer Network 2 J02 J01
J02 J05 zBX
SE
SE
EC12 BPH-B Rear Z-Fram e A-F rame BPH-A Front
Figure 7-8 HMC configuration in an ensemble node
229
8049ch07.fm
Table 7-6 shows the port assignments for both bulk power hubs (BPHs).
Table 7-6 Port assignments for the BPHs BPH A Port # Connects to: Port # BPH B Connects to:
J01 J02 J03 J04 J05 J06 J07 J08 J09-J32
HMC to SE Customer Network2 (VLAN 0.40) HMC to SE Customer Network1 (VLAN 0.30) BPH B J03 BPH B J04 SE A-Side (Top SE) zBX TOR Switch B36P, Port 47 (INMN-A) OSA 1000BASE-T (CHPID type OSM) Not used Used for internal zEC12 components
J01 J02 J03 J04 J05 J06 J07 J08 J09-J32
HMC to SE Customer Network2 (VLAN 0.40) HMC to SE Customer Network1 (VLAN 0.30) BPH A J03 BPH A J04 SE B-Side (Bottom SE) zBX TOR Switch B35P, Port 47 (INMN-B) OSA 1000BASE-T (CHPID type OSM) Not used Used for internal zEC12 components
For more information, see Chapter 12, Hardware Management Console and Support Element on page 397.
7.4.3 Intraensemble data network

The intraensemble data network (IEDN) is the main application data path provisioned and managed by the Unified Resource Manager of the controlling zEC12. Data communications for ensemble-defined workloads flow over the IEDN between nodes of an ensemble. All of the physical and logical resources of the IEDN are configured and managed by the Unified Resource Manager. The IEDN extends from the zEC12 through the OSA-Express4S 10 GbE or OSA-Express3 10 GbE ports when defined as CHPID type OSX. The minimum number of OSA 10 GbE features is two per zEC12. Similarly, a 10 GbE networking infrastructure within the zBX is used for IEDN access. Terminology: If not specifically stated otherwise, the term OSA10 GbE applies to the OSA-Express4S 10 GbE and OSA-Express3 10GbE features throughout this chapter.
IEDN configuration
The IEDN connections can be configured in a number of ways. The key points to consider for an IEDN are as follows: Each zEC12 must have a minimum of two OSA 10 GbE ports connected to the zBX through the IEDN: The two ports provide a redundant configuration for failover purposes in case one link fails. For availability, each connection must be from a different OSA 10 GbE feature within the same zEC12. The zBX can have a maximum of 16 IEDN connections (8 pairs of OSA 10 GbE ports). 230
8049ch07.fm
Four connections between IEDN TOR switches and high speed switch modules in each BladeCenter chassis (2 pairs of 10GbE ports) For redundancy two connections between both high speed switch modules in each BladeCenter. Figure 7-9 shows the OSA 10 GbE feature (long reach or short reach) and the required fiber optic cable types.
OSA-Express3 10 GbE 2 CHPIDs 1 PORT/CHPID
OSA-Express4S 10 GbE 1 CHPID 1 PORT/CHPID
FC 3370 and FC0406 (Long Reach)

Single Mode 9 Micron LC Dupl ex
Port 0 CHPID type=OSX FC 3371 and FC0407 (Short Reach)

Multimode 50/62.5 Micron LC Duplex
Figure 7-9 OSA-Express4S 10 GbE and OSA-Express3 10 GbE features and cables
OSA 10 GbE ports can be defined in the IOCDS as SPANNED, SHARED, or DEDICATED: DEDICATED restricts the OSA 10 GbE port to a single LPAR. SHARED allows the OSA 10 GbE port to be used by all or selected LPARs in the same zEC12. SPANNED allows the OSA 10 GbE port to be used by all or selected LPARs across multiple Channel Subsystems (CSSs) in the same zEC12. SHARED and SPANNED ports can be restricted by the PARTITION keyword in the CHPID statement to allow only a subset of LPARs on the zEC12 to use OSA 10 GbE port. SPANNED, SHARED, and DEDICATED link pairs can be defined within the maximum of 16 links supported by the zBX z/OS Communication Server requires minimal configuration: IPv4 or IPv6 addresses are used. VLAN must be configured to match HMC (Unified Resource Manager) configuration. z/VM virtual switch types provide IEDN access: Up-link can be a virtual machine NIC. Ensemble membership conveys Ensemble UUID and MAC prefix. IEDN network definition are completed from the primary HMC Manage Virtual Network task.
231
8049ch07.fm
Two 10 GbE top of rack switches in the zBX (Rack B) are used for the IEDN, no additional Ethernet switches are required. Figure 7-10 on page 232 shows the 10 GbE TOR switches.
Figure 7-10 Two 10 GbE TOR switches
The IBM zEnterprise EC12 provides the capability to integrate HiperSockets connectivity to the intraensemble data network (IEDN). This extends the reach of the HiperSockets network outside the CPC to the entire ensemble, appearing as a single, Layer 2. Because HiperSockets and IEDN are both internal System z networks, the combination allows System z virtual servers to use the optimal path for communications. The support of HiperSockets integration with the IEDN function is available on z/OS Communication Server V1R13 and z/VM V6R2.
Port assignments
The port assignments for both 10 GbE TOR switches are listed in Table 7-7.
Table 7-7 Port assignments for the 10 GbE TOR switches Ports Description
J00 - J07 J08 - J23 J24 - J30 J31 - J37 J38 - J39 J40 J41
SFP and reserved for zEC12 (OSX) IEDN connections DAC reserved for BladeCenter SM07/SM09 IEDN connections SFP reserved for zBX-to-zBX IEDN connections SFP reserved for client IEDN connections (OSD) DAC for TOR switch-to-TOR switch IEDN communication RJ-45 (not used) RJ-45 IEDN Switch Management Port to INMN TOR switch port 46
All IEDN connections must be point-to-point to the 10 GbE switch: The IEDN connection uses MAC address, not IP address (Layer 2 connection). No additional switches or routers are needed. This limits the distances CPCs the 10 GbE TOR switches in an ensemble. The 10 GbE TOR switches utilize small form factor plugable (SFP) optics for the external connections and direct attach cables (DAC) for connections, as follows: Ports J00-J07 are reserved for the zEC12 OSX IEDN connections. These ports utilize SFPs (Small Form Factor Pluggable Modules) plugged according to the zBX order: FC 0632 LR SFP to FC 0406 OSA-Express4S 10 GbE LR
232
8049ch07.fm
FC 0633 SR SFP to FC 0407 OSA-Express4S 10 GbE SR FC 0632 LR SFP to FC 3370 OSA-Express3 10 GbE LR FC 0633 SR SFP to FC 3371 OSA-Express3 10 GbE SR Ports J08-J23 are reserved for IEDN to BladeCenter attachment. The cables used are Direct Attached Cables (DAC) and are included with the zBX. These are hard wired 10 GbE SFP cables. The Feature codes indicate the length of the cable: FC 0626 - one meter for Rack B BladeCenters and IEDN to IEDN FC 0627 - five meters for Rack C BladeCenter FC 0628 - seven meters for Racks D and E BladeCenters Ports J31-J37 are reserved for the zEC12 OSD IEDN connections. These ports utilize SFPs (Small Form Factor Pluggable Modules) plugged according to the zBX order.10 GbE fiber optic cable types and maximum distance: Client provides all IEDN cables (except for zBX internal connections). Multimode fiber: 50 micron fiber at 2000 MHz-km: 300 meters 50 micron fiber at 500 MHz-km: 82 meters 62.5 micron fiber at 200 MHz-km: 33 meters Single mode fiber: 10 km
7.4.4 Network connectivity rules with zBX

The network connectivity rules for interconnecting a zBX are as follows: Only one zBX is allowed per controlling zEC12. The zBX can be installed next to the controlling zEC12 or within the limitation of the 26 meter cable. Customer managed data networks are outside the ensemble. A customer managed data network is connected with: CHPID type OSD from zEC12 IEDN TOR switch ports J31 to J37 from zBX
7.4.5 Network security considerations with zBX

The private networks involved in connecting the zEC12 to the zBX are constructed with extreme security in mind, for example: The INMN is entirely private and can be accessed only by the SE (standard HMC security applies). There are also additions to Unified Resource Manager role-based security, so that not just any user can reach the Unified Resource Manager panels even if that user can perform other functions of the HMC. Very strict authorizations for users and programs control who is allowed to take advantage of the INMN. The INMN network is using link-local IP addresses. Link-local addresses are not advertised and are accessible only within a single LAN segment. There is no routing in this network, as it is a flat network with all Virtual Servers residing on the same IPv6 network. The Unified Resource Manager communicates with the Virtual Servers through the SE over the INMN. The Virtual Servers cannot communicate with each other directly through INMN, they can only communicate with the SE. Only authorized programs or agents can take advantage of the INMN; currently the Performance Agent can do so. However, there can be other platform management applications in the future, these must be authorized to access the INMN.
233
8049ch07.fm
The IEDN is built on a flat network design (same IPv4 or IPv6 network) and each server accessing the IEDN must be an authorized Virtual Server and must belong to an authorized Virtual LAN within the physical IEDN. VLAN enforcement sits within the hypervisor functions of the ensemble; controls reside in the OSA (CHPID type OSX), in the z/VM VSWITCH, and in the VSWITCH hypervisor function of the blades on the zBX. The VLAN IDs and the Virtual MACs that are assigned to the connections from the Virtual Servers are tightly controlled through the Unified Resource Manager and thus there is no chance of either MAC or VLAN spoofing for any of the servers on the IEDN. If a you decide to attach your network to the TOR switches of the zBX in order to communicate with the Virtual Servers on the zBX blades, access must be authorized in the TOR switches (MACor VLAN-based). Although the TOR switches will enforce the VMACs and VLAN IDs here, you must take the usual network security measures to ensure that the devices in the Customer Managed Data Network are not subject to MAC or VLAN spoofing; the Unified Resource Manager functions cannot control the assignment of VLAN IDs and VMACs in those devices. In other words, whenever you decide to interconnect the external network to the secured IEDN, the security of that external network must involve all the usual layers of the IBM Security Framework: physical security, platform security, application and process security, data and information security, and so on. The INMN and the IEDN are both subject to Network Access Controls as implemented in z/OS and z/VM; so that not just any virtual server on the zEC12 that can utilize these networks. INMN is not accessible at all from within the virtual servers. Although we deem it unnecessary to implement firewalls, IP filtering, or encryption for data flowing over the IEDN, if company policy or security require such measures to be taken, then these are supported. You can implement any of the security technologies available: for example, SSL/TLS or IP filtering. The centralized and internal network design of both the INMN and the IEDN limit the vulnerability to security breaches. Both networks reduce the amount of network equipment and administration tasks, as well as routing hops that are under the control of multiple individuals and subject to security threats. Both use IBM-only equipment (switches, blades) which have been tested previously and in certain cases pre-installed. In summary, many more technologies than in the past have been architected in a more robust, secure fashion to integrate into the client network. This is achieved with the help of either the Unified Resource Manager, or additional SAF controls specific to zEnterprise System and the ensemble, such as these: MAC filtering VLAN enforcement ACCESS control Role-based security The following standard security implementations are still available for use in the IEDN: Authentication Authorization and access control (including Multilevel Security (MLS); also firewall IP filtering. Only stateless firewalls or IP filtering implementations can be installed in a Virtual Server in the ensemble) Confidentiality Data integrity Non-repudiation
234
8049ch07.fm
7.4.6 zBX storage connectivity

The Fibre Channel (FC) connections can be established between the zBX and a SAN environment. Client supplied FC switches are required and must support N_Port ID Virtualization (NPIV). Some FC switch vendors also require interop mode. Check the interoperability matrix for the latest details, at this website: http://www-03.ibm.com/systems/support/storage/ssic/interoperability.wss Cables: It is the clients responsibility to supply the cables for the IEDN, the customer managed network, and the connection between the zBX and the SAN environment. Each BladeCenter chassis in the zBX has two 20-port 8 Gbps Fibre Channel (FC) switch modules. Each switch has 14 internal ports and six shortwave (SX) external ports. The internal ports are reserved for the blades in the chassis. The six external ports (J00, J15-J19) are used to connect to the SAN. Figure 7-11 shows an image of the external ports.
Switch Module 1 SM03
J00 J15 J16 J17 J18 J19
J00 J15 J16
BladeCenter
Switch Module 2 SM04
Figure 7-11 8 Gb FC switch external ports
J17 J18 J19
Client provided multi-mode LC duplex cables are used for the FC switch connections to support speeds of 8 Gbps, 4 Gbps, or 2 Gbps. (1 Gbps is not supported.) Maximum distance depends on the speed and fiber type. Cabling specifications are defined by the Fibre Channel - Physical Interface - 4 (FC-PI-4) standard. Table 7-8 on page 236 identifies cabling types and link data rates that are supported in the zBX SAN environment, including their allowable maximum distances and link loss budget.
235
8049ch07.fm
The link loss budget is derived from the channel insertion loss budget defined by the FC-PI-4 standard (Revision 8.00).
Table 7-8 Fiber optic cabling for zBX FC switch - maximum distances and link loss budget FC-PI-4 2 Gbps 4 Gbps 8 Gbps
Fiber core (light source) 50 m MMa (SX laser) 50 m MMb (SX laser) 62.5 m MMc (SX laser)
Distance in meters 500 300 150
Link loss budget (dB) 3.31 2.62 2.1
a. OM3: 50/125 m laser optimized multimode fiber with a minimum overfilled launch bandwidth of 1500 MHz-km at 850nm as well as an effective laser launch bandwidth of 2000 MHz-km at 850 nm in accordance with IEC 60793-2-10 Type A1a.2 fiber b. OM2: 50/125 m multimode fiber with a bandwidth of 500 MHzkm at 850 nm and 500 MHz-km at 1300 nm in accordance with IEC 60793-2-10 Type A1a.1 fiber. c. OM1: 62.5/125 m multimode fiber with a minimum overfilled launch bandwidth of 200 MHzkm at 850 nm and 500 MHz-km at 1300 nm in accordance with IEC 60793-2-10 Type A1b fiber.
Cabling: IBM does not support a mix of 50 m and 62.5 m fiber optic cabling in the same physical link.
IBM blade storage connectivity

IBM blades use six ports in both FC switch modules (SM03 and SM04) of the BladeCenter chassis and must connect through an FC switch to FC disk storage. Figure 7-12 illustrates the FC connectivity with two FC switches for redundancy and high availability.
Up to six 8 Gbps FC links
SM03
Up to six 8 Gbps FC links
SM04
NPIV support required

Customer provided FC switch Customer provided FC switch
Fibre Channel Disk Storage
Figure 7-12 BladeCenter chassis storage connectivity
236
8049ch07.fm
Up to six external ports of each BladeCenter switch module can be used to connect to the SAN. All fiber links of a switch module have to be attached to the same SAN switch as shown in Figure 7-12. SAN switches must support NPIV to allow virtualization. The client provides all cables, FC disk storage, and FC switches. It is also the clients responsibility to configure and cable the FC switches that connect to the zBX.
Supported FC disk storage

Supported FC disk types and vendors with IBM blades are listed on the IBM System Storage Interoperation Center (SSIC) website, at the following website: http://www-03.ibm.com/systems/support/storage/config/ssic/displayesssearchwithoutj s.wss?start_over=yes
7.5 zBX connectivity examples

This section illustrates various ensemble configuration examples containing a zBX and the necessary connectivity for operation. For simplicity, we do not show the redundant connections in configuration examples. Subsequent configuration diagrams build on the previous configuration and only additional connections will be noted.
237
8049ch07.fm
7.5.1 A single node ensemble with a zBX

Figure 7-13 shows a single node ensemble with a zBX. The necessary components include the controlling zEC12 (CPC1) and the attached zBX, FC switches, and FC disk storage.
Primary HM C 1
Alternate HMC
O S M
EC12 CPC1 Controls 2458

I N M N O S X
4 FC Switch
2458-003 zBX
IBM Blades F ibre Channel Disk Storage
Figure 7-13 Single node ensemble with zBX
The diagram shows the following components: 1. Client provided management network: IBM supplies a 15 meter Ethernet RJ-45 cable with the 1000BASE-T (1GbE) switch (FC 0070). The 1000BASE-T switch (FC 0070) connects to the reserved client network ports of the Bulk Power Hubs in zEC12 - Z29BPS11 (on A side) and Z29BPS31 (on B side) port J02. A second switch connects to Z29BPS11 and Z29BPS31 on port J01. 2. Intranode management network: Two CHPIDs from two different OSA1000BASE-T features are configured as CHPID type OSM. IBM supplies two 3.2 meter Ethernet Category 6 cables from the OSM CHPIDs (ports) to both Z29BPS11 and Z29BPS31 on port J07. (This is a zEC12 internal connection supplied with feature code 0025.) 3. Intranode management network - extension:
238
8049ch07.fm
IBM supplies two 26 meter Category 5 Ethernet cables (chrome gray plenum rated cables) from zBX Rack B INMN-A/B switches port J47 to Z29BPS11 and Z29BPS31 on port J06. 4. Intraensemble data network: Two ports from two different OSA10 GbE (SR Short Reach or LR Long Reach) features are configured as CHPID type OSX. The client supplies the fiber optic cables (single mode or multimode). 5. 8 Gbps Fibre Channel switch: The client supplies all Fibre Channel cables (multimode) from the zBX to the attached FC switch. The client is responsible for the configuration and management of the FC switch.
7.5.2 A dual node ensemble with a single zBX

A second zEC12 (CPC2) is introduced in Figure 7-14, showing the additional hardware. Up to eight additional nodes (CPCs) can be added in the same fashion.
Prim ary HMC Alternate HMC
1
O S M
EC12 CPC 1 EC 12 Controls 2458

I N M N
CPC2
O S M O S X
2
O S X
2458-003 zBX
FC S witc h
IBM B lades
Fibre Channel Disk Sto rage
Figure 7-14 Dual node ensemble with a single zBX
The diagram shows the following components: 1. Client provided management network: The client supplies an Ethernet RJ-45 cable.
239
8049ch07.fm
The 1000BASE-T switch (FC 0070) connects to the reserved client network ports of Z29BPS11 and Z29BPS31 - J02. A second switch connects to Z29BPS11 and Z29BPS31 on port J01. 2. Intranode management network: Two ports from two different OSA1000BASE-T features are configured as CHPID type OSM IBM supplies two 3.2 meter Ethernet Category 6 cables from the OSM CHPIDs (ports) to both Z29BPS11 and Z29BPS31 on port J07. (this is a zEC12 internal connection supplied with feature code 0025). 3. Intraensemble data network: Two ports from two different OSA10 GbE (SR Short Reach or LR Long Reach) features are configured as CHPID type OSX. The client supplies the fiber optic cables (single mode or multimode).
7.5.3 A dual node ensemble with two zBXs

Figure 7-15 introduces a second zBX added to the original configuration. The two zBXs are interconnected through fiber optic cables to SFPs in the IEDN switches for isolated communication (SR or LR) over the IEDN network.
P rim ary HMC Alternate HMC
O S M
EC1 2 CPC1 C ontrols 24 58

I N M N O S X
O S M
EC12 CPC 1 Contr ols 2 45 8

I N M N
O S X
FC Switch
FC Switch
2 458 -003 zB X
1
IEDN
2 458 -003 zB X
I BM Bl ades Fib re Channel Disk S torage
IBM Blades Fi bre Chan nel Di sk Storag e
Figure 7-15 Dual node ensemble
The diagram shows the following components: Intraensemble data network: Two 10 GbE ports in the TORs are used to connect the two zBXs (10 GbE TOR switch to 10 GbE TOR switch).
240
8049ch07.fm
Up to eight CPCs can be connected to a zBX using the IEDN. Additional CPCs are added and connected to the zBX through the OSA 10 GbE (SR Short Reach or LR Long Reach) features configured as CHPID type OSX.
7.6 References
Installation details can be found in IBM zEnterprise BladeCenter Extension Model 003 Installation Manual for Physical Planning, GC27-2611 and IBM zEnterprise BladeCenter Extension Model 003 Installation Manual, GC27-2610. For details about the BladeCenter components, see IBM BladeCenter Products and Technology, SG24-7523. For more information about DataPower XI50z blades; visit this web site: http://www-01.ibm.com/software/integration/datapower/xi50z Additional documentation is available on the IBM Resource Link at this web site: http://www.ibm.com/servers/resourcelink
241
8049ch07.fm
242
8049ch08.fm
Chapter 8.
Software support
This chapter lists the minimum operating system requirements and support considerations for the zEC12 and its features. It discusses z/OS, z/VM, z/VSE, z/TPF, and Linux on System z. Because this information is subject to change, see the Preventive Service Planning (PSP) bucket for 2827DEVICE for the most current information. Also discussed is generic software support for zEnterprise BladeCenter Extension (zBX) Model 003. Support of IBM zEnterprise EC12 functions is dependent on the operating system, its version, and release. This chapter discusses the following topics: Operating systems summary on page 244 Support by operating system on page 244 Support by function on page 256 Cryptographic support on page 288 z/OS migration considerations on page 295 Coupling facility and CFCC considerations on page 297 MIDAW facility on page 299 IOCP on page 302 ICKDSF on page 303 zEnterprise BladeCenter Extension (zBX) Model 003 software support on page 304 Software licensing considerations on page 305 References on page 309
243
8049ch08.fm
8.1 Operating systems summary

Table 8-1 lists the minimum operating system levels required on the zEC12. For zBX, see 8.11, zEnterprise BladeCenter Extension (zBX) Model 003 software support on page 304. Note that operating system levels that are no longer in service are not covered in this publication. These older levels might provide support for some features.
Table 8-1 EC12 minimum operating systems requirements Operating systems ESA/390 (31-bit mode) z/Architecture (64-bit mode) Notes
z/OS V1R10a z/VM V5R4

b
No No No Yes Nod
Yes Yesc Yes Yes See Table 8-2 on page 246 Service is required. See the following box, entitled Features
z/VSE V4R3 z/TPF V1R1 Linux on System z
a. Regular service support for z/OS V1R10 and V1R11 ended in September 2011 and September 2012, respectively. However, by ordering the IBM Lifecycle Extension for z/OS V1R10 and V1R11 product, fee-based corrective service can be obtained for up to September 2013 and September 2014, respectively. b. z/VM V5R4, V6R1 and V6R2 provide compatibility support only. z/VM 6.1 and 6.2 require an ALS at z10 or higher c. VM supports both 31-bit and 64-bit mode guests d. 64-bit distributions included 31-bit emulation layer to run 31-bit software products.
Features: Exploitation of certain features depends on a particular operating system. In all cases, PTFs might be required with the operating system level indicated. Check the z/OS, z/VM, z/VSE, and z/TPF subsets of the 2827DEVICE Preventive Service Planning (PSP) buckets. The PSP buckets are continuously updated and contain the latest information about maintenance. Hardware and software buckets contain installation information, hardware and software service levels, service guidelines, and cross-product dependencies. For Linux on System z distributions, consult the distributors support information.
8.2 Support by operating system

IBM zEnterprise EC12 introduces several new functions. In this section, we discuss support of those functions by the current operating systems. Also included are some of the functions introduced in previous System z servers and carried forward or enhanced in the zEC12. Features and functions available on previous servers but no longer supported by zEC12 have been removed. For a list of supported functions and the z/OS and z/VM minimum required support levels, see Table 8-3 on page 247. For z/VSE, z/TPF, and Linux on System z see Table 8-4 on page 252. The tabular format is intended to help determine, by a quick scan, which functions are supported and the minimum operating system level required. 244
8049ch08.fm
8.2.1 z/OS
z/OS Version 1 Release 12 is the earliest in-service release supporting the EC12. After September 2014, a fee-based Extended Service for defect support (for up to three years) can be obtained for z/OS V1.12. Although service support for z/OS Version 1 Release 11 ended in September of 2012, a fee-based extension for defect support (for up to two years) can be obtained by ordering the IBM Lifecycle Extension for z/OS V1.11. Similarly, IBM Lifecycle Extension for z/OS V1.10 provides fee-based support for z/OS Version 1 Release 10 until September 2013. Also note that z/OS.e is not supported on EC12 and that z/OS.e Version 1 Release 8 was the last release of z/OS.e. See Table 8-3 on page 247 for a list of supported functions and their minimum required support levels.
8.2.2 z/VM
At general availability, z/VM V5R4, V6R1 and V6R2 provide compatibility support with limited exploitation of new zEC12 functions. See Table 8-3 on page 247 for a list of supported functions and their minimum required support levels. Capacity: For the capacity of any z/VM logical partitions, and any z/VM guests, in terms of the number of IFLs and CPs, real or virtual, it is desirable that these be adjusted to accommodate the PU capacity of the EC12.
8.2.3 z/VSE
Support is provided by z/VSE V4R3 and above. Note the following considerations: z/VSE executes in z/Architecture mode only. z/VSE exploits 64-bit real memory addressing. Support for 64-bit virtual addressing is provided by z/VSE V5R1. z/VSE V5R1 requires an architectural level set specific to the IBM System z9. See Table 8-4 on page 252 for a list of supported functions and their minimum required support levels.
8.2.4 z/TPF
See Table 8-4 on page 252 for a list of supported functions and their minimum required support levels.
8.2.5 Linux on System z

Linux on System z distributions are built separately for the 31-bit and 64-bit addressing modes of the z/Architecture. The newer distribution versions are built for 64-bit only. You can run 31-bit applications in the 31-bit emulation layer on a 64-bit Linux on System z distribution. None of the current versions of Linux on System z distributions (SUSE Linux Enterprise Server - SLES 10, SLES 11, Red Hat RHEL 5, and Red Hat RHEL 6)1 require zEC12
1
SLES is SUSE Linux Enterprise Server RHEL is Red Hat Enterprise Linux
Chapter 8. Software support
245
8049ch08.fm
toleration support. Table 8-2 on page 246 shows the service levels of SUSE and Red Hat releases supported at the time of writing.
Table 8-2 Current Linux on System z distributions Linux on System z distribution z/Architecture (64-bit mode)
SUSE SLES 10 SP4 SUSE SLES 11 SP2 Red Hat RHEL 5.8 Red Hat RHEL 6.2
Yes Yes Yes Yes
IBM is working with its Linux distribution partners to provide further exploitation of selected zEC12 functions in future Linux on System z distribution releases. Consider the following guidelines: Use SUSE SLES 11 or Red Hat RHEL 6 in any new projects for the zEC12. Update any Linux distributions to their latest service level before migration to zEC12. Adjust the capacity of any z/VM and Linux on System z logical partitions guests, as well as z/VM guests, in terms of the number of IFLs and CPs, real or virtual, according to the PU capacity of the zEC12.
8.2.6 zEC12 functions support summary

In the following tables, although we attempt to note all functions requiring support, the PTF numbers are not given. Therefore, for the most current information, see the Preventive Service Planning (PSP) bucket for 2827DEVICE. The following two tables summarize the zEC12 functions and their minimum required operating system support levels: Table 8-3 on page 247 is for z/OS and z/VM. Table 8-4 on page 252 is for z/VSE, z/TPF, and Linux on System z. Information about Linux on System z refers exclusively to appropriate distributions of SUSE and Red Hat. Both tables use the following conventions: Y N The function is supported. The function is not supported. The function is not applicable to that specific operating system.
246
8049ch08.fm
Table 8-3 EC12 functions minimum support requirements summary, part 1 Function z/OS V1 R13 z/OS V1 R12 z/OS V1 R11 z/OS V1 R10 z/VM V6 R2 z/VM V6 R1 z/VM V5 R4
zEC12 Support of Unified Resource Manager Greater than 64 PUs single system image Greater than 54 PUs single system image Support of IBM zAware zIIP zAAP zAAP on zIIP Java Exploitation of Transactional Execution Large memory (> 128 GB) Large page support Out-of-order execution Guest support for execute-extensions facility Hardware decimal floating point Zero address detection 60 logical partitions LPAR group capacity limit CPU measurement facility Separate LPAR management of PUs Dynamic add and delete logical partition name Capacity provisioning Enhanced flexibility for CoD HiperDispatch 63.75 K subchannels Four logical channel subsystems (LCSS) Dynamic I/O support for multiple LCSS Third subchannel set Multiple subchannel sets IPL from alternate subchannel set MIDAW facility
Ym Y Y Y Ym Y Y Y Ym Y Ye f Y Y
h
Ym Ym Y Y N Y Y Y N Y Y Y Yh Y Y Y Ym Y Y Y Yh Y Y Y Y Ym Y Ym Y
Ym Ym Y Y N Y Y Y N Y Y Y Yh N Y Y Ym Y Y Y Yh Y Y Y Y Ym Y Ym Y
Ym Ym Ym Y N Y Y Ym N Y Y Y Yh N Y Y Ym Y Y Y Yh Y Y Y Y Ym Y N Y
Ym Y N Na Yb Yb Yc N Yd Ng Y Y Yb N Y Yb Y Y Ng Yh Ng Y Y Y Ng Ng Ng Yb
Ym Y N Na Yb Yb Yc N Yd Ng Y Y Yb N Y Ybm Y Y Ng Yh Ng Y Y Y Ng Ng Ng Yb
Ym N N Na Yb Yb Yc N Yd Ng Y Y Yb N Y Ybm Y Y Ng Yh Ng Y Y Y Ng Ng Ng Yb
Y Y Y Ym Y Y Y Y Y Y Y Y Y Y Ym Y
Cryptography
CPACF
Yb
Yb
Yb
247
8049ch08.fm
Function
z/OS V1 R13
z/OS V1 R12
z/OS V1 R11
z/OS V1 R10
z/VM V6 R2
z/VM V6 R1
z/VM V5 R4
CPACF AES-128, AES-192 and AES-256 CPACF SHA-1, SHA-224, SHA-256, SHA-384, SHA-512 CPACF protected key Crypto Express4S Secure IBM Enterprise PKCS #11 (EP11) coprocessor mode Crypto Express3 Elliptic Curve Cryptography (ECC)
Y Y Y Yjk Yjk Y Y
HiperSockets
Y Y Y Yjk Yjk Y Yi
Y Y Yi Yj Yj Yi Yi
Y Y Yi Yj Yj Yi Yi
Yb Yb Yb Ybm Ybm Yb Yb
Yb Yb Ybm Ybm Ybm Ybm Ybm
Yb Yb Ybm Ybm Ybm Ybm Ybm
32 Hipersockets HiperSockets Completion Queue HiperSockets integration with IEDN HiperSockets Virtual Switch Bridge HiperSockets Network Traffic Analyzer HiperSockets Multiple Write Facility HiperSockets support of IPV6 HiperSockets Layer 2 support HiperSockets
Y Ym Ym N Y Y Y Y
Ym N N N Y Y Y Y
Ym N N N Y Y N Y
Ym N N N Y Y N Y
Y N Ym Ym Yb Ng Y Yb Y
Ym N N N Ybm Ng Y Yb Y
Ym N N N Ybm Ng Y Yb Y
Flash Express Storage
Flash Express
Yilm
FICON (FIber CONnection) and FCP (Fibre Channel Protocol)
z/OS Discovery and auto configuration (zDAC) FICON Express8S support of zHPF enhanced multitrack CHPID type FC FICON Express8 support of zHPF enhanced multitrack CHPID type FC High Performance FICON for System z (zHPF) FCP - increased performance for small block sizes Request node identification data FICON link incident reporting N_Port ID Virtualization for FICON (NPIV) CHPID type FCP
Y Y
Ym Y
N Ym
N Ym
N Ym
N N
N N
Ym
Ym
Ng
Ng
Ng
Y N Y Y N
Y N Y Y N
Y N Y Y N
Ym N Y Y N
Ng Y N N Y
Ng Y N N Y
Ng Y N N Y
248
8049ch08.fm
Function
z/OS V1 R13
z/OS V1 R12
z/OS V1 R11
z/OS V1 R10
z/VM V6 R2
z/VM V6 R1
z/VM V5 R4
FCP point-to-point attachments FICON SAN platform & name server registration FCP SAN management SCSI IPL for FCP Cascaded FICON Directors CHPID type FC Cascaded FICON Directors CHPID type FCP FICON Express8S support of hardware data router CHPID type FCP FICON Express8S and FICON Express8 support of T10-DIF CHPID type FCP FICON Express8S, FICON Express8, FICON Express4 10KM LX, and FICON Express4 SX support of SCSI disks CHPID type FCP FICON Express8S CHPID type FC FICON Express8 CHPID type FC FICON Express4 10KM LX and SXo CHPID type FC
N Y N N Y N N
N Y N N Y N N
N Y N N Y N N
N Y N N Y N N
Y Y N Y Y Y N
Y Y N Y Y Y N
Y Y N Y Y Y N
Ybm
Ym
Y Y Y
Y Y Y
Ym Yn Y
Ym Yn Y
Y Yn Y
Y Yn Y
Y Yn Y
OSA (Open Systems Adapter)
VLAN management VLAN (IEE 802.1q) support QDIO data connection isolation for z/VM virtualized environments OSA Layer 3 Virtual MAC OSA Dynamic LAN idle OSA/SF enhancements for IP, MAC addressing (CHPID type OSD) QDIO diagnostic synchronization Network Traffic Analyzer Large send for IPv6 packet Broadcast for IPv4 packets Checksum offload for IPv4 packets
Y Y Y Y Y Y Y Ym Y Y
Y Y Y Y Y Y Y N Y Y
Y Y Y Y Y Y Y N Y Y
Y Y Y Y Y Y Y N Y Y
Y Y Y Yb Yb Y Yb Yb Yb Y Yp
Y Y Y Yb Yb Y Yb Yb Yb Y Yp
Y Y Ym Yb Yb Y Yb Yb Yb Y Yp
249
8049ch08.fm
Function
z/OS V1 R13
z/OS V1 R12
z/OS V1 R11
z/OS V1 R10
z/VM V6 R2
z/VM V6 R1
z/VM V5 R4
OSA-Express4S and OSA-Express3 inbound workload queueing for Enterprise Extender OSA-Express4S 10 Gigabit Ethernet LR and SR CHPID type OSD OSA-Express4S 10 Gigabit Ethernet LR and SR CHPID type OSX OSA-Express4S Gigabit Ethernet LX and SX CHPID type OSD (using two ports per CHPID) OSA-Express4S Gigabit Ethernet LX and SX CHPID type OSD (using one port per CHPID) OSA-Express4S 1000BASE-T CHPID type OSC (using one or two ports per CHPID) OSA-Express4S 1000BASE-T CHPID type OSD (using two ports per CHPID) OSA-Express4S 1000BASE-T CHPID type OSD (using one port per CHPID) OSA-Express4S 1000BASE-T CHPID type OSE (using one or two ports per CHPID) OSA-Express4S 1000BASE-T CHPID type OSMr (using one port per CHPID) OSA-Express4S 1000BASE-T CHPID type OSNq OSA-Express3 10 Gigabit Ethernet LR and SR CHPID type OSD OSA-Express3 10 Gigabit Ethernet LR and SR CHPID type OSX OSA-Express3 Gigabit Ethernet LX and SX CHPID types OSD, OSN q (using two ports per CHPID) OSA-Express3 Gigabit Ethernet LX and SX CHPID types OSD, OSN q (using one port per CHPID) OSA-Express3 1000BASE-T CHPID type OSC (using two ports per CHPID) OSA-Express3 1000BASE-T CHPID type OSD (using two ports per CHPID) OSA-Express3 1000BASE-T CHPID types OSC and OSD (using one port per CHPID)
Y Y Y Y Y Y
N Y Ym Y Y Y
N Ym Ym Ym Ym Ym
N Ym Ym Ym Ym Ym
Yb Y Y Y Y Y
Yb Y
Yb Y
Ym Y Y Y
Ym Ym Y Y
Y Y Y
Y Y Y
Ym Ym Ym
Ym Ym Ym
Y Y Y
Y Y Y
Ym Y Y
Y Y Y Y Y
Ym Y Y Ym Y
Ym Ym Y Ym Y
Ym Ym Y Ym Y
Y Y Y Y Y
Ym Y Y Ym Y
Ym Y Y Yms Ym
Y Y Y
Y Y Y
Y Y Y
Y Y Y
Y Y Y
Y Y Y
Y Ym Y
250
8049ch08.fm
Function
z/OS V1 R13
z/OS V1 R12
z/OS V1 R11
z/OS V1 R10
z/VM V6 R2
z/VM V6 R1
z/VM V5 R4
OSA-Express3 1000BASE-T CHPID type OSE (using one or two ports per CHPID) OSA-Express3 1000BASE-T CHPID type OSM r (using one port per CHPID) OSA-Express3 1000BASE-T CHPID type OSN q
Y Y
Ym Y
Ym Y
Ym Y
Y Y
Ym Y
Yms Y
Parallel Sysplex and other
z/VM integrated systems management System-initiated CHPID reconfiguration Program-directed re-IPL Multipath IPL STP enhancements Server Time Protocol Coupling over InfiniBand CHPID type CIB InfiniBand coupling links 12x at a distance of 150 m InfiniBand coupling links 1x at an unrepeated distance of 10 km Dynamic I/O support for InfiniBand CHPIDs CFCC Level 18 CFCC Level 17
Y Y Y Y Y Y Y Ym Y
Y Y Y Y Y Y Y Ym Ym
Y Y Y Y Y Ym Ym N Ym
Y Y Y Y Y Ym Ym N Ym
Y Y N Y
s
Y Y N Ys Ys Ys Ys Yb Yb
m
Y Y N Ys Ys Ys Ys Yb Yb
m
Ys Ys Ys Yb Yb
m
a. A maximum of 32 PUs per system image is supported. Guests can be defined with up to 64 virtual PUs. z/VM V5R4 and above support up to 32 real PUs. b. Support is for guest use only. c. Available for z/OS on virtual machines without virtual zAAPs defined when the z/VM LPAR does not have zAAPs defined. d. 256 GB of central memory are supported by z/VM V5R4 and later. z/VM V5R4 and later are designed to support more than 1 TB of virtual memory in use for guests. e. Web Deliverable required for Pageable 1M Large Page Support. f. 2G Large Page Support is planned as Web Deliverable in first quarter 2013. All statements regarding IBM's plans, directions, and intent are subject to change or withdrawal without notice. g. Not available to guests. h. Support varies by operating system and by version and release. i. FMIDs are shipped in a Web deliverable. j. Crypto Express4S Toleration requires Web deliverable and PTFs. k. Crypto Express4S Exploitation requires Web deliverable. l. Dynamic Reconfiguration Support for Flash Express is planned as Web Deliverable in first quarter 2013. All statements regarding IBM's plans, directions, and intent are subject to change or withdrawal without notice. m. Service is required. n. Support varies with operating system and level. For details see 8.3.37, FCP provides increased performance on page 274.
251
8049ch08.fm
o. FICON Express4 features are withdrawn from marketing. p. Supported for dedicated devices only. q. CHPID type OSN does not use ports. All communication is LPAR to LPAR. r. One port is configured for OSM. The other port in the pair is unavailable. s. Support is for dynamic I/O configuration only.
Table 8-4 EC12 functions minimum support requirements summary, part 2 Function z/VSE V5R1a z/VSE V4R3b z/TPF V1R1 Linux on System z
zEC12 Support of Unified Resource Manager Greater than 64 PUs single system image Greater than 54 PUs single system image Support of IBM zAware zIIP zAAP zAAP on zIIP Java Exploitation of Transactional Execution Large memory (> 128 GB) Large page support Out-of-order execution Guest support for Execute-extensions facility Hardware decimal floating point c Zero address detection 60 logical partitions CPU measurement facility LPAR group capacity limit Separate LPAR management of PUs Dynamic add/delete logical partition name Capacity provisioning Enhanced flexibility for CoD HiperDispatch 63.75 K subchannels Four logical channel subsystems Dynamic I/O support for multiple LCSS Third subchannel set Multiple subchannel sets IPL from alternate subchannel set
Yg N N N N N Y Y N N Y N Y N N N Y N N N N
Yg N N N N N Y Y N N Y N Y N N N Y N N N N
Y N Y Y N Y N Y N N Y N Y N N N N N N N N N N
Y N N Y N Y Y Y Y N Y N Y Y N Y Y Y N Y N
252
8049ch08.fm
Function
z/VSE V5R1a
z/VSE V4R3b
z/TPF V1R1
Linux on System z
MIDAW facility
Cryptography
CPACF CPACF AES-128, AES-192 and AES-256 CPACF SHA-1, SHA-224, SHA-256, SHA-384, SHA-512 CPACF protected key Crypto Express4S Toleration Secure IBM Enterprise PKCS #11 (EP11) coprocessor mode Crypto Express3 Elliptic Curve Cryptography (ECC)
HiperSockets
Y Y Y N Yf N Y N
Y Y Y N N N Y N
Y Yd Ye N Ygh N Ygh N
Y Y Y N Yj N Y Nj
32 Hipersockets HiperSockets Completion Queue HiperSockets integration with IEDN HiperSockets Virtual Switch Bridge HiperSockets Network Traffic Analyzer HiperSockets Multiple Write Facility HiperSockets support of IPV6 HiperSockets Layer 2 support HiperSockets
Flash Express Storage
Y Yg N N N Y N Y
Y N N N N Y N Y
Y N N N N N N N
Y Y N Yi Yj N Y Y Y
Flash Express
Nj
FIber CONnection (FICON) and Fibre Channel Protocol (FCP)
z/OS Discovery and auto configuration (zDAC) FICON Express8S support of zHPF enhanced multitrack CHPID type FC FICON Express8 support of zHPF enhanced multitrack CHPID type FC High Performance FICON for System z (zHPF) FCP - increased performance for small block sizes Request node identification data FICON link incident reporting N_Port ID Virtualization for FICON (NPIV) CHPID type FCP
N N N N Y N Y
N N N N Y N Y
N N N N N N N
N Y N Yk Y N Y
253
8049ch08.fm
Function
z/VSE V5R1a
z/VSE V4R3b
z/TPF V1R1
Linux on System z
FCP point-to-point attachments FICON SAN platform and name registration FCP SAN management SCSI IPL for FCP Cascaded FICON Directors CHPID type FC Cascaded FICON Directors CHPID type FCP FICON Express8S support of hardware data router CHPID type FCP FICON Express8S and FICON Express8 support of T10-DIF CHPID type FCP FICON Express8S, FICON Express8, FICON Express4 10KM LX, and FICON Express4 SX support of SCSI disks CHPID type FCP FICON Express8S c CHPID type FC FICON Express8 c CHPID type FC FICON Express4 10KM LX and SX c CHPID type FC
m
Y Y N Y Y Y N N Y
Y Y N Y Y Y N N Y
N Y N N Y N N N N
Y Y Y Y Y Y Nj Yk Y
Y Y Y
Y Yl Y
Y Yl Y
Y Yl Y
Open Systems Adapter (OSA)
VLAN management VLAN (IEE 802.1q) support QDIO data connection isolation for z/VM virtualized environments OSA Layer 3 Virtual MAC OSA Dynamic LAN idle OSA/SF enhancements for IP, MAC addressing (CHPID=OSD) QDIO Diagnostic Synchronization Network Traffic Analyzer Large send for IPv6 packets Broadcast for IPv4 packets Checksum offload for IPv4 packets OSA-Express4S and OSA-Express3 inbound workload queueing for Enterprise Extender OSA-Express4S 10 Gigabit Ethernet LR and SR CHPID type OSD
N Y N N N N N N N N Y
N N N N N N N N N N Y
N N N N N N N N N N Y
N Y N N N N N Y Y N Y
254
8049ch08.fm
Function
z/VSE V5R1a
z/VSE V4R3b
z/TPF V1R1
Linux on System z
OSA-Express4S 10 Gigabit Ethernet LR and SR CHPID type OSX OSA-Express4S Gigabit Ethernet LX and SX CHPID type OSD (using two ports per CHPID) OSA-Express4S Gigabit Ethernet LX and SX CHPID type OSD (using one port per CHPID) OSA-Express4S 1000BASE-T CHPID type OSC (using one or two ports per CHPID) OSA-Express4S 1000BASE-T CHPID type OSD (using two ports per CHPID) OSA-Express4S 1000BASE-T CHPID type OSD (using one port per CHPID) OSA-Express4S 1000BASE-T CHPID type OSE (using one or two ports per CHPID) OSA-Express4S 1000BASE-T CHPID type OSMp OSA-Express4S 1000BASE-T CHPID type OSNo (using one or two ports per CHPID) OSA-Express3 10 Gigabit Ethernet LR and SR CHPID type OSD OSA-Express3 10 Gigabit Ethernet LR and SR CHPID type OSX OSA-Express3 Gigabit Ethernet LX and SX CHPID types OSD, OSN o (using two ports per CHPID) OSA-Express3 Gigabit Ethernet LX and SX CHPID types OSD, OSN o (using one port per CHPID) OSA-Express3 1000BASE-T CHPID type OSC (using four ports) OSA-Express3 1000BASE-T (using two ports per CHPID) CHPID type OSD OSA-Express3 1000BASE-T (using one port per CHPID) CHPID type OSD OSA-Express3 1000BASE-T (using one or two ports per CHPID) CHPID type OSE OSA-Express3 1000BASE-T Ethernet CHPID type OSN o OSA-Express3 1000BASE-T CHPID type OSM p(using two ports)
Parallel Sysplex and other
Y Y Y Y Y Y Y N Y Y Y Y Y Y Y Y Y Y N
N Y Y Y Y Y Y N Y Y N Y Y Y Y Y Y Y N
Yn Yn Y N Yg Y N N Yg Y N Yn Y Y Yn Y N Y N
Y Y Y Y Y N Y Y Y Yj Y Y Y Y N Y N
255
8049ch08.fm
Function
z/VSE V5R1a
z/VSE V4R3b
z/TPF V1R1
Linux on System z
z/VM integrated systems management System-initiated CHPID reconfiguration Program-directed re-IPL q Multipath IPL STP enhancements Server Time Protocol Coupling over InfiniBand CHPID type CIB InfiniBand coupling links 12x at a distance of 150 m InfiniBand coupling links 1x at unrepeated distance of 10 km Dynamic I/O support for InfiniBand CHPIDs CFCC Level 18 CFCC Level 17
Y -
Y -
Y Y Y
Y Y -
a. z/VSE V5R1 is designed to exploit z/Architecture, specifically 64-bit real and virtual-memory addressing. z/VSE V5R1 requires an architectural level set available with IBM System z9 or later. b. z/VSE V4 is designed to exploit z/Architecture, specifically 64-bit real-memory addressing, but does not support 64-bit virtual-memory addressing. c. Support varies with operating system and level. d. z/TPF supports only AES-128 and AES-256. e. z/TPF supports only SHA-1 and SHA-256 f. Crypto Express4S Exploitation requires PTFs g. Service is required. h. Supported only running in accelerator mode i. Applicable to Guest Operating Systems. j. IBM is working with its Linux distribution partners to include support in future Linux on System z distribution releases. k. Supported by Novell SUSE SLES 11. l. For details see 8.3.37, FCP provides increased performance on page 274. m. FICON Express4 features are withdrawn from marketing. n. Requires PUT4 with PTFs. o. CHPID type OSN does not use ports. All communication is LPAR to LPAR. p. One port is configured for OSM. The other port is unavailable. q. This is for FCP-SCSI disks.
8.3 Support by function

In this section, we discuss operating system support by function. Only the currently in-support releases are covered. Tables in this section use the following convention: N/A NA Not applicable Not available
256
8049ch08.fm
8.3.1 Single system image

A single system image can control several processor units such as CPs, zIIPs, zAAPs, or IFLs, as appropriate.
Maximum number of PUs per system image

Table 8-5 lists the maximum number of PUs supported by each operating system image and by special purpose logical partitions.
Table 8-5 Single system image size software support Operating system Maximum number of PUs per system image
z/OS V1R13 z/OS V1R12 z/OS V1R11 z/OS V1R10 z/VM V6R2 z/VM V6R1 z/VM V5R4 z/VSE V4R3 and above z/TPF V1R1 CFCC Level 18 zAware Linux on System zd
99a 99a 99a 64ab 32c 32 32 z/VSE Turbo Dispatcher can exploit up to 4 CPs and tolerates up to 10-way LPARs 86 CPs 16 CPs or ICFs; CPs and ICFs can not be mixed 80 SUSE SLES 10: 64 CPs or IFLs SUSE SLES 11: 64 CPs or IFLs Red Hat RHEL 5: 80 CPs or IFLs Red Hat RHEL 6: 80 CPs or IFLs
a. The number of purchased zAAPs and the number of purchased zIIPs each cannot exceed the number of purchased CPs. A logical partition can be defined with any number of the available zAAPs and zIIPs. The total refers to the sum of these PU characterizations. b. Service is required. c. When running on a VM-mode LPAR z/VM can manage CPs, IFLs, zAAPs and zIIPS. Otherwise only CPs or IFLs (but not both simultaneously) are supported. d. Values are for z196 support. IBM is working with its Linux distribution partners to provide exploitation of this function in future Linux on System z distribution releases.
The zAware-mode logical partition

zEC12 introduces a new logical partition (LPAR) mode, named zAware-mode, which is exclusive for running the IBM zAware virtual appliance. The IBM zAware virtual appliance can pinpoint deviations in z/OS normal system behavior and improves real-time event diagnostics by monitoring z/OS operations log (OPERLOG) to detect unusual messages, unusual message patterns that typical monitoring systems miss, and unique messages that might indicate system health issues. The IBM zAware virtual appliance requires the monitored clients to run z/OS V1R13 or later.
257
8049ch08.fm
The z/VM-mode logical partition

zEC12 supports a logical partition (LPAR) mode, named z/VM-mode, which is exclusive for running z/VM as the first-level operating system. The z/VM-mode requires z/VM V5R4 or later and allows z/VM to utilize a wider variety of specialty processors in a single LPAR. For instance, in a z/VM-mode LPAR, z/VM can manage Linux on System z guests running on IFL processors while also managing z/VSE and z/OS on central processors (CPs), and allow z/OS to fully exploit IBM System z Integrated Information Processors (zIIPs) and IBM System z Application Assist Processors (zAAPs).
8.3.2 zAAP support

zAAPs do not change the model capacity identifier of the zEC12. IBM software product license charges based on the model capacity identifier are not affected by the addition of zAAPs. On a zEC12, z/OS Version 1 Release 10 is the minimum level for supporting zAAPs, together with current IBM SDKs for z/OS Java 2 Technology Edition Exploiters of zAAPs include: Any Java application that is using the current IBM SDK. WebSphere Application Server V5R1and later, and products based on it, such as WebSphere Portal, WebSphere Enterprise Service Bus (WebSphere ESB), WebSphere Business Integration (WBI) for z/OS and so on. CICS/TS V2R3 and later. DB2 UDB for z/OS Version 8 and later. IMS Version 8 and later. All z/OS XML System Services validation and parsing that execute in TCB mode, which might be eligible for zAAP processing. This eligibility requires z/OS V1R9 and later. For z/OS 1R10 (with appropriate maintenance), middleware and applications requesting z/OS XML System Services can have z/OS XML System Services processing execute on the zAAP. In order to exploit zAAPs, DB2 V9 has the following prerequisites: DB2 V9 for z/OS in new function mode The C API for z/OS XML System Services, available with z/OS V1R9 with rollback APARs to z/OS V1R7, and z/OS V1R8 One of the following items: z/OS V1R92 has native support. z/OS V1R82 requires an APAR for zAAP support. The functioning of a zAAP is transparent to all Java programming on JVM V1.4.1 and later. Use the PROJECTCPU option of the IEAOPTxx parmlib member to help determine whether zAAPs can be beneficial to the installation. Setting PROJECTCPU=YES directs z/OS to record the amount of eligible work for zAAPs and zIIPs in SMF record type 72 subtype 3. Field APPL% AAPCP of the Workload Activity Report listing by WLM service class indicates what percentage of a processor is zAAP eligible. Because of zAAPs lower prices, as compared to CPs, a utilization as low as 10% might provide benefit.
z/OS V1R10 is the minimum z/OS level to support zAAP on zEC12.
258
8049ch08.fm
8.3.3 zIIP support

zIIPs do not change the model capacity identifier of the zEC12. IBM software product license charges based on the model capacity identifier are not affected by the addition of zIIPs. On a zEC12, z/OS Version 1 Release 10 is the minimum level for supporting zIIPs. No changes to applications are required to exploit zIIPs. Exploiters of zIIPs include: DB2 V8 and above for z/OS Data serving, for applications using data DRDA over TCP/IP, such as data serving and data warehousing, and selected utilities z/OS XML services z/OS CIM Server z/OS Communications Server for network encryption (IPSec) and for large messages sent by HiperSockets IBM GBS Scalable Architecture for Financial Reporting z/OS Global Mirror (formerly XRC) and System Data Mover OMEGAMON XE on z/OS, OMEGAMON XE on DB2 Performance Expert and DB2 Performance Monitor The functioning of a zIIP is transparent to application programs. Use the PROJECTCPU option of the IEAOPTxx parmlib member to help determine whether zIIPs can be beneficial to the installation. Setting PROJECTCPU=YES directs z/OS to record the amount of eligible work for zAAPs and zIIPs in SMF record type 72 subtype 3. Field APPL% IIPCP of the Workload Activity Report listing by WLM service class indicates what percentage of a processor is zIIP eligible. Because of zIIPs lower prices, as compared to CPs, a utilization as low as 10% might provide benefit.
8.3.4 zAAP on zIIP capability

This capability, first made available on System z9 servers under defined circumstances, enables workloads eligible to run on Application Assist Processors (zAAPs) to run on Integrated Information Processors (zIIP). It is intended as a means to optimize the investment on existing zIIPs and not as a replacement for zAAPs. The rule of at least one CP installed per zAAP and zIIP installed still applies. Statement of Direction: IBM zEnterprise EC12 is planned to be the last high-end System z server to offer support for zAAP specialty engine processors. IBM intends to continue support for running zAAP workloads on zIIP processors (zAAP on zIIP). This is intended to help simplify capacity planning and performance management, while still supporting all the currently eligible workloads. In addition, IBM plans to provide a PTF for APAR OA38829 on z/OS V1.12 and V1.13 in September 2012 to remove the restriction that prevents zAAP-eligible workloads from running on zIIP processors when a zAAP is installed on the server. This is intended only to help facilitate migration and testing of zAAP workloads on zIIP processors. Exploitation of this capability is by z/OS only, and is only available when zIIPs are installed and one of the following situations occurs: There are no zAAPs installed on the server. z/OS is running as a guest of z/VM V5R4 or later and there are no zAAPs defined to the z/VM LPAR. The server can have zAAPs installed. Because z/VM can dispatch both virtual zAAPs and virtual zIIPs on real CPs3, the z/VM partition does not require any real zIIPs defined to it, although it is best to use real zIIPs due to software licensing reasons.
259
8049ch08.fm
Support is available on z/OS V1R11 and above. This capability is enabled by default (ZAAPZIIP=YES). To disable it, specify NO for the ZAAPZIIP parameter in the IEASYSxx PARMLIB member. On z/OS V1R10 support is provided by PTF for APAR OA27495 and the default setting in the IEASYSxx PARMLIB member is ZAAPZIIP=NO. Enabling or disabling this capability is disruptive. After changing the parameter, z/OS must be re-IPLed for the new setting to take effect.
8.3.5 Transactional Execution

IBM zEnterprise EC12 introduces a new architectural feature called Transactional Execution (TX). This capability is know in academia and industry as hardware transactional memory. This feature enables software to indicate to the hardware the beginning and end of a group of instructions that should be treated in an atomic way: either all of their results happen or none happens, in true transactional style. The execution is optimistic: the hardware provides a memory area to record the original contents of affected registers and memory as the instructions execution takes place. If the transactional execution group is aborted or has to be rolled back, the hardware transactional memory is used to reset the values. Software can implement a fallback capability. This capability enables significantly more efficient software, by providing a way to avoid locks (lock elision). This is of special importance for speculative code generation and highly-parallelized applications. It has been designed to be exploited by the IBM Java Virtual Machine but potentially it can be exploited by other software. z/OS V1R13 with PTFs is required.
8.3.6 Maximum main storage size

Table 8-6 lists the maximum amount of main storage supported by current operating systems. Expanded storage, although part of the z/Architecture, is currently exploited only by z/VM. A maximum of 1 TB of main storage can be defined for a logical partition.
Table 8-6 Maximum memory supported by operating system Operating system Maximum supported main storagea
z/OS z/VM z/VSE z/TPF CFCC zAware Linux on System z (64-bit)
z/OS V1R10 and above support 4 TB and up to 3 TB per servera z/VM V5R4 and above support 256 GB z/VSE V4R3 and above support 32 GB z/TPF supports 4 TBa Level 18 supports up to 3 TB per servera Supports up to 3 TB per servera SUSE SLES 11 supports 4 TBa SUSE SLES 10 supports 4 TBa Red Hat RHEL 5 supports 3 TBa Red Hat RHEL 6 supports 3 TBa
a. zEC12 restricts the maximum LPAR memory size to 1 TB

3
The z/VM system administrator can use the SET CPUAFFINITY command to influence the dispatching of virtual specialty engines on CPs or real specialty engines.
260
8049ch08.fm
8.3.7 Flash Express

IBM zEnterprise EC12 introduces the Flash Express which can help improve resilience and performance of the z/OS system. Flash Express is designed to assist with the handling of workload spikes or increased workload demand that might occur at the opening of the business day, or in the event of a workload shift from one system to another. z/OS is the first exploiter to use Flash Express storage as Storage Class Memory (SCM) for paging store and SVC dump. Flash storage is a faster paging device as compared to hard disk. SVC dump data capture time is expected to be substantially reduced. As a paging store, Flash Express storage is suitable for workloads that can tolerate paging and will not benefit workloads that cannot afford to page. The z/OS design for Flash Express storage does not completely remove the virtual storage constraints created by a paging spike in the system although some scalability relief is expected due to faster paging I/O with Flash Express storage. Flash Express storage is allocated to logical partition similarly to main memory. The initial and maximum amount of Flash Express Storage available to a particular logical partition is specified at the SE or HMC via a new Flash Storage Allocation panel. The Flash Express storage granularity is 16GB. The amount of Flash Express storage in the partition can be changed dynamically, between the initial and the maximum amount at, the SE or HMC. For z/OS, this can also be done via an operator command. Each partition's Flash Express storage is isolated similarly to main storage and sees only its own space in the Flash Storage Space. Flash Express provides 1.6 TB of storage per feature pair, and up to four pairs can be installed, for a total of 6.4 TB. All paging data can easily reside on Flash Express storage but not all types of page data can reside on it. For example, VIO data is always placed on an external disk. Local page data sets are still required to support peak paging demands that require more capacity than provided by the amount of configured SCM. The z/OS paging subsystem works with mix of internal Flash Express storage and External Disk. The placement of data on Flash Express storage and external disk is self tuning, based on measured performance. At IPL time, z/OS detects whether Flash Express storage is assigned to the partition. z/OS automatically uses Flash Express storage for paging unless specified otherwise via PAGESCM=NONE in IEASYSxx. No definition is required for placement of data on Flash Express storage. The support is delivered in the z/OS V1R13 Real storage management (RSM) Enablement Offering Web Deliverable (FMID JBB778H) for z/OS V1R134. The installation of this Web Deliverable requires careful planning as the size of the Nucleus, ESQA per CPU and RSM stack is increased, and there is a new memory pool for Pageable Large Pages. For Web-deliverable code on z/OS, see the z/OS downloads: http://www.ibm.com/systems/z/os/zos/downloads/ Table 8-7 on page 261 list the minimum support requirements for Flash Express
Table 8-7 Minimum support requirements for Flash Express Operating system Support requirements
z/OS
z/OS V1R13a
a. Web delivery and PTFs required

4
Dynamic reconfiguration support for Storage Class Memory (SCM) is planned as Web Deliverable in first quarter 2013. All statements regarding IBM's plans, directions, and intent are subject to change or withdrawal without notice.
261
8049ch08.fm
8.3.8 Large page support

In addition to the existing 1 MB large pages and page frames, zEC12 supports pageable 1MB large pages, large pages 2Gb in size, and large page frames, as described in Large page support on page 104. Table 8-8 lists the support requirements for 1MB large pages.
Table 8-8 Minimum support requirements for 1MB large page Operating system Support requirements
z/OS z/VM z/VSE Linux on System z
z/OS V1R10 z/OS V1R13a for Pageable 1MB large page Not supported; not available to guests z/VSE V4R3; supported for data spaces SUSE SLES 10 SP2 Red Hat RHEL 5.2
a. Web Deliverable and PTFs required.
Table 8-9 lists the support requirements for 2GB large pages.
Table 8-9 Minimum support requirements for 2GB large page Operating system Support requirements
z/OS
z/OS V1R13a
a. IBM plans to support 2GB Large Page in first quarter 2013. All statements regarding IBM's plans, directions, and intent are subject to change or withdrawal without notice.
8.3.9 Guest support for execute-extensions facility

The execute-extensions facility contains several machine instructions. Support is required in z/VM so that guests can exploit this facility. Table 8-10 lists the minimum support requirements.
Table 8-10 Minimum support requirements for execute-extensions facility Operating system Support requirements
z/VM
z/VM V5R4: support is included in the base
8.3.10 Hardware decimal floating point

Industry support for decimal floating point is growing, with IBM leading the open standard definition. Examples of support for the draft standard IEEE 754r include Java BigDecimal, C#, XML, C/C++, GCC, COBOL, and other key software vendors such as Microsoft and SAP. Decimal floating point support was introduced with the z9 EC. zEC12 has inherited the decimal floating point accelerator feature introduced with the z10 EC and described in 3.3.4, Decimal floating point accelerator on page 87. Table 8-11 lists the operating system support for decimal floating point. See also 8.5.7, Decimal floating point and z/OS XL C/C++ considerations on page 296.
262
8049ch08.fm
Table 8-11 Minimum support requirements for decimal floating point Operating system Support requirements
z/OS z/VM Linux on System z
z/OS V1R10: Support includes XL, C/C++, HLASM, Language Environment, DBX, and CDA RTLE. z/VM V5R4: Support is for guest use. SUSE SLES 11 SP1 Red Hat RHEL 6
8.3.11 Up to 60 logical partitions

This feature, first made available in the z9 EC, allows the system to be configured with up to 60 logical partitions. Because channel subsystems can be shared by up to 15 logical partitions, it is necessary to configure four channel subsystems to reach 60 logical partitions. Table 8-12 lists the minimum operating system levels for supporting 60 logical partitions.
Table 8-12 Minimum support requirements for 60 logical partitions Operating system Support requirements
z/OS z/VM z/VSE z/TPF Linux on System z
z/OS V1R10 z/VM V5R4 z/VSE V4R3 z/TPF V1R1 SUSE SLES 10 Red Hat RHEL 5
Note: The IBM zAware virtual appliance runs in a dedicated LPAR so, when activated, it will reduce by one the maximum number of logical partitions available.
8.3.12 Separate LPAR management of PUs

The zEC12 uses separate PU pools for each optional PU type. The separate management of PU types enhances and simplifies capacity planning and management of the configured logical partitions and their associated processor resources. Table 8-13 lists the support requirements for separate LPAR management of PU pools.
Table 8-13 Minimum support requirements for separate LPAR management of PUs Operating system Support requirements
z/OS z/VM z/VSE z/TPF Linux on System z
z/OS V1R10 z/VM V5R4 z/VSE V4R3 z/TPF V1R1 SUSE SLES 10 Red Hat RHEL 5
263
8049ch08.fm
8.3.13 Dynamic LPAR memory upgrade

A logical partition can be defined with both an initial and a reserved amount of memory. At activation time the initial amount is made available to the partition and the reserved amount can be added later, partially or totally. Those two memory zones do not have to be contiguous in real memory but appear as logically contiguous to the operating system running in the LPAR. z/OS is able to take advantage of this support and nondisruptively acquire and release memory from the reserved area. z/VM V5R4 and higher are able to acquire memory nondisruptively, and immediately make it available to guests. z/VM virtualizes this support to its guests, which now can also increase their memory nondisruptively, if supported by the guest operating system. Releasing memory from z/VM is a disruptive operation to z/VM. Releasing memory from the guest depends on the guests operating system support. Dynamic LPAR memory upgrade is not supported for IBM zAware-mode LPARs.
8.3.14 Capacity Provisioning Manager

The provisioning architecture, described in 9.8, Nondisruptive upgrades on page 347, enables customers to better control the configuration and activation of the On/Of Capacity on Demand. The new process is inherently more flexible and can be automated. This capability can result in easier, faster, and more reliable management of the processing capacity. The Capacity Provisioning Manager, a function first available with z/OS V1R9, interfaces with z/OS Workload Manager (WLM) and implements capacity provisioning policies. Several implementation options are available: from an analysis mode that only issues guidelines, to an autonomic mode providing fully automated operations. Replacing manual monitoring with autonomic management or supporting manual operation with guidelines, can help ensure that sufficient processing power is available with the least possible delay. Support requirements are listed on Table 8-14.
Table 8-14 Minimum support requirements for capacity provisioning Operating system Support requirements
z/OS z/VM
z/OS V1R10 Not supported; not available to guests
8.3.15 Dynamic PU add

z/OS has long been able to define reserved PUs to an LPAR for the purpose of non-disruptively bringing online the additional computing resources when needed. Starting with z/OS V1R10, z/VM V5R4, and z/VSE V4R3, an enhanced capability, the ability to dynamically define and change the number and type of reserved PUs in an LPAR profile can be used for that purpose. No pre-planning is required. The new resources are immediately made available to the operating system and, in the z/VM case, to its guests. Dynamic PU add is not supported for IBM zAware-mode LPARs.
264
8049ch08.fm
8.3.16 HiperDispatch
HiperDispatch, which is exclusive to zEC12, z196 and System z10, represents a cooperative effort between the z/OS operating system and the zEC12 hardware. It improves efficiencies in both the hardware and the software in the following ways: Work can be dispatched across fewer logical processors, therefore reducing the multiprocessor (MP) effects and lowering the interference across multiple partitions. Specific z/OS tasks can be dispatched to a small subset of logical processors that Processor Resource/Systems Manager (PR/SM) then ties to the same physical processors, thus improving the hardware cache reuse and locality of reference characteristics, and reducing the rate of cross-book communications. For more information, see 3.6, Logical partitioning on page 107. Table 8-15 lists HiperDispatch support requirements.
Table 8-15 Minimum support requirements for HiperDispatch Operating system Support requirements
z/OS z/VM
z/OS V1R10 and later with PTFs Not supported; not available to guests
8.3.17 The 63.75 K Subchannels

Servers prior to the z9 EC reserved 1024 subchannels for internal system use out of the maximum of 64 K subchannels. Starting with the z9 EC, the number of reserved subchannels has been reduced to 256, thus increasing the number of subchannels available. Reserved subchannels exist only in subchannel set 0. No subchannels are reserved in subchannel sets 1 and 2. The informal name, 63.75 K subchannels, represents 65280 subchannels, as shown in the following equation: 63 x 1024 + 0.75 x 1024 = 65280 Table 8-16 lists the minimum operating system level required on zEC12.
Table 8-16 Minimum support requirements for 63.75 K subchannels Operating system Support requirements
z/OS V1R10 z/VM V5R4 SUSE SLES 10 Red Hat RHEL 5
8.3.18 Multiple Subchannel Sets

Multiple subchannel sets, first introduced in z9 EC, provide a mechanism for addressing more than 63.75 K I/O devices and aliases for ESCON5 (CHPID type CNC) and FICON (CHPID types FC) on the zEC12, z196, z10 EC, and z9 EC. z196 introduced the third subchannel set (SS2). Multiple subchannel sets are not supported for z/OS when running as a guest of z/VM.
5
ESCON features are not supported on the zEC12.
265
8049ch08.fm
Table 8-17 lists the minimum operating systems level required on the zEC12.
Table 8-17 Minimum software requirement for MSS Operating system Support requirements
z/OS Linux on System z
z/OS V1R10 SUSE SLES 10 Red Hat RHEL 5
8.3.19 Third Subchannel Set

With z196 a third subchannel set (SS2) is introduced. It applies to ESCON5 (CHPID type CNC) and FICON (CHPID type FC for both FICON and zHPF paths) channels. Together with the second set (SS1) it can be used for disk alias devices of both primary and secondary devices, and as Metro Mirror secondary devices. This should help facilitate storage growth and complements other functions such as EAV and HyperPAV. Table 8-18 lists the minimum operating systems level required on the zEC12.
Table 8-18 Minimum software requirement for SS2 Operating system Support requirements
z/OS
z/OS V1R10 with PTFs
8.3.20 IPL from an alternate subchannel set

zEC12 supports IPL from subchannel set 1 (SS1) or subchannel set 2 (SS2), in addition to subchannel set 0. For further information refer, IPL from an alternate subchannel set on page 173.
8.3.21 MIDAW facility

The modified indirect data address word (MIDAW) facility improves FICON performance. The MIDAW facility provides a more efficient CCW/IDAW structure for certain categories of data-chaining I/O operations. Support for the MIDAW facility when running z/OS as a guest of z/VM requires z/VM V5R4 or higher. See 8.7, MIDAW facility on page 299. Table 8-19 lists the minimum support requirements for MIDAW.
Table 8-19 Minimum support requirements for MIDAW Operating system Support requirements
z/OS z/VM
z/OS V1R10 z/VM V5R4 for guest exploitation
8.3.22 HiperSockets Completion Queue

The HiperSockets Completion Queue is exclusive to zEC12, z196 and z114. The Completion Queue function is designed to allow HiperSockets to transfer data synchronously if possible 266
8049ch08.fm
and asynchronously if necessary, thus combining ultra-low latency with more tolerance for traffic peaks. This could be especially helpful in burst situations. HiperSockets Completion Queue is planned to be supported in the z/VM environments. Table 8-20 lists the minimum support requirements for HiperSockets Completion Queue.
Table 8-20 Minimum support requirements for HiperSockets Completion Queue Operating system Support requirements
z/OS z/VSE Linux on System z a. PTFs required
z/OS V1R13a z/VSE V5R1a SLES 11 SP2 Red Hat RHEL 6.2
8.3.23 HiperSockets integration with the intraensemble data network (IEDN)

The HiperSockets integration with IEDN is exclusive to zEC12, z196 and z114. HiperSockets integration with IEDN combines HiperSockets network and the physical IEDN to appear as a single Layer 2 network. This will extend the reach of the HiperSockets network outside the CPC to the entire ensemble, appearing as a single Layer 2 network. Table 8-21 lists the minimum support requirements for HiperSockets integration with IEDN.
Table 8-21 Minimum support requirements for HiperSockets integration with IEDN Operating system Support requirements
z/OS z/VM a. PTFs required
z/OS V1R13a z/VM V6R2a
8.3.24 HiperSockets Virtual Switch Bridge

The HiperSockets Virtual Switch Bridge is exclusive to zEC12, z196 and z114. HiperSockets Virtual Switch Bridge can integrate with the intraensemble data network (IEDN) through OSX adapters and bridge to another central processor complex (CPC) through OSD adapters, extending the reach of the HiperSockets network outside of the CPC to the entire ensemble and hosts external to the CPC, appearing as a single Layer 2 network. Table 8-22 lists the minimum support requirements for HiperSockets Virtual Switch Bridge.
Table 8-22 Minimum support requirements for HiperSockets Virtual Switch Bridge Operating system Support requirements
z/VM Linux on System zb
z/VM V6R2a SLES 10 SP4 update (kernel 2.6.16.60-0.95.1) Red Hat RHEL 5.8 (GA-level)
a. PTFs required b. Applicable to Guest Operating Systems
267
8049ch08.fm
8.3.25 HiperSockets Multiple Write Facility

This capability allows the streaming of bulk data over a HiperSockets link between two logical partitions. Multiple output buffers are supported on a single SIGA write instruction.The key advantage of this enhancement is that it allows the receiving logical partition to process a much larger amount of data per I/O interrupt. This is transparent to the operating system in the receiving partition. HiperSockets Multiple Write Facility with fewer I/O interrupts is designed to reduce CPU utilization of the sending and receiving partitions. Support for this function is required by the sending operating system. See 4.9.6, HiperSockets on page 159. Table 8-23 lists the minimum support requirements for HiperSockets Virtual Multiple Write Facility.
Table 8-23 Minimum support requirements for HiperSockets multiple write Operating system Support requirements
z/OS
z/OS V1R10
8.3.26 HiperSockets IPv6

IPv6 is expected to be a key element in future networking. The IPv6 support for HiperSockets permits compatible implementations between external networks and internal HiperSockets networks. Table 8-24 lists the minimum support requirements for HiperSockets IPv6 (CHPID type IQD).
Table 8-24 Minimum support requirements for HiperSockets IPv6 (CHPID type IQD) Operating system Support requirements
z/OS V1R10 z/VM V5R4 SUSE SLES 10 SP2 Red Hat RHEL 5.2
8.3.27 HiperSockets Layer 2 support

For flexible and efficient data transfer for IP and non-IP workloads, the HiperSockets internal networks on zEC12 can support two transport modes, which are Layer 2 (Link Layer) and the current Layer 3 (Network or IP Layer). Traffic can be Internet Protocol (IP) Version 4 or Version 6 (IPv4, IPv6) or non-IP (AppleTalk, DECnet, IPX, NetBIOS, or SNA). HiperSockets devices are protocol-independent and Layer 3 independent. Each HiperSockets device has its own Layer 2 Media Access Control (MAC) address, which allows the use of applications that depend on the existence of Layer 2 addresses such as Dynamic Host Configuration Protocol (DHCP) servers and firewalls. Layer 2 support can help facilitate server consolidation. Complexity can be reduced, network configuration is simplified and intuitive, and LAN administrators can configure and maintain the mainframe environment the same way as they do a non-mainframe environment. Table 8-25 on page 269 show the minimum support requirements for HiperSockets Layer 2.
268
8049ch08.fm
Table 8-25 Minimum support requirements for HiperSockets Layer 2 Operating system Support requirements
z/VM Linux on System z
z/VM V5R4 for guest exploitation SUSE SLES 10 SP2 Red Hat RHEL 5.2
8.3.28 HiperSockets network traffic analyzer for Linux on System z

HiperSockets network traffic analyzer (HS NTA), introduced with the z196, is an enhancement to HiperSockets architecture, with support to trace Layer2 and Layer3 HiperSockets network traffic in Linux on System z. This allows Linux on System z to control the trace for the internal virtual LAN, to capture the records into host memory and storage (file systems). Linux on System z tools can be used to format, edit, and process the trace records for analysis by system programmers and network administrators.
8.3.29 FICON Express8S

The FICON Express8S feature resides exclusively in the PCIe I/O drawer and provides a link rate of 8 Gbps, with auto negotiation to 4 or 2 Gbps, for compatibility with previous devices and investment protection. Both 10KM LX and SX connections are offered (in a given feature all connections must have the same type). With FICON Express 8S, you might be able to consolidate existing FICON, FICON Express26, and FICON Express46 channels, while maintaining and enhancing performance. FICON Express8S introduced a hardware data router for more efficient zHPF data transfers becoming the first channel with hardware specifically designed to support zHPF, as contrasted to FICON Express8, FICON Express46 , and FICON Express26 which have a firmware only zHPF implementation. Table 8-26 lists the minimum support requirements for FICON Express8S.
Table 8-26 Minimum support requirements for FICON Express8S Operating system z/OS z/VM z/VSE z/TPF Linux on System z
Native FICON and Channel-to-Channel (CTC) CHPID type FC zHPF single track operations CHPID type FC zHPF multitrack operations CHPID type FC
V1R10a
V5R4
V4R3
V1R1
SUSE SLES 10 Red Hat RHEL 5 SUSE SLES 11 SP1 Red Hat RHEL 6 SUSE SLES 11 SP2 Red Hat RHEL 6.1 SUSE SLES 10 Red Hat RHEL 5
V1R10b
V6R2b
N/A
N/A
V1R10b
V6R2b
N/A
N/A
Support of SCSI devices CHPID type FCP
N/A
V5R4b
V4R3
N/A
All FICON Express4, FICON Express2 and FICON features are withdrawn from marketing.
269
8049ch08.fm
Operating system
z/OS
z/VM
z/VSE
z/TPF
Linux on System z
Support of hardware data router CHPID type FCP Support of T10-DIF CHPID type FCP
N/A
N/A
N/A
N/A
N/Ac
N/A
V5R4b
N/A
N/A
SUSE SLES 11 SP2c
a. PTFs required to support GRS Ficon CTC toleration. b. PTFs required c. IBM is working with its Linux distribution partners to provide exploitation of this function in future Linux on System z distribution releases.
8.3.30 FICON Express8

The FICON Express8 features provide a link rate of 8 Gbps, with auto negotiation to 4 or 2 Gbps, for compatibility with previous devices and investment protection. Both 10KM LX and SX connections are offered (in a given feature all connections must have the same type). With FICON Express 8, customers might be able to consolidate existing FICON, FICON Express26 , and FICON Express46 channels, while maintaining and enhancing performance. Table 8-27 lists the minimum support requirements for FICON Express8.
Table 8-27 Minimum support requirements for FICON Express8 Operating system z/OS z/VM z/VSE z/TPF Linux on System z
Native FICON and Channel-to-Channel (CTC) CHPID type FC zHPF single track operations CHPID type FC zHPF multitrack operations CHPID type FC Support of SCSI devices CHPID type FCP Support of T10-DIF CHPID type FCP
V1R10
V5R4
V4R3
V1R1
SUSE SLES 10 Red Hat RHEL 5 N/Ab
V1R10a
N/A
N/A
N/A
V1R10a N/A N/A
N/A V5R4a V5R4a
N/A V4R3 N/A
N/A N/A N/A
N/A SUSE SLES 10 Red Hat RHEL 5 SUSE SLES 11 SP2b
a. PTFs required b. IBM is working with its Linux distribution partners to provide exploitation of this function in future Linux on System z distribution releases.
8.3.31 z/OS discovery and autoconfiguration (zDAC)

z/OS discovery and autoconfiguration for FICON channels (zDAC) is designed to automatically perform a number of I/O configuration definition tasks for new and changed disk and tape controllers connected to a switch or director, when attached to a FICON channel. The zDAC function is integrated into the existing Hardware Configuration Definition (HCD). Customers can define a policy which can include preferences for availability and bandwidth
270
8049ch08.fm
including parallel access volume (PAV) definitions, control unit numbers, and device number ranges. Then, when new controllers are added to an I/O configuration or changes are made to existing controllers, the system is designed to discover them and propose configuration changes based on that policy. zDAC provides real-time discovery for the FICON fabric, subsystem and I/O device resource changes from z/OS. By exploring the discovered control units for defined logical control units (LCU) and devices, zDAC compares the discovered controller information with the current system configuration to determine delta changes to the configuration for a proposed configuration. All new added or changed logical control units and devices will be added into the proposed configuration, with proposed control unit and device numbers, and channel paths based on the defined policy. zDAC uses channel path chosen algorithms to minimize single points of failure. The zDAC proposed configurations are created as work I/O definition files (IODF), that can be converted to production IODF and activated. zDAC is designed to perform discovery for all systems in a sysplex that support the function. Thus, zDAC helps simplifying I/O configuration on zEC12 systems running z/OS and reduces complexity and setup time. zDAC applies to all FICON features supported on zEC12 when configured as CHPID type FC. Table 8-28 lists the minimum support requirements for zDAC.
Table 8-28 Minimum support requirements for zDAC Operating system Support requirements
z/OS a. PTFs required
z/OS V1R12a
8.3.32 High performance FICON (zHPF)

High performance FICON (zHPF), first provided on System z10, is a FICON architecture for protocol simplification and efficiency, reducing the number of information units (IUs) processed. Enhancements have been made to the z/Architecture and the FICON interface architecture to provide optimizations for on line transaction processing (OLTP) workloads. When exploited by the FICON channel, the z/OS operating system, and the DS8000 control unit or other subsystems (appropriate levels of Licensed Internal Code are required) the FICON channel overhead can be reduced and performance can be improved. Additionally, the changes to the architectures provide end-to-end system enhancements to improve reliability, availability, and serviceability (RAS). zHPF is compatible with these standards: Fibre Channel Physical and Signaling standard (FC-FS) Fibre Channel Switch Fabric and Switch Control Requirements (FC-SW) Fibre Channel Single-Byte-4 (FC-SB-4) standards The zHPF channel programs can be exploited, for instance, by z/OS OLTP I/O workloads; DB2, VSAM, PDSE and zFS. At announcement zHPF supported the transfer of small blocks of fixed size data (4 K) from a single track. This has been extended first to 64k bytes then to multitrack operations. The 64k byte data transfer limit on multitrack operations was removed by z196. This improvement
271
8049ch08.fm
allows the channel to fully exploit the bandwidth of FICON channels, resulting in higher throughputs and lower response times. On the zEC12 and z196 the multitrack operations extension applies exclusively to the FICON Express8S, FICON Express8, and FICON Express47, when configured as CHPID type FC, and connecting to z/OS. zHPF requires matching support by the DS8000 series, otherwise the extended multitrack support is transparent to the control unit. From the z/OS point of view, the existing FICON architecture is called command mode and zHPF architecture is called transport mode. During link initialization, the channel node and the control unit node indicate whether they support zHPF. Attention: All FICON channel paths (CHPIDs) defined to the same Logical Control Unit (LCU) must support zHPF. The inclusion of any non-compliant zHPF features in the path group will cause the entire path group to support command mode only. The mode used for an I/O operation depends on the control unit supporting zHPF and settings in the z/OS operating system. For z/OS exploitation there is a parameter in the IECIOSxx member of SYS1.PARMLIB (ZHPF=YES or NO) and in the SETIOS system command to control whether zHPF is enabled or disabled. The default is ZHPF=NO. Support is also added for the D IOS,ZHPF system command to indicate whether zHPF is enabled, disabled, or not supported on the server. Similar to the existing FICON channel architecture, the application or access method provides the channel program (channel command words, CCWs). The way that zHPF (transport mode) manages channel program operations is significantly different from the CCW operation for the existing FICON architecture (command mode). While in command mode, each single CCW is sent to the control unit for execution. In transport mode, multiple channel commands are packaged together and sent over the link to the control unit in a single control block. Less overhead is generated compared to the existing FICON architecture. Certain complex CCW chains are not supported by zHPF. The zHPF is exclusive to zEC12, z196, z114 and System z10. The FICON Express8S, FICON Express8, and FICON Express47,8 (CHPID type FC) concurrently support both the existing FICON protocol and the zHPF protocol in the server Licensed Internal Code. Table 8-29 lists the minimum support requirements for zHPF
Table 8-29 Minimum support requirements for zHPF Operating system Support requirements
z/OS
Single track operations: z/OS V1R10 with PTFs Multitrack operations: z/OS V1R10 with PTFs 64K enhancement: z/OS V1R10 with PTFs Not supported; not available to guests SLES 11 SP1 supports zHPF. IBM continues to work with its Linux distribution partners on exploitation of appropriate zEC12 functions be provided in future Linux on System z distribution releases.
z/VM Linux on System z
For more information about FICON channel performance, see the performance technical papers on the System z I/O connectivity web site:
7 8
FICON Express4 LX 4KM is not support on zEC12 All FICON Express4, FICON Express2 and FICON features are withdrawn from marketing.
272
8049ch08.fm
http://www-03.ibm.com/systems/z/hardware/connectivity/ficon_performance.html
8.3.33 Request node identification data

First offered on z9 EC, the request node identification data (RNID) function for native FICON CHPID type FC allows isolation of cabling-detected errors. Table 8-30 lists the minimum support requirements for RNID.
Table 8-30 Minimum support requirements for RNID Operating system Support requirements
z/OS
z/OS V1R10
8.3.34 Extended distance FICON

An enhancement to the industry standard FICON architecture (FC-SB-3) helps avoid degradation of performance at extended distances by implementing a new protocol for persistent information unit (IU) pacing. Extended distance FICON is transparent to operating systems and applies to all the FICON Express 8S, FICON Express8, and FICON Express48 features carrying native FICON traffic (CHPID type FC). For exploitation, the control unit must support the new IU pacing protocol. IBM System Storage DS8000 series supports extended distance FICON for IBM System z environments. The channel defaults to current pacing values when it operates with control units that cannot exploit extended distance FICON.
8.3.35 Platform and name server registration in FICON channel

The FICON Express8S, FICON Express8, and FICON Express49 features on the zEC12 servers support platform and name server registration to the fabric for both CHPID type FC and FCP. Information about the channels connected to a fabric, if registered, allows other nodes or storage area network (SAN) managers to query the name server to determine what is connected to the fabric. The following attributes are registered for the zEC12 servers: Platform information Channel information World Wide Port Name (WWPN) Port type (N_Port_ID) FC-4 types supported Classes of service supported by the channel The platform and name server registration service are defined in the Fibre Channel - Generic Services 4 (FC-GS-4) standard.
FICON Express4 LX 4KM is not support on zEC12
273
8049ch08.fm
8.3.36 FICON link incident reporting

FICON link incident reporting allows an operating system image (without operator intervention) to register for link incident reports. Table 8-31 lists the minimum support requirements for this function.
Table 8-31 Minimum support requirements for link incident reporting Operating system Support requirements
z/OS
z/OS V1R10
8.3.37 FCP provides increased performance

The Fibre Channel Protocol (FCP) Licensed Internal Code has been modified to help provide increased I/O operations per second for both small and large block sizes and to support 8 Gbps link speeds. For more information about FCP channel performance, see the performance technical papers on the System z I/O connectivity web site: http://www-03.ibm.com/systems/z/hardware/connectivity/fcp_performance.html
8.3.38 N_Port ID virtualization

N_Port ID virtualization (NPIV) provides a way to allow multiple system images (in logical partitions or z/VM guests) to use a single FCP channel as though each were the sole user of the channel. This feature, first introduced with z9 EC, can be used with earlier FICON features that have been carried forward from earlier servers. Table 8-32 lists the minimum support requirements for NPIV.
Table 8-32 Minimum support requirements for NPIV Operating system Support requirements
z/VM
z/VM V5R4 provides support for guest operating systems and VM users to obtain virtual port numbers. Installation from DVD to SCSI disks is supported when NPIV is enabled. z/VSE V4R3 SUSE SLES 10 SP3 Red Hat RHEL 5.4
z/VSE Linux on System z
8.3.39 OSA-Express4S 1000BASE-T Ethernet

The OSA-Express4S 1000BASE-T Ethernet feature resides exclusively in the PCIe I/O drawer. Each feature has one PCIe adapter and two ports. The two ports share a channel path identifier, defined as one of OSC, OSD, OSE, OSM or OSN. The ports can be defined as a spanned channel and can be shared among logical partitions and across logical channel subsystems. The OSM CHPID type was introduced with z196, see 8.3.47, Intranode management network (INMN) on page 280. Each adapter can be configured in the following modes: QDIO mode, with CHPID types OSD and OSN 274
8049ch08.fm
Non-QDIO mode, with CHPID type OSE Local 3270 emulation mode, including OSA-ICC, with CHPID type OSC Ensemble management, with CHPID type OSM. Operating system support is required in order to recognize and use the second port on the OSA-Express4S 1000BASE-T feature. Table 8-33 lists the minimum support requirements for OSA-Express4S 1000BASE-T.
Table 8-33 Minimum support requirements for OSA-Express4S 1000BASE-T Ethernett Operating system Support requirements exploiting two ports per CHPID Support requirements exploiting one port per CHPID
z/OS z/VM
OSC, OSD, OSE, OSNb : z/OS V1R10a OSC: z/VM V5R4 OSD: z/VM V5R4a OSE: z/VM V5R4 OSNb: z/VM V5R4 OSC, OSD, OSE, OSNb : z/VSE V4R3 OSD: z/TPF V1R1 PUT 4a OSNb : z/TPF V1R1 PUT 4a OSD OSD: SUSE SLES 10 SP2 Red Hat RHEL 5.2 OSNb : SUSE SLES 10 Red Hat RHEL 5
OSC, OSD, OSE, OSM, OSNb : z/OS V1R10a OSC: z/VM V5R4 OSD: z/VM V5R4 OSE: z/VM V5R4 OSM: z/VM V5R4a OSNb : z/VM V5R4 OSC, OSD, OSE, OSNb : z/VSE V4R3 OSD: z/TPF V1R1 OSNb : z/TPF V1R1 PUT4a OSD OSD: SUSE SLES 10 Red Hat RHEL 5 OSM: SUSE SLES 10 SP4 Red Hat RHEL 5.6 OSNb : SUSE SLES 10 Red Hat RHEL 5
z/VSE z/TPF IBM zAware Linux on System z
a. PTFs required b. Although CHPID type OSN does not use any ports (because all communication is LPAR to LPAR), it is listed here for completeness.
8.3.40 OSA-Express4S 10 Gigabit Ethernet LR and SR

The OSA-Express4S 10 Gigabit Ethernet feature, introduced with the zEC12, resides exclusively in the PCIe I/O drawer. Each feature has one port, defined as either CHPID type OSD or OSX. CHPID type OSD supports the queued direct input/output (QDIO) architecture for high-speed TCP/IP communication. The z196 introduced the CHPID type OSX, see 8.3.48, Intraensemble data network (IEDN) on page 280. The OSA-Express4S features have half the number of ports per feature, compared to OSA-Express3, and half the size as well. This actually results in an increased number of installable features while facilitating the purchase of the right number of ports to help satisfy your application requirements and to better optimize for redundancy. Table 8-34 lists the minimum support requirements for OSA-Express4S 10 Gigabit Ethernet LR and SR features.
275
8049ch08.fm
Table 8-34 Minimum support requirements for OSA-Express4S 10 Gigabit Ethernet LR and SR Operating system Support requirements
z/OS z/VM z/VSE z/TPF IBM zAware Linux on System z a. PTFs required
OSD: z/OS V1R10a OSX: z/OS V1R10a OSD: z/VM V5R4 OSX: z/VM V5R4a for dynamic I/O only OSD: z/VSE V4R3 OSX: z/VSE V5R1 OSD: z/TPF V1R1 OSX: z/TPF V1R1 PUT4a OSD OSX OSD: SUSE SLES 10, Red Hat RHEL 5 OSX: SUSE SLES 10 SP4, Red Hat RHEL 5.6
8.3.41 OSA-Express4S Gigabit Ethernet LX and SX

The OSA-Express4S Gigabit Ethernet feature resides exclusively in the PCIe I/O drawer. Each feature has one PCIe adapter and two ports. The two ports share a channel path identifier (CHPID type OSD exclusively). Each port supports attachment to a one Gigabit per second (Gbps) Ethernet Local Area Network (LAN). The ports can be defined as a spanned channel and can be shared among logical partitions and across logical channel subsystems. Operating system support is required in order to recognize and use the second port on the OSA-Express4S Gigabit Ethernet feature. Table 8-35 lists the minimum support requirements for OSA-Express4S Gigabit Ethernet LX and SX.
Table 8-35 Minimum support requirements for OSA-Express4S Gigabit Ethernet LX and SX Operating system Support requirements exploiting two ports per CHPID Support requirements exploiting one port per CHPID
z/OS z/VM z/VSE z/TPF IBM zAware Linux on System z a. PTFs required
OSD: z/OS V1R10a OSD: z/VM V5R4a OSD: z/VSE V4R3 OSD: z/TPF V1R1 PUT 4a OSD OSD: SUSE SLES 10 SP2 Red Hat RHEL 5.2
OSD: z/OS V1R10a OSD: z/VM V5R4 OSD: z/VSE V4R3 OSD: z/TPF V1R1
OSD: SUSE SLES 10 Red Hat RHEL 5
8.3.42 OSA-Express3 10 Gigabit Ethernet LR and SR

The OSA-Express3 10 Gigabit Ethernet features offer two ports, defined as CHPID type OSD or OSX. CHPID type OSD supports the queued direct input/output (QDIO) architecture for high-speed TCP/IP communication. The z196 introduced the CHPID type OSX, see 8.3.48, Intraensemble data network (IEDN) on page 280. 276
8049ch08.fm
Table 8-36 lists the minimum support requirements for OSA-Express3 10 Gigabit Ethernet LR and SR features.
Table 8-36 Minimum support requirements for OSA-Express3 10 Gigabit Ethernet LR and SR Operating system Support requirements
z/OS z/VM z/VSE z/TPF IBM zAware Linux on System z
OSD: z/OS V1R10 OSX: z/OS V1R12, z/OS V1R10 and z/OS V1R11, with service OSD: z/VM V5R4 OSX: z/VM V5R4 for dynamic I/O only; z/VM V6R1 with service OSD: z/VSE V4R3 OSD: z/TPF V1R1 OSD OSD: SUSE SLES 10 OSD: Red Hat RHEL 5
8.3.43 OSA-Express3 Gigabit Ethernet LX and SX

The OSA-Express3 Gigabit Ethernet features offer two cards with two PCI Express adapters each. Each PCI Express adapter controls two ports, giving a total of four ports per feature. Each adapter has its own CHPID, defined as either OSD or OSN, supporting the queued direct input/output (QDIO) architecture for high-speed TCP/IP communication. Thus, a single feature can support both CHPID types, with two ports for each type. Operating system support is required in order to recognize and use the second port on each PCI Express adapter. Minimum support requirements for OSA-Express3 Gigabit Ethernet LX and SX features are listed in Table 8-37 (four ports) and Table 8-38 (two ports).
Table 8-37 Minimum support requirements for OSA-Express3 Gigabit Ethernet LX and SX, four ports Operating system Support requirements
z/OS V1R10; service required z/VM V5R4; service required z/VSE V4R3 z/TPF V1R1; service required
SUSE SLES 10 SP2 Red Hat RHEL 5.2
Table 8-38 Minimum support requirements for OSA-Express3 Gigabit Ethernet LX and SX, two ports Operating system Support requirements
z/OS z/VM z/VSE z/TPF
z/OS V1R10 z/VM V5R4 z/VSE V4R3 z/TPF V1R1
277
8049ch08.fm
Operating system
Support requirements
IBM zAware Linux on System z
OSD SUSE SLES 10 Red Hat RHEL 5
8.3.44 OSA-Express3 1000BASE-T Ethernet

The OSA-Express3 1000BASE-T Ethernet features offer two cards with two PCI Express adapters each. Each PCI Express adapter controls two ports, giving a total of four ports for each feature. Each adapter has its own CHPID, defined as one of OSC, OSD, OSE, OSM or OSN. A single feature can support two CHPID types, with two ports for each type. The OSM CHPID type is introduced with the z196, see 8.3.47, Intranode management network (INMN) on page 280. Each adapter can be configured in the following modes: QDIO mode, with CHPID types OSD and OSN Non-QDIO mode, with CHPID type OSE Local 3270 emulation mode, including OSA-ICC, with CHPID type OSC Ensemble management, with CHPID type OSM. Operating system support is required in order to recognize and use the second port on each PCI Express adapter. Minimum support requirements for OSA-Express3 1000BASE-T Ethernet feature are listed in Table 8-39 (four ports) and Table 8-40.
Table 8-39 Minimum support requirements for OSA-Express3 1000BASE-T Ethernet, four ports Operating system Support requirementsa b
z/OS
OSD: z/OS V1R10; service required OSE: z/OS V1R10 OSM: z/OS V1R12, z/OS V1R10 and z/OS V1R11, with service OSNb : z/OS V1R10 OSD: z/VM V5R4; service required OSE: z/VM V5R4 OSM: z/VM service required; V5R4 for dynamic I/O only OSNb : z/VM V5R4 OSD: z/VSE V4R3; service required OSE: z/VSE V4R3 OSNb : z/VSE V4R3 OSD and OSNb : z/TPF V1R1; service required OSD OSD: SUSE SLES 10 SP2 Red Hat RHEL 5.2 OSNb : SUSE SLES 10 SP2 Red Hat RHEL 5.2
z/VM
z/VSE
z/TPF IBM zAware Linux on System z
a. Applies to CHPID types OSC, OSD, OSE, OSM and OSN. For support, see Table 8-40 on page 279. b. Although CHPID type OSN does not use any ports (because all communication is LPAR to LPAR), it is listed here for completeness.
278
8049ch08.fm
Table 8-40 lists the minimum support requirements for OSA-Express3 1000BASE-T Ethernet (two ports).
Table 8-40 Minimum support requirements for OSA-Express3 1000BASE-T Ethernet, two ports Operating system Support requirements
OSD, OSE, OSM and OSN: V1R10 OSD, OSE, OSM and OSN: V5R4 V4R3 OSD, OSN, and OSC: V1R1 OSD OSD: SUSE SLES 10 Red Hat RHEL 5 OSN: SUSE SLES 10 SP3 Red Hat RHEL 5.4
8.3.45 Open Systems Adapter for IBM zAware

The IBM zAware server requires connections to graphical user interface (GUI) browser and z/OS monitored clients. An OSA channel is the most logical choice for allowing GUI browser connections to the server, so users can view the analytical data for the monitored clients through the IBM zAware GUI. For z/OS monitored clients connecting an IBM zAware server, one of the following network options is supported: A customer-provided data network provided via an OSA Ethernet channel. A HiperSockets subnetwork within the zEC12. Intraensemble data network (IEDN) on the zEC12 to other CPC nodes in the ensemble. The zEC12 server also supports the use of HiperSockets over the IEDN.
8.3.46 Open Systems Adapter for Ensemble

Five different OSA-Express4S features are used to connect the zEC12 to its attached zEnterprise BladeCenter Extension (zBX) Model 003, and other ensemble nodes. These connections are part of the ensembles two private and secure internal networks. For the intranode management network (INMN): OSA Express4S 1000BASE-T Ethernet, feature code 0408 For the intraensemble data network (IEDN): OSA-Express4S Gigabit Ethernet (GbE) LX, feature code 0404 OSA-Express4S Gigabit Ethernet (GbE) SX, feature code 0405 OSA-Express4S 10 Gigabit Ethernet (GbE) Long Range (LR), feature code 0406 OSA-Express4S 10 Gigabit Ethernet (GbE) Short Reach (SR), feature code 0407 For detailed information about OSA-Express4S in an ensemble network, see 7.4, zBX connectivity on page 226.
279
8049ch08.fm
8.3.47 Intranode management network (INMN)

The intranode management network (INMN) is one of the ensembles two private and secure internal networks. The INMN is used by the Unified Resource Manager functions. The INMN is a private and physically isolated 1000Base-T ethernet internal platform management network, operating at 1 Gbps, that connects all resources (CPC and zBX components) of an ensemble node, for management purposes. It is pre-wired, internally switched, configured, and managed with fully redundancy for high availability. The z196 introduced the OSA Direct-Express Management (OSM) CHPID type. INMN requires two OSA Express4S 1000BASE-T ports, from two different OSA-Express4S 1000Base-T features, configured as CHPID type OSM. One port per CHPID is available with CHPID type OSM. The OSA connection is through the Bulk Power Hub (BPH) port J07 on the zEC12 to the Top of the Rack (TORs) switches on zBX.
8.3.48 Intraensemble data network (IEDN)

The intraensemble Data Network (IEDN) is one of the ensembles two private and secure internal networks. IEDN provides applications with a fast data exchanging path between ensemble nodes. Specifically, it is used for communications across the virtualized images (LPARs, z/VMs virtual machines, and blades LPARs). The IEDN is a private and secure 10 Gbps ethernet network that connects all elements of an ensemble and is access-controlled using integrated virtual LAN (VLAN) provisioning. No customer managed switches or routers are required. IEDN is managed by the primary HMC that controls the ensemble, helping to reduce the need for firewalls and encryption, and simplifying network configuration and management, with full redundancy for high availability. The z196 introduced the OSA Direct-Express zBX (OSX) CHPID type. The OSA connection is from the zEC12 to the Top of the Rack (TORs) switches on zBX. IEDN requires two OSA Express4S 10 GbE ports configured as CHPID type OSX.
8.3.49 OSA-Express4S NCP support (OSN)

OSA-Express4S 1000BASE-T Ethernet features can provide channel connectivity from an operating system in a zEC12 to IBM Communication Controller for Linux on System z (CCL) with the Open Systems Adapter for NCP (OSN), in support of the Channel Data Link Control (CDLC) protocol. OSN eliminates the requirement for an external communication medium for communications between the operating system and the CCL image. Because ESCON channels are not supported on zEC12, OSN is the only option. Data flow of the logical-partition to the logical-partition is accomplished by the OSA-Express4S feature without ever exiting the card. OSN support allows multiple connections between the same CCL image and the same operating system (such z/OS or z/TPF). The operating system must reside in the same physical server as the CCL image. For CCL planning information see IBM Communication Controller for Linux on System z V1.2.1 Implementation Guide, SG24-7223. For the most recent CCL information, see: http://www-01.ibm.com/software/network/ccl/
280
8049ch08.fm
Channel Data Link Control (CDLC), when used with the Communication Controller for Linux, emulates selected functions of IBM 3745/NCP operations. The port used with the OSN support appears as an ESCON channel to the operating system. This support can be used with OSA-Express4S 1000BASE-T features. Table 8-41 lists the minimum support requirements for OSN.
Table 8-41 Minimum support requirements for OSA-Express4S OSN Operating system Support requirements
z/OS z/VM z/VSE z/TPF Linux on System z a. PTFs required
z/OS V1R10a z/VM V5R4 z/VSE V4R3 z/TPF V1R1 PUT 4a SUSE SLES 10 Red Hat RHEL 5
8.3.50 Integrated Console Controller

The 1000BASE-T Ethernet features provide the Integrated Console Controller (OSA-ICC) function, which supports TN3270E (RFC 2355) and non-SNA DFT 3270 emulation. The OSA-ICC function is defined as CHIPD type OSC and console controller, and has multiple logical partitions support, both as shared or spanned channels. With the OSA-ICC function, 3270 emulation for console session connections is integrated in the zEC12 through a port on the OSA-Express4S 1000BASE-T or OSA-Express3 1000BASE-T features. This function eliminates the requirement for external console controllers, such as 2074 or 3174, helping to reduce cost and complexity. Each port can support up to 120 console session connections. OSA-ICC can be configured on a PCHID-by-PCHID basis and is supported at any of the feature settings (10, 100, or 1000 Mbps, half-duplex or full-duplex).
8.3.51 VLAN management enhancements

Table 8-42 lists minimum support requirements for VLAN management enhancements for the OSA-Express3 features (CHPID type OSD).
Table 8-42 Minimum support requirements for VLAN management enhancements Operating system Support requirements
z/OS z/VM
z/OS V1R10 z/VM V5R4. Support of guests is transparent to z/VM if the device is directly connected to the guest (pass through).
8.3.52 GARP VLAN Registration Protocol

All OSA-Express3 features support VLAN prioritization, a component of the IEEE 802.1 standard. GARP VLAN Registration Protocol (GVRP) support allows an OSA-Express3 port
281
8049ch08.fm
to register or unregister its VLAN IDs with a GVRP-capable switch and dynamically update its table as the VLANs change. This simplifies the network administration and management of VLANs as manually entering VLAN IDs at the switch is no longer necessary. Minimum support requirements are listed in Table 8-43.
Table 8-43 Minimum support requirements for GVRP Operating system Support requirements
z/OS z/VM
z/OS V1R10 z/VM V5R4
8.3.53 Inbound workload queueing (IWQ) for OSA-Express4S and OSA-Express3

OSA-Express-3 introduced inbound workload queueing (IWQ), which creates multiple input queues and allows OSA to differentiate workloads off the wire and then assign work to a specific input queue (per device) to z/OS. The support is also available with OSA-Express4S. CHPID types OSD and OSX are supported. With each input queue representing a unique type of workload, each having unique service and processing requirements, the IWQ function allows z/OS to preassign the appropriate processing resources for each input queue. This approach allows multiple concurrent z/OS processing threads to process each unique input queue (workload) avoiding traditional resource contention. In a heavily mixed workload environment, this off the wire network traffic separation provided by OSA-Express4S and OSA-Express3 IWQ reduces the conventional z/OS processing required to identify and separate unique workloads, which results in improved overall system performance and scalability. A primary objective of IWQ is to provide improved performance for business critical interactive workloads by reducing contention created by other types of workloads. The types of z/OS workloads that are identified and assigned to unique input queues are as follows: z/OS Sysplex Distributor traffic: Network traffic which is associated with a distributed virtual internet protocol address (VIPA) is assigned to a unique input queue allowing the Sysplex Distributor traffic to be immediately distributed to the target host z/OS bulk data traffic: Network traffic which is dynamically associated with a streaming (bulk data) TCP connection is assigned to a unique input queue allowing the bulk data processing to be assigned the appropriate resources and isolated from critical interactive workloads. IWQ is exclusive to OSA-Express4S and OSA-Express3 CHPID types OSD and OSX and the z/OS operating system. This applies to zEC12, z196, z114 and System z10. Minimum support requirements are listed in Table 8-44.
Table 8-44 Minimum support requirements for IWQ Operating system Support requirements
z/OS z/VM
z/OS V1R12 z/VM V5R4 for guest exploitation only; service required
282
8049ch08.fm
8.3.54 Inbound workload queueing (IWQ) for Enterprise Extender

Inbound workload queuing (IWQ) for the OSA-Express features has been enhanced to differentiate and separate inbound Enterprise Extender traffic to a new input queue. IWQ for Enterprise Extender is exclusive to OSA-Express4S and OSA-Express3 CHPID types OSD and OSX and the z/OS operating system. This applies to zEC12, z196 and z114. Minimum support requirements are listed in Table 8-45.
Table 8-45 Minimum support requirements for IWQ Operating system Support requirements
z/OS z/VM
z/OS V1R13 z/VM V5R4 for guest exploitation only; service required
8.3.55 Query and display OSA configuration

OSA-Express3 introduced the capability for the operating system to directly query and display the current OSA configuration information (similar to OSA/SF). z/OS exploits this OSA capability by introducing a TCP/IP operator command called display OSAINFO. Using display OSAINFO allows the operator to monitor and verify the current OSA configuration which will help to improve the overall management, serviceability, and usability of OSA-Express4S and OSA-Express3. The display OSAINFO command is exclusive to z/OS and applies to OSA-Express4S and OSA-Express3 features, CHPID types OSD, OSM, and OSX.
8.3.56 Link aggregation support for z/VM

Link aggregation (IEEE 802.3ad) controlled by the z/VM Virtual Switch (VSWITCH) allows the dedication of an OSA-Express4S or OSA-Express3 port to the z/VM operating system, when the port is participating in an aggregated group configured in Layer 2 mode. Link aggregation (trunking) is designed to allow combining multiple physical OSA-Express4S or OSA-Express3 ports into a single logical link for increased throughput and for nondisruptive fail over, in the event that a port becomes unavailable. The target links for aggregation must be of the same type. Link aggregation is applicable to the OSA-Express4S and OSA-Express3 features when configured as CHPID type OSD (QDIO). Link aggregation is supported by z/VM V5R4 and above.
8.3.57 QDIO data connection isolation for z/VM

The Queued Direct I/O (QDIO) data connection isolation function provides a higher level of security when sharing the same OSA connection in z/VM environments that use the Virtual Switch (VSWITCH). The VSWITCH is a virtual network device that provides switching between OSA connections and the connected guest systems. QDIO data connection isolation allows disabling internal routing for each QDIO connected, and provides a means for creating security zones and preventing network traffic between the zones.
283
8049ch08.fm
VSWITCH isolation support is provided by APAR VM64281. z/VM 5R4 and later support is provided by CP APAR VM64463 and TCP/IP APAR PK67610. QDIO data connection isolation is supported by all OSA-Express4S and OSA-Express3 features on zEC12.
8.3.58 QDIO interface isolation for z/OS

Some environments require strict controls for routing data traffic between severs or nodes. In certain cases, the LPAR-to-LPAR capability of a shared OSA connection can prevent such controls from being enforced. With interface isolation, internal routing can be controlled on an LPAR basis. When interface isolation is enabled, the OSA will discard any packets destined for a z/OS LPAR that is registered in the OAT as isolated. QDIO interface isolation is supported by Communications Server for z/OS V1R11 and all OSA-Express4S and OSA-Express3 features on zEC12.
8.3.59 QDIO optimized latency mode

QDIO optimized latency mode (OLM) can help improve performance for applications that have a critical requirement to minimize response times for inbound and outbound data. OLM optimizes the interrupt processing as follows: For inbound processing, the TCP/IP stack looks more frequently for available data to process, ensuring that any new data is read from the OSA-Express4S or OSA-Express3 without requiring additional program controlled interrupts (PCIs). For outbound processing, the OSA-Express4S or OSA-Express3 also look more frequently for available data to process from the TCP/IP stack, thus not requiring a Signal Adapter (SIGA) instruction to determine whether more data is available.
8.3.60 Large send for IPv6 packets

Large send for IPv6 packets improves performance by offloading outbound TCP segmentation processing from the host to an OSA-Express4S feature by employing a more efficient memory transfer into OSA-Express4. Large send support for IPv6 packets applies to the OSA-Express4S features (CHPID type OSD and OSX), and is exclusive to zEC12, z196 and z114. Large send is not supported for LPAR-to-LPAR packets. Minimum support requirements are listed in Table 8-46 on page 284.
Table 8-46 Minimum support requirements for Large send for IPv6 packets Operating system Support requirements
z/OS z/VM a. PTFs required.
z/OS V1R13a z/VM V5R4 for guest exploitation only
8.3.61 OSA-Express4S checksum offload

OSA-Express4S features, when configured as CHPID type OSD, provide checksum offload for several types of traffic, as indicated in Table 8-47.
284
8049ch08.fm
Table 8-47 Minimum support requirements for OSA-Express4S checksum offload Traffic Support requirements
LPAR to LPAR IPv6 LPAR to LPAR traffic for IPv4 and IPv6
z/OS V1R12 a z/VM V5R4 for guest exploitationb z/OS V1R13 z/VM V5R4 for guest exploitationb z/OS V1R13 z/VM V5R4 for guest exploitationb
a. PTFs are required b. Device is directly attached to guest, PTFs are required.
8.3.62 Checksum offload for IPv4 packets when in QDIO mode

A function referred to as checksum offload, supports z/OS and Linux on System z environments. It is offered on the OSA-Express4S GbE, OSA-Express4S 1000BASE-T Ethernet, OSA-Express3 GbE, and OSA-Express3 1000BASE-T Ethernet features. Checksum offload provides the capability of calculating the Transmission Control Protocol (TCP), User Datagram Protocol (UDP), and Internet Protocol (IP) header checksum. Checksum verifies the accuracy of files. By moving the checksum calculations to a Gigabit or 1000BASE-T Ethernet feature, host CPU cycles are reduced and performance is improved. When checksum is offloaded, the OSA-Express feature performs the checksum calculations for Internet Protocol Version 4 (IPv4) packets. The checksum offload function applies to packets that go to or come from the LAN. When multiple IP stacks share an OSA-Express, and an IP stack sends a packet to a next hop address owned by another IP stack that is sharing the OSA-Express, the OSA-Express then sends the IP packet directly to the other IP stack without placing it out on the LAN. Checksum offload does not apply to such IP packets. Checksum offload is supported by the GbE features (FC 0404, FC 0405, FC 3362, and FC 3363) and the 1000BASE-T Ethernet features (FC 0408 and FC 3367) when operating at 1000 Mbps (1 Gbps). Checksum offload is applicable to the QDIO mode only (channel type OSD). z/OS support for checksum offload is available in all in-service z/OS releases, and in all supported Linux on System z distributions.
8.3.63 Adapter interruptions for QDIO

Linux on System z and z/VM work together to provide performance improvements by exploiting extensions to the Queued Direct I/O (QDIO) architecture. Adapter interruptions, first added to z/Architecture with HiperSockets, provide an efficient, high-performance technique for I/O interruptions to reduce path lengths and overhead in both the host operating system and the adapter (OSA-Expres4S and OSA-Express3 when using CHPID type OSD). In extending the use of adapter interruptions to OSD (QDIO) channels, the programming overhead to process a traditional I/O interruption is reduced. This benefits OSA-Express TCP/IP support in z/VM, z/VSE, and Linux on System z. Adapter interruptions apply to all of the OSA-Express4S and OSA-Express3 features on zEC12 when in QDIO mode (CHPID type OSD).
285
8049ch08.fm
8.3.64 OSA Dynamic LAN idle

OSA Dynamic LAN idle parameter change helps reduce latency and improve performance by dynamically adjusting the inbound blocking algorithm. System administrators can authorize the TCP/IP stack to enable a dynamic setting which, previously, was a static setting. For latency-sensitive applications, the blocking algorithm is modified to be latency sensitive. For streaming (throughput-sensitive) applications, the blocking algorithm is adjusted to maximize throughput. In all cases, the TCP/IP stack determines the best setting based on the current system and environmental conditions (inbound workload volume, processor utilization, traffic patterns, and so on) and can dynamically update the settings. OSA-Express4S and OSA-Express3 features adapt to the changes, avoiding thrashing and frequent updates to the OSA address table (OAT). Based on the TCP/IP settings, OSA holds the packets before presenting them to the host. A dynamic setting is designed to avoid or minimize host interrupts. OSA Dynamic LAN idle is supported by the OSA-Express4S and OSA-Express3 features on zEC12 when in QDIO mode (CHPID type OSD), and is exploited by z/OS V1R8 and higher, with program temporary fixes (PTFs).
8.3.65 OSA Layer 3 Virtual MAC for z/OS environments

To help simplify the infrastructure and to facilitate load balancing when a logical partition is sharing the same OSA Media Access Control (MAC) address with another logical partition, each operating system instance can have its own unique logical or virtual MAC (VMAC) address. All IP addresses associated with a TCP/IP stack are accessible by using their own VMAC address, instead of sharing the MAC address of an OSA port, which also applies to Layer 3 mode and to an OSA port spanned among channel subsystems. OSA Layer 3 VMAC is supported by the OSA-Express4S and OSA-Express3 features on zEC12 when in QDIO mode (CHPID type OSD), and is exploited by z/OS V1R8 and later.
8.3.66 QDIO Diagnostic Synchronization

QDIO Diagnostic Synchronization enables system programmers and network administrators to coordinate and simultaneously capture both software and hardware traces. It allows z/OS to signal OSA-Express4S and OSA-Express3 features (by using a diagnostic assist function) to stop traces and capture the current trace records. QDIO Diagnostic Synchronization is supported by the OSA-Express4S and OSA-Express3 features on zEC12 when in QDIO mode (CHPID type OSD), and is exploited by z/OS V1R8 and later.
8.3.67 Network Traffic Analyzer

With the large volume and complexity of today's network traffic, the zEC12 offers systems programmers and network administrators the ability to more easily solve network problems. With the availability of the OSA-Express Network Traffic Analyzer and QDIO Diagnostic Synchronization on the server, you can capture trace and trap data, and forward it to z/OS tools for easier problem determination and resolution. The Network Traffic Analyzer is supported by the OSA-Express4S and OSA-Express3 features on zEC12 when in QDIO mode (CHPID type OSD), and is exploited by z/OS V1R8 and later. 286
8049ch08.fm
8.3.68 Program directed re-IPL

First available on System z9, program directed re-IPL allows an operating system on a zEC12 to re-IPL without operator intervention. This function is supported for both SCSI and ECKD devices. Table 8-48 lists the minimum support requirements for program directed re-IPL.
Table 8-48 Minimum support requirements for program directed re-IPL Operating system Support requirements
z/VM Linux on System z z/VSE
z/VM V5R4 SUSE SLES 10 SP3 Red Hat RHEL 5.4 V4R3 on SCSI disks
8.3.69 Coupling over InfiniBand

InfiniBand technology can potentially provide high-speed interconnection at short distances, longer distance fiber optic interconnection, and interconnection between partitions on the same system without external cabling. Several areas of this book discuss InfiniBand characteristics and support. For example, see 4.10, Parallel Sysplex connectivity on page 161.
InfiniBand coupling links

Table 8-49 lists the minimum support requirements for coupling links over InfiniBand.
Table 8-49 Minimum support requirements for coupling links over InfiniBand Operating system Support requirements
z/OS z/VM z/TPF
z/OS V1R10 z/VM V5R4 (dynamic I/O support for InfiniBand CHPIDs only; coupling over InfiniBand is not supported for guest use) z/TPF V1R1
InfiniBand coupling links at an unrepeated distance of 10 km

Support for HCA2-O LR (1xIFB) fanout supporting InfiniBand coupling links 1x at an unrepeated distance of 10 KM is listed in Table 8-50.
Table 8-50 Minimum support requirements for coupling links over InfiniBand at 10 km Operating system Support requirements
z/OS z/VM
z/OS V1R10; service required z/VM V5R4 (dynamic I/O support for InfiniBand CHPIDs only; coupling over InfiniBand is not supported for guest use)
8.3.70 Dynamic I/O support for InfiniBand CHPIDs

This function refers exclusively to the z/VM dynamic I/O support of InfiniBand coupling links. Support is available for the CIB CHPID type in the z/VM dynamic commands, including the change channel path dynamic I/O command. Specifying and changing the system name
287
8049ch08.fm
when entering and leaving configuration mode is also supported. z/VM does not use InfiniBand and does not support the use of InfiniBand coupling links by guests. Table 8-51 lists the minimum support requirements of dynamic I/O support for InfiniBand CHPIDs.
Table 8-51 Minimum support requirements for dynamic I/O support for InfiniBand CHPIDs Operating system Support requirements
z/VM
z/VM V5R4
8.4 Cryptographic support

IBM zEnterprise EC12 provides two major groups of cryptographic functions: Synchronous cryptographic functions, which are provided by the CP Assist for Cryptographic Function (CPACF) Asynchronous cryptographic functions, which are provided by the Crypto Express4S and by the Crypto Express3 features The minimum software support levels are listed in the following sections. Obtain and review the most recent Preventive Service Planning (PSP) buckets to ensure that the latest support levels are known and included as part of the implementation plan.
8.4.1 CP Assist for Cryptographic Function

In zEC12, the CP Assist for Cryptographic Function (CPACF) supports the full complement of the Advanced Encryption Standard (AES, symmetric encryption), the full complement of the Data Encryption Standard (DES, symmetric encryption) and the secure hash algorithm (SHA, hashing). For a detailed description, see 6.6, CP Assist for Cryptographic Function on page 194. Table 8-52 on page 288 lists the support requirements for CPACF at zEC12.
Table 8-52 Support requirements for CPACF Operating system Support requirements
z/OSa z/VM z/VSE z/TPF Linux on System z
z/OS V1R10 and later with the Cryptographic Support for z/OS V1R10-V1R12 Web deliverable. z/VM V5R4 with PTFs and higher: Supported for guest use. z/VSE V4R2 and later: Supports the CPACF features with the functionality supported on IBM System z10. z/TPF V1R1 SUSE SLES 11 SP1 Red Hat RHEL 6.1 For Message-Security-Assist-Extension 4 exploitation, IBM is working with its Linux distribution partners to include support in future Linux on System z distribution releases.
a. CPACF is also exploited by several IBM Software product offerings for z/OS, such as IBM WebSphere Application Server for z/OS.
288
8049ch08.fm
8.4.2 Crypto Express4S

Support of Crypto Express4S functions varies by operating system and release. Table 8-53 lists the minimum software requirements for the Crypto Express4S features when configured as a coprocessor or an accelerator. For a full description, see 6.7, Crypto Express4S on page 194.
Table 8-53 Crypto Express4S support on zEC12 Operating system Crypto Express4S
z/OS
z/OS V1R13, or z/OS V1R12 with the Cryptographic Support for z/OS V1R12-V1R13 Web deliverable. z/OS V1R10, or z/OS V1R11 with toleration maintenance. z/VM V6R2, or z/VM V6R1, or z/VM V5R4 with maintenance. z/VSE V5R1 with PTFs. Service required (accelerator mode only). IBM is working with its Linux distribution partners to include support in future Linux on System z distribution releases
z/VM z/VSE z/TPF V1R1 Linux on System z
8.4.3 Crypto Express3

Support of Crypto Express3 functions varies by operating system and release. Table 8-54 lists the minimum software requirements for the Crypto Express3 features when configured as a coprocessor or an accelerator. For a full description, see 6.8, Crypto Express3 on page 197.
Table 8-54 Crypto Express3 support on zEC12 Operating system Crypto Express3
z/OS
z/OS V1R12 (ICSF FMID HCR7770) and above z/OS V1R10, or z/OS V1R11 with the Cryptographic Support for z/OS V1R9-V1R11 Web deliverable. Z/OS V1R8 with toleration maintenance: Crypto Express3 features handled as Crypto Express2 features. z/VM V5R4: Service required; supported for guest use only. z/VSE V4R2 Service required (accelerator mode only). For toleration: SUSE SLES10 SP3 and SLES 11. Red Hat RHEL 5.4 and RHEL 6.0. For exploitation: SUSE SLES11 SP1. Red Hat RHEL 6.1.
z/VM z/VSE z/TPF V1R1 Linux on System z
8.4.4 Web deliverables

For Web-deliverable code on z/OS, see the z/OS downloads : http://www.ibm.com/systems/z/os/zos/downloads/
289
8049ch08.fm
For Linux on System z, support is delivered through IBM and distribution partners. For more information see Linux on System z on the developerWorks website: http://www.ibm.com/developerworks/linux/linux390/
8.4.5 z/OS ICSF FMIDs

Integrated Cryptographic Service Facility (ICSF) is a base component of z/OS, and is designed to transparently use the available cryptographic functions, whether CPACF or Crypto Express features to balance the workload and help address the bandwidth requirements of the applications. Despite being a z/OS base component, ICSF new functions are generally made available through a web deliverable support a couple of months after a new z/OS release is launched. Due to this fact, new functions must be related to an ICSF FMID instead of a z/OS version. For a list of ICSF versions and FMID cross-references, see the Technical Documents page: http://www.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/TD103782 Table 8-55 on page 291 lists the ICSF FMIDs and web-deliverable codes for z/OS V1R10 through V1R13. Later FMIDs include the functions of previous ones.
290
8049ch08.fm
Table 8-55 z/OS ICSF FMIDs z/OS ICSF FMID Web deliverable name Supported function
V1R10
HCR7750
Included as a z/OS base element
CPACF AES-192 and AES-256 CPACF SHA-224, SHA-384 and SHA-512 4096-bit RSA keys ISO-3 PIN block format IBM System z10 BC support Secure key AES Key store policy PKDS Sysplex-wide consistency In-storage copy of the PKDS 13-digit through 19-digit PANs Crypto Query service Enhanced SAF checking Crypto Express3 and Crypto Express3-1P support PKA Key Management Extensions CPACF Protected Key Extended PKCS #11 ICSF Restructure (Performance, RAS, ICSF-CICS Attach Facility) IBM zEnterprise 196 support Elliptic Curve Cryptography Message-Security-Assist-4 HMAC Support ANSI X9.8 Pin ANSI X9.24 (CBC Key Wrapping) CKDS constraint relief PCI Audit All callable services AMODE(64) PKA RSA OAEP with SHA-256 algorithmb
HCR7751
Cryptographic Support for z/OS V1R8-V1R10 and z/OS.e V1R8a
HCR7770
Cryptographic support for z/OS V1R9-V1R11
HCR7780
291
8049ch08.fm
z/OS
ICSF FMID
Web deliverable name
Supported function
V1R11
HCR7751
IBM System z10 BC support Secure key AES Key store policy PKDS Sysplex-wide consistency In-storage copy of the PKDS 13-digit through 19-digit PANs Crypto Query service Enhanced SAF checking Crypto Express3 and Crypto Express3-1P support PKA Key Management Extensions CPACF Protected Key Extended PKCS #11 ICSF Restructure (Performance, RAS, ICSF-CICS Attach Facility) IBM zEnterprise 196 support Elliptic Curve Cryptography Message-Security-Assist-4 HMAC Support ANSI X9.8 Pin ANSI X9.24 (CBC Key Wrapping) CKDS constraint relief PCI Audit All callable services AMODE(64) PKA RSA OAEP with SHA-256 algorithmb Expanded key support for AES algorithm Enhanced ANSI TR-31 PIN block decimalization table protection Elliptic Curve Diffie-Hellman (ECDH) algorithm RSA in the Modulus Exponent (ME) and Chinese Remainder Theorem (CRT) formats.
HCR7770
HCR7780
HCR7790
Cryptographic Support for z/OS V1R11-V1R13
292
8049ch08.fm
z/OS
ICSF FMID
Supported function
V1R12
HCR7770
Crypto Express3 and Crypto Express3-1P support PKA Key Management Extensions CPACF Protected Key Extended PKCS #11 ICSF Restructure (Performance, RAS, ICSF-CICS Attach Facility) IBM zEnterprise 196 support Elliptic Curve Cryptography Message-Security-Assist-4 HMAC Support ANSI X9.8 Pin ANSI X9.24 (CBC Key Wrapping) CKDS constraint relief PCI Audit All callable services AMODE(64) PKA RSA OAEP with SHA-256 algorithmb Expanded key support for AES algorithm Enhanced ANSI TR-31 PIN block decimalization table protection Elliptic Curve Diffie-Hellman (ECDH) algorithm RSA in the ME and CRT formats. Support for the Crypto Express4S feature when configured as an EP11 coprocessor Support for the Crypto Express4S feature when configured as a CCA coprocessor Support for 24-byte Data Encryption Standard (DES) master keys Improved wrapping key strength DUKPT for Message Authentication Code (MAC) and encryption keys Secure Cipher Text Translate2 Compliance with new random number generation standards EMV enhancements for applications supporting American Express cards
HCR7780
HCR7790
HCR77A0
293
8049ch08.fm
z/OS
ICSF FMID
Supported function
V1R13
HCR7780
IBM zEnterprise 196 support Elliptic Curve Cryptography Message-Security-Assist-4 HMAC Support ANSI X9.8 Pin ANSI X9.24 (CBC Key Wrapping) CKDS constraint relief PCI Audit All callable services AMODE(64) PKA RSA OAEP with SHA-256 algorithmb Expanded key support for AES algorithm Enhanced ANSI TR-31 PIN block decimalization table protection Elliptic Curve Diffie-Hellman (ECDH) algorithm RSA in the ME and CRT formats. Support for the Crypto Express4S feature when configured as an EP11 coprocessor Support for the Crypto Express4S feature when configured as a CCA coprocessor Support for 24-byte Data Encryption Standard (DES) master keys Improved wrapping key strength DUKPT for Message Authentication Code (MAC) and encryption keys Secure Cipher Text Translate2 Compliance with new random number generation standards EMV enhancements for applications supporting American Express cards
HCR7790
HCR77A0
a. Service is required b. Service is required
8.4.6 ICSF migration considerations

Consider the following points about the Cryptographic Support for z/OS V1R12-V1R13 Web deliverable ICSF HCR77A0 code: It is not integrated in ServerPac (even for new z/OS V1R13 orders). It is only required to exploit the new zEC12 functions. All systems in a sysplex sharing a PKDS/TKDS must be at HCR77A0 to exploit the new PKDS/TKDS Coordinated Administration support. New ICSF toleration PTFs needed to:
294
8049ch08.fm
Permit the use of a PKDS with RSA private key tokens encrypted under the ECC Master Key. Support for installation options data sets which use the keyword BEGIN(FMID). New SMP/E Fix Category will be created for ICSF coexistence IBM.Coexistence.ICSF.z/OS_V1R12-V1R13-HCR77A0
8.5 z/OS migration considerations

With the exception of base processor support, z/OS software changes do not require the new zEC12 functions. Equally, the new functions do not require functional software. The approach has been, where applicable, to let z/OS automatically decide to enable a function based on the presence or absence of the required hardware and software.
8.5.1 General guidelines

The IBM zEnterprise EC12 introduces the latest System z technology. Although support is provided by z/OS starting with z/OS V1R10, exploitation of zEC12 is dependent on the z/OS release. The z/OS.e is not supported on zEC12. In general, consider the following guidelines: Do not migrate software releases and hardware at the same time. Keep members of sysplex at same software level, except during brief migration periods. Migrate to STP-only or Mixed-CTN network prior to introducing a zEC12 into a Sysplex. Review zEC12 restrictions and migration considerations prior to creating an upgrade plan.
8.5.2 HCD
On z/OS V1R10 and above, HCD or the Hardware Configuration Manager (HCM) assist in the defining a configuration for zEC12.
8.5.3 InfiniBand coupling links

Each system can use, or not use, InfiniBand coupling links independently of what other systems are doing, and do so in conjunction with other link types. InfiniBand coupling connectivity can only be obtained with other systems that also support InfiniBand coupling. Note that zEC12 does not support InfiniBand connectivity with System z9 and earlier systems.
8.5.4 Large page support

The large page support function must not be enabled without the respective software support. If large page is not specified, page frames are allocated at the current size of 4 K. In z/OS V1R9 and later, the amount of memory to be reserved for large page support is defined by using parameter LFAREA in the IEASYSxx member of SYS1.PARMLIB, as follows: LFAREA=xx%|xxxxxxM|xxxxxxG
295
8049ch08.fm
The parameter indicates the amount of storage, in percentage, megabytes, or gigabytes. The value cannot be changed dynamically.
8.5.5 HiperDispatch
The HIPERDISPATCH=YES/NO parameter in the IEAOPTxx member of SYS1.PARMLIB and on the SET OPT=xx command can control whether HiperDispatch is enabled or disabled for a z/OS image. It can be changed dynamically, without an IPL or any outage. The default is that HiperDispatch is disabled on all releases, from z/OS V1R10 (requires PTFs for zIIP support) through z/OS V1R13. To effectively exploit HiperDispatch, the Workload Manager (WLM) goal adjustment might be required. Review the WLM policies and goals, and update them as necessary. You might want to run with the new policies and HiperDispatch on for a period, turn it off and use the older WLM policies while analyzing the results of using HiperDispatch, re-adjust the new policies and repeat the cycle, as needed. In order to change WLM policies, turning HiperDispatch off then on is not necessary. A health check is provided to verify whether HiperDispatch is enabled on a system image that is running on zEC12.
8.5.6 Capacity Provisioning Manager

Installation of the capacity provision function on z/OS requires: Setting up and customizing z/OS RMF, including the Distributed Data Server (DDS) Setting up the z/OS CIM Server (included in z/OS base) Performing capacity provisioning customization as described in the publication z/OS MVS Capacity Provisioning User's Guide, SA33-8299 Exploitation of the capacity provisioning function requires: TCP/IP connectivity to observed systems RMF Distributed Data Server must be active CIM server must be active Security and CIM customization Capacity Provisioning Manager customization In addition, the Capacity Provisioning Control Center has to be downloaded from the host and installed on a PC server. This application is only used to define policies. It is not required for regular operation. Customization of the capacity provisioning function is required on the following systems: Observed z/OS systems. These are the systems in one or multiple sysplexes that are to be monitored. For a description of the capacity provisioning domain, see 9.8, Nondisruptive upgrades on page 347. Runtime systems. These are the systems where the Capacity Provisioning Manager is running, or to which the server can fail over after server or system failures.
8.5.7 Decimal floating point and z/OS XL C/C++ considerations

z/OS V1R13 with PTFs is required in order to use the latest level (10) of the following two C/C++ compiler options: 296
8049ch08.fm
ARCHITECTURE: this option selects the minimum level of machine architecture on which the program will run. Note that certain features provided by the compiler require a minimum architecture level. ARCH(10) exploit instructions available on the zEC12. TUNE: this option allows optimization of the application for a specific machine architecture, within the constraints imposed by the ARCHITECTURE option. The TUNE level must not be lower than the setting in the ARCHITECTURE option. For more information about the ARCHITECTURE and TUNE compiler options, see the z/OS V1R13.0 XL C/C++ Users Guide, SC09-4767. Attention: The previous System z ARCHITECTURE or TUNE options for C/C++ programs should be used if the same applications should run on both the zEC12 as well as on previous System z servers. However, if C/C++ applications will only run on zEC12 servers the latest ARCHITECTURE and TUNE options should be used assuring that the best performance possible is delivered through the latest instruction set additions.
8.5.8 IBM System z Advanced Workload Analysis Reporter (IBM zAware)

IBM zAware is designed to offer a real-time, continuous learning, diagnostics and monitoring capability intended to help customers pinpoint and resolve potential problems quickly enough to minimize impacts to their businesses. IBM zAware runs analytics in firmware and intelligently examines the message logs for potential deviations, inconsistencies, or variations from the norm. For many z/OS environments, there is a large volume of OPERLOG messages, which makes it difficult for operations personnel to consume and analyze easily. IBM zAware provides a simple graphical user interface (GUI) for easy drill-down and identification of message anomalies, which can facilitate faster problem resolution. IBM zAware is ordered through specific features of zEC12 and requires z/OS 1.13 with IBM zAware exploitation support to collect specific log stream data. It requires a properly configured LPAR, see The zAware-mode logical partition on page 257. In order to exploit IBM zAware feature, the following tasks should be completed in z/OS: For each z/OS that is to be monitored through the IBM zAware client, configure a network connection in the TCP/IP profile. If necessary, update firewall settings. Verify that each z/OS system meets the sysplex configuration and OPERLOG requirements for monitored clients of the IBM zAware virtual appliance. Configure the z/OS system logger to send data to the IBM zAware virtual appliance server. Prime the IBM zAware server with prior data from monitored clients.
8.6 Coupling facility and CFCC considerations

Coupling facility connectivity to a zEC12 is supported on the z196, z114, z10EC, z10 BC, or another zEC12. The logical partition running the Coupling Facility Control Code (CFCC) can reside on any of the supported systems previously listed. See Table 8-56 on page 298 for Coupling Facility Control Code requirements for supported systems. Important: Because coupling link connectivity to System z9, and previous systems is not supported, this could affect the introduction of zEC12 into existing installations, and require additional planning. Also consider the level of CFCC. For more information, see Coupling link migration considerations on page 165.
297
8049ch08.fm
The initial support of the CFCC on the zEC12 is level 18. CFCC level 18 offers the following enhancements: Coupling channel reporting enhancements Enables RMF to differentiate various IFB link types and detect if CIB link running degraded Serviceability enhancements Additional structure control info in CF dumps Enhanced CFCC tracing support Enhanced Triggers for CF non-disruptive dumping. Performance enhancements Dynamic structure size alter improvement DB2 GBP cache bypass Cache structure management Attention: Having more than 1024 structures requires a new version of the CFRM CDS. In addition, all systems in the sysplex need to be at z/OS V1R12 (or above) or have the coexistence/preconditioning PTFs installed. Falling back to a previous level, without the coexistence PTF installed, is not supported without at sysplex IPL. zEC12 systems with CFCC level 18 require z/OS V1R10 or later, and z/VM V5R4 or later for guest virtual coupling. The current CFCC level for zEC12 servers is CFCC level 18, see Table 8-56. To support migration from one CFCC level to the next, different levels of CFCC can be run concurrently while the coupling facility logical partitions are running on different servers (CF logical partitions running on the same server share the same CFCC level).
Table 8-56 System z CFCC code level considerations
zEC12 z196 and z114 z10 EC or z10 BC z9 EC or z9 BC z990 or z890
CFCC level 18 or later CFCC level 17 or later CFCC level 15 or later CFCC level 14 or later CFCC level 13 or later
CF structure sizing changes are expected when going from CFCC Level 17 (or below) to CFCC Level 18. We suggest reviewing the CF LPAR size by using the CFSizer tool available at the following web page: http://www.ibm.com/systems/z/cfsizer Previous to migration, installation of compatibility/coexistence PTFs is highly desirable. A planned outage is required when migrating the CF or the CF LPAR to CFCC level 18. For additional details about CFCC code levels, see the Parallel Sysplex web site: http://www.ibm.com/systems/z/pso/cftable.html
298
8049ch08.fm
8.7 MIDAW facility

The modified indirect data address word (MIDAW) facility is a system architecture and software exploitation designed to improve FICON performance. This facility was first made available on System z9 servers and is exploited by the media manager in z/OS. The MIDAW facility provides a more efficient CCW/IDAW structure for certain categories of data-chaining I/O operations: MIDAW can significantly improve FICON performance for extended format data sets. Non-extended data sets can also benefit from MIDAW. MIDAW can improve channel utilization and can significantly improve I/O response time. It reduces FICON channel connect time, director ports, and control unit overhead. IBM laboratory tests indicate that applications using EF data sets, such as DB2, or long chains of small blocks can gain significant performance benefits by using the MIDAW facility. MIDAW is supported on FICON channels configured as CHPID types FC.
8.7.1 MIDAW technical description

An indirect address word (IDAW) is used to specify data addresses for I/O operations in a virtual environment10. The existing IDAW design allows the first IDAW in a list to point to any address within a page. Subsequent IDAWs in the same list must point to the first byte in a page. Also IDAWs (except the first and last IDAW) in a list must deal with complete 2 K or 4 K units of data. Figure 8-1 on page 299 shows a single channel command word (CCW) to control the transfer of data that spans non-contiguous 4 K frames in main storage. When the IDAW flag is set, the data address in the CCW points to a list of words (IDAWs), each of which contains an address designating a data area within real storage.
Figure 8-1 IDAW usage

10
There are exceptions to this statement and we skip a number of details in the following description. We assume that the reader can merge this brief description with an existing understanding of I/O operations in a virtual memory environment.
299
8049ch08.fm
The number of IDAWs required for a CCW is determined by the IDAW format as specified in the operation request block (ORB), by the count field of the CCW, and by the data address in the initial IDAW. For example, three IDAWS are required when the following three events occur: 1. The ORB specifies format-2 IDAWs with 4 KB blocks. 2. The CCW count field specifies 8 KB. 3. The first IDAW designates a location in the middle of a 4 KB block. CCWs with data chaining can be used to process I/O data blocks that have a more complex internal structure, in which portions of the data block are directed into separate buffer areas (this is sometimes known as scatter-read or scatter-write). However, as technology evolves and link speed increases, data chaining techniques are becoming less efficient in modern I/O environments for reasons involving switch fabrics, control unit processing and exchanges, and others. The MIDAW facility is a method of gathering and scattering data from and into discontinuous storage locations during an I/O operation. The modified IDAW (MIDAW) format is shown in Figure 8-2 on page 300. It is 16 bytes long and is aligned on a quadword.
Figure 8-2 MIDAW format
An example of MIDAW usage is shown in Figure 8-3 on page 300.
Figure 8-3 MIDAW usage
300
8049ch08.fm
The use of MIDAWs is indicated by the MIDAW bit in the CCW. If this bit is set, then the skip flag cannot be set in the CCW. The skip flag in the MIDAW can be used instead. The data count in the CCW should equal the sum of the data counts in the MIDAWs. The CCW operation ends when the CCW count goes to zero or the last MIDAW (with the last flag) ends. The combination of the address and count in a MIDAW cannot cross a page boundary. This means that the largest possible count is 4 K. The maximum data count of all the MIDAWs in a list cannot exceed 64 K, which is the maximum count of the associated CCW. The scatter-read or scatter-write effect of the MIDAWs makes it possible to efficiently send small control blocks embedded in a disk record to separate buffers from those used for larger data areas within the record. MIDAW operations are on a single I/O block, in the manner of data chaining. Do not confuse this operation with CCW command chaining.
8.7.2 Extended format data sets

z/OS extended format data sets use internal structures (usually not visible to the application program) that require scatter-read (or scatter-write) operation. This means that CCW data chaining is required and this produces less than optimal I/O performance. Because the most significant performance benefit of MIDAWs is achieved with extended format (EF) data sets, a brief review of the EF data sets is included here. Both Virtual Storage Access Method (VSAM) and non-VSAM (DSORG=PS) can be defined as extended format data sets. In the case of non-VSAM data sets, a 32-byte suffix is appended to the end of every physical record (that is, block) on disk. VSAM appends the suffix to the end of every control interval (CI), which normally corresponds to a physical record (a 32 K CI is split into two records to be able to span tracks.) This suffix is used to improve data reliability and facilitates other functions described in the following paragraphs. Thus, for example, if the DCB BLKSIZE or VSAM CI size is equal to 8192, the actual block on DASD consists of 8224 bytes. The control unit itself does not distinguish between suffixes and user data. The suffix is transparent to the access method or database. In addition to reliability, EF data sets enable three other functions: DFSMS striping Access method compression Extended addressability (EA) EA is especially useful for creating large DB2 partitions (larger than 4 GB). Striping can be used to increase sequential throughput, or to spread random I/Os across multiple logical volumes. DFSMS striping is especially useful for utilizing multiple channels in parallel for one data set. The DB2 logs are often striped to optimize the performance of DB2 sequential inserts. To process an I/O operation to an EF data set would normally require at least two CCWs with data chaining. One CCW would be used for the 32-byte suffix of the EF data set. With MIDAW, the additional CCW for the EF data set suffix can be eliminated. MIDAWs benefit both EF and non-EF data sets. For example, to read twelve 4 K records from a non-EF data set on a 3390 track, Media Manager would chain 12 CCWs together using data chaining. To read twelve 4 K records from an EF data set, 24 CCWs would be chained (two CCWs per 4 K record). Using Media Manager track-level command operations and MIDAWs, an entire track can be transferred using a single CCW.
301
8049ch08.fm
8.7.3 Performance benefits

z/OS Media Manager has the I/O channel programs support for implementing Extended Format data sets and it automatically exploits MIDAWs when appropriate. Today, most disk I/Os in the system are generated using media manager. Users of the Executing Fixed Channel Programs in Real Storage (EXCPVR) instruction can construct channel programs containing MIDAWs provided that they construct an IOBE with the IOBEMIDA bit set. Users of EXCP instruction cannot construct channel programs containing MIDAWs. The MIDAW facility removes the 4 K boundary restrictions of IDAWs and, in the case of EF data sets, reduces the number of CCWs. Decreasing the number of CCWs helps to reduce the FICON channel processor utilization. Media Manager and MIDAWs do not cause the bits to move any faster across the FICON link, but they do reduce the number of frames and sequences flowing across the link, thus using the channel resources more efficiently. Use of the MIDAW facility with FICON Express8S, operating at 8 Gbps, compared to use of IDAWs with FICON Express2, operating at 2 Gbps, showed an improvement in throughput for all reads on DB2 table scan tests with EF data sets. The performance of a specific workload can vary according to the conditions and hardware configuration of the environment. IBM laboratory tests found that DB2 gains significant performance benefits by using the MIDAW facility in the following areas: Table scans Logging Utilities Using DFSMS striping for DB2 data sets Media Manager with the MIDAW facility can provide significant performance benefits when used in combination applications that use EF data sets (such as DB2) or long chains of small blocks. For additional information relating to FICON and MIDAW, consult the following resources: The I/O Connectivity web site contains the material about FICON channel performance: http://www.ibm.com/systems/z/connectivity/ The following publication: DS8000 Performance Monitoring and Tuning, SG24-7146
8.8 IOCP
All System z servers require a description of their I/O configuration. This description is stored in input/output configuration data set (IOCDS) files. The input/output configuration program (IOCP) allows creation of the IOCDS file from a source file known as the input/output configuration source (IOCS). The IOCS file contains detailed information for each channel and path assignment, each control unit, and each device in the configuration. The required level of IOCP for the zEC12 is V2 R1 L0 (IOCP 2.1.0) or later with PTF. See the Input/Output Configuration Program Users Guide, SB10-7037, for details.
302
8049ch08.fm
8.9 Worldwide portname (WWPN) tool

A part of the installation of your zEC12 system is the pre-planning of the Storage Area Network (SAN) environment. IBM has made available a stand alone tool to assist with this planning prior to the installation. The capability of the worldwide port name (WWPN) tool has been extended to calculate and show WWPNs for both virtual and physical ports ahead of system installation. The tool assigns WWPNs to each virtual Fibre Channel Protocol (FCP) channel/port using the same WWPN assignment algorithms that a system uses when assigning WWPNs for channels utilizing N_Port Identifier Virtualization (NPIV). Thus, the SAN can be set up in advance, allowing operations to proceed much faster after the server is installed. In addition, the SAN configuration can be retained instead of altered by assigning the WWPN to physical FCP ports when a FICON feature is replaced. The WWPN tool takes a .csv file containing the FCP-specific I/O device definitions and creates the WWPN assignments that are required to set up the SAN. A binary configuration file that can be imported later by the system is also created. The .csv file can either be created manually, or exported from the Hardware Configuration Definition/Hardware Configuration Manager (HCD/HCM). The WWPN tool on zEC12 (CHPID type FCP) requires the following levels: z/OS V1R10, V1R11, with PTFs, or V1R12 and above z/VM V5R4 or V6R1, with PTFs, or V6R2 and above The WWPN tool is available for download at Resource Link and is applicable to all FICON channels defined as CHPID type FCP (for communication with SCSI devices) on zEC12. http://www.ibm.com/servers/resourcelink/
8.10 ICKDSF
Device Support Facilities, ICKDSF, Release 17 is required on all systems that share disk subsystems with a zEC12 processor. ICKDSF supports a modified format of the CPU information field, which contains a two-digit logical partition identifier. ICKDSF uses the CPU information field instead of CCW reserve/release for concurrent media maintenance. It prevents multiple systems from running ICKDSF on the same volume, and at the same time allows user applications to run while ICKDSF is processing. To prevent any possible data corruption, ICKDSF must be able to determine all sharing systems that can potentially run ICKDSF. Therefore, this support is required for zEC12. Important: The need for ICKDSF Release 17 applies even to systems that are not part of the same sysplex, or that are running an operating system other than z/OS, such as z/VM.
303
8049ch08.fm
8.11 zEnterprise BladeCenter Extension (zBX) Model 003 software support

zBX Model 003 houses two types of blades: general purpose and solution specific.
8.11.1 IBM Blades

IBM offers a selected subset of IBM POWER7 blades that can be installed and operated on the zBX Model 003. These blades have been thoroughly tested to ensure compatibility and manageability in the IBM zEnterprise EC12 environment. The blades are virtualized by PowerVM Enterprise Edition, and their LPARs run either AIX Version 5 Release 3 TL12 (POWER6 mode), AIX Version 6 Release 1 TL5 (POWER7 mode), and AIX Version 7 Release 1, and subsequent releases. Applications supported on AIX can be deployed to blades. Also offered are selected IBM System x HX5 blades. Virtualization is provided by an integrated hypervisor, using Kernel-based Virtual Machines, and supporting Linux on System x and Microsoft Windows operating systems. Table 8-57 lists the operating systems supported by HX5 blades.
Table 8-57 Operating Support for zBX Model 003 HX5 Blades Operating system Support requirements
Linux on System x Microsoft Windows
Red Hat RHEL 5.5 and up, 6.0 and up SUSE SLES 10 (SP4)and up, SLES 11 (SP1)a and up Microsoft Windows Server 2008 R2b Microsoft Windows Server 2008 (SP2)b (Datacenter Edition recommended)
a. Latest patch level required b. 64 bit only
8.11.2 IBM WebSphere DataPower Integration Appliance XI50 for zEnterprise

The IBM WebSphere DataPower Integration Appliance XI50 for zEnterprise (DataPower XI50z) is a special-purpose, double-wide blade. The DataPower XI50z is a multifunctional appliance that can help provide multiple levels of XML optimization, streamline and secure valuable service-oriented architecture (SOA) applications, and provide drop-in integration for heterogeneous environments by enabling core enterprise service bus (ESB) functionality, including routing, bridging, transformation, and event handling. It can help to simplify, govern, and enhance the network security for XML and web services. Table 8-58 lists the minimum support requirements for DataPower Sysplex Distributor support.
Table 8-58 Minimum support requirements for DataPower Sysplex Distributor support. Operating system Support requirements
z/OS
z/OS V1R11 for IPv4 z/OS V1R12 for IPv4 and IPv6
304
8049ch08.fm
8.12 Software licensing considerations

The IBM software portfolio for the zEC12 includes operating system software11 (that is, z/OS, z/VM, z/VSE, and z/TPF) and middleware that runs on these operating systems. It also includes middleware for Linux on System z environments. zBX software products are covered by IPLA and additional agreements, such as the IBM International Passport Advantage Agreement, similarly to other AIX, Linux on System x and Windows environments. POWER/VM Enterprise Edition licenses need to be ordered for POWER7 blades. For the zEC12 two metric groups for software licensing are available from IBM, depending on the software product: Monthly License Charge (MLC) International Program License Agreement (IPLA) MLC pricing metrics have a recurring charge that applies each month. In addition to the right to use the product, the charge includes access to IBM product support during the support period. MLC metrics, in turn, include a variety of offerings. IPLA metrics have a single, up-front charge for an entitlement to use the product. An optional and separate annual charge called subscription and support entitles customers to access IBM product support during the support period and also receive future releases and versions at no additional charge. For details, consult the following references: The page web Learn about Software licensing, which has pointers to many documents and can be reached at: http://www-01.ibm.com/software/lotus/passportadvantage/about_software_licensing .html The web page Base license agreements allows finding several documents and can be reached at: http://www-03.ibm.com/software/sla/sladb.nsf/viewbla The IBM System z Software Pricing Reference Guide, at this web page: http://www.ibm.com/systems/z/resources/swprice/reference/index.html IBM System z Software Pricing web pages, which can be reached as follows: http://www.ibm.com/systems/z/resources/swprice/mlc/index.html The IBM International Passport Advantage Agreement can be downloaded from the Learn about Software licensing web page. ftp://ftp.software.ibm.com/software/passportadvantage/PA_Agreements/PA_Agreemen t_International_English.pdf The remainder of this section describes the software licensing options available on the zEC12.
11
Linux on System z distributions are not IBM products.
305
8049ch08.fm
8.12.1 MLC pricing metrics

MLC pricing applies to z/OS z/VSE or z/TPF operating systems. Any mix of z/OS, z/VM, Linux, z/VSE, and z/TPF images is allowed. Charges are based on processor capacity which is measured in Millions of Service Units (MSU) per hour.
Charge models
There is a variety of Workload Licence Charges (WLC) pricing structures that support two charge models: Variable charges (several pricing metrics): Variable charges apply to products such as z/OS, z/VSE, z/TPF, DB2, IMS, CICS, MQSeries, and Lotus Domino. There are several pricing metrics employing the following charge types: Full-capacity: The CPCs total number of MSUs is used for charging. Full-capacity is applicable when the clients CPC is not eligible for sub-capacity. Sub-capacity: Software charges are based on the utilization of the logical partitions where the product is running. Flat charges: Software products licensed under flat charges are not eligible for sub-capacity pricing. There is a single charge per CPC on the zEC12.
Sub-capacity
For eligible programs, sub-capacity allows software charges based on utilization of logical partitions instead of the CPCs total number of MSUs. Sub-capacity removes the dependency between software charges and CPC (hardware) installed capacity. The sub-capacity licensed products are charged monthly based on the highest observed 4-hour rolling average utilization of the logical partitions in which the product runs (with the exception of products licensed using the SALC pricing metric). This requires measuring the utilization and reporting it to IBM. The logical partitions 4-hour rolling average utilization can be limited by a defined capacity value on the partitions image profile. This activates the soft capping function of PR/SM, limiting the 4-hour rolling average partition utilization to the defined capacity value. Soft capping controls the maximum 4-hour rolling average usage (the last 4-hour average value at every 5-minute interval), but does not control the maximum instantaneous partition use. Also available is an LPAR group capacity limit, which allows you to set soft capping by PR/SM for a group of logical partitions running z/OS. Even using the soft capping option, the partitions use can reach up to its maximum share based on the number of logical processors and weights in the image profile. Only the 4-hour rolling average utilization is tracked, allowing utilization peaks above the defined capacity value. Some pricing metrics apply to stand alone System z servers, others apply to the aggregation of multiple zEC12 and System z servers workloads within the same Parallel Sysplex.
306
8049ch08.fm
For further information about WLC and details about how to combine logical partitions utilization, see the publication z/OS Planning for Workload License Charges, SA22-7506, available from the following web page: http://www-03.ibm.com/systems/z/os/zos/bkserv/find_books.html Metrics applicable to a stand-alone zEC12 are: Advanced Workload License Charges (AWLC) System z New Application License Charges (zNALC) Parallel Sysplex License Charges (PSLC) Metrics applicable to a zEC12 in an actively coupled Parallel Sysplex are: Advanced Workload License Charges (AWLC), when all CPCs are zEC12, z196 or z114 Variable Workload License Charges (VWLC) are only allowed under the AWLC Transition Charges for Sysplexes when not all CPCs are zEC12, z196 or z114. System z New Application License Charges (zNALC) Parallel Sysplex License Charges (PSLC)
8.12.2 Advanced Workload License Charges (AWLC)

Advanced Workload License Charges were introduced with the IBM zEnterprise 196. They utilize the measuring and reporting mechanisms, as well as the existing MSU tiers, from VWLC. As compared to VWLC the prices per tier have been lowered and prices for tiers 4, 5, and 6 are different, allowing for lower costs for charges above 875 MSUs. AWLC offers improved price performance as compared to with VWLC for all customers above 3 MSUs. Similarly to Workload Licence Charges, AWLC can be implemented in full-capacity or sub-capacity mode. AWLC applies to z/OS and z/TPF and their associated middleware products such as DB2, IMS, CICS, MQSeries, and Lotus Domino, when running on a zEC12. For additional information, see the AWLC web page: http://www-03.ibm.com/systems/z/resources/swprice/mlc/awlc.html
8.12.3 System z New Application License Charges (zNALC)

System z New Application License Charges offers a reduced price for the z/OS operating system on logical partitions running a qualified new workload application such as Java language business applications running under WebSphere Application Server for z/OS, Domino, SAP, PeopleSoft, and Siebel. z/OS with zNALC provides a strategic pricing model available on the full range of System z servers for simplified application planning and deployment. zNALC allows for aggregation across a qualified Parallel Sysplex, which can provide a lower cost for incremental growth across new workloads that span a Parallel Sysplex. For additional information, see the zNALC web page: http://www-03.ibm.com/systems/z/resources/swprice/mlc/znalc.html
307
8049ch08.fm
8.12.4 Select Application License Charges (SALC)

Select Application License Charges applies only to WebSphere MQ for System z. It allows a WLC customer to license MQ under product use rather than the sub-capacity pricing provided under WLC. WebSphere MQ is typically a low-usage product that runs pervasively throughout the environment. Clients who run WebSphere MQ at a low usage can benefit from SALC. Alternatively, you can still choose to license WebSphere MQ under the same metric as the z/OS software stack. A reporting function, which IBM provides in the operating system IBM Software Usage Report Program, is used to calculate the daily MSU number. The rules to determine the billable SALC MSUs for WebSphere MQ use the following algorithm: 1. Determines the highest daily usage of a program family, which is the highest of 24 hourly measurements recorded each day. Program refers to all active versions of MQ 2. Determines the monthly usage of a program family, which is the fourth highest daily measurement recorded for a month 3. Uses the highest monthly usage determined for the next billing period For additional information about SALC, see the Other MLC Metrics web page: http://www.ibm.com/systems/z/resources/swprice/mlc/other.html
8.12.5 Midrange Workload Licence Charges (MWLC)

Midrange Workload Licence Charges (MWLC) applies to z/VSE V4 and above when running on zEC12, z196, System z10 and z9 servers. The exceptions are the z10 BC and z9 BC servers at capacity setting A01 to which zELC applies. Similarly to Workload Licence Charges, MWLC can be implemented in full-capacity or sub-capacity mode. MWLC applies to z/VSE V4 and above, and several IBM middleware products for z/VSE. All other z/VSE programs continue to be priced as before. The z/VSE pricing metric is independent of the pricing metric for other systems (for instance, z/OS) that might be running on the same server. When z/VSE is running as a guest of z/VM, z/VM V5R4 or later is required. To report usage, the sub-capacity report tool is used. One SCRT report per server is required. For additional information, see the MWLC web page: http://www.ibm.com/systems/z/resources/swprice/mlc/mwlc.html
8.12.6 Parallel Sysplex Licence Charges (PSLC)

Parallel Sysplex Licence Charges (PSLC) applies to a large range of mainframe servers. The list can be obtained from the web page: http://www-03.ibm.com/systems/z/resources/swprice/reference/exhibits/hardware.html Although it can be applied to stand alone CPCs, the metric only provides aggregation benefits when applied to group of CPCs in an actively coupled Parallel Sysplex cluster according to IBMs terms and conditions.
308
8049ch08.fm
Aggregation allows charging a product based on the total MSU value of the machines where the product executes (as opposed to all the machines in the cluster). In an uncoupled environment software charges are based on the MSU capacity of the machine. For additional information, see the PSLC web page: http://www.ibm.com/systems/z/resources/swprice/mlc/pslc.html
8.12.7 System z International Program License Agreement (IPLA)

For zEC12 and System z systems, the following types of products are generally in the IPLA category: Data management tools CICS tools Application development tools Certain WebSphere for z/OS products Linux middleware products z/VM Versions V5 and V6 Generally, three pricing metrics apply to IPLA products for zEC12 and System z: Value unit (VU): Value unit pricing, which applies to the IPLA products that run on z/OS. Value unit pricing is typically based on the number of MSUs and allows for lower cost of incremental growth. Examples of eligible products are IMS tools, CICS tools, DB2 tools, application development tools, and WebSphere products for z/OS. Engine-based value unit (EBVU): Engine-based value unit pricing enables a lower cost of incremental growth with additional engine-based licenses purchased. Examples of eligible products include z/VM V5 and V6, and certain z/VM middleware, which are priced based on the number of engines. PVU: Processor value units (PVU). Here the number of engines is converted into processor value units under the Passport Advantage terms and conditions. Most Linux middleware is also priced based on the number of engines. For additional information, see the System z IPLA web page: http://www.ibm.com/systems/z/resources/swprice/zipla/index.html
8.13 References
For the most current planning information, see the support web site for each of the following operating systems: z/OS: http://www.ibm.com/systems/support/z/zos/ z/VM: http://www.ibm.com/systems/support/z/zvm/ z/VSE: http://www.ibm.com/servers/eserver/zseries/zvse/support/preventive.html
309
8049ch08.fm
z/TPF: http://www.ibm.com/software/htp/tpf/pages/maint.htm Linux on System z: http://www.ibm.com/systems/z/os/linux/
310
8049ch09.fm
Chapter 9.
System upgrades
This chapter provides an overview of IBM zEnterprise EC12 upgrade capabilities and procedures, with an emphasis on Capacity on Demand offerings. The upgrade offerings to the zEC12 systems have been developed from previous IBM System z systems. In response to customer demands and changes in market requirements, a number of features have been added. The provisioning environment gives the customer an unprecedented flexibility and a finer control over cost and value. For detailed tutorials on all aspects of system upgrades, see Resource Link1 - Customer Initiated Upgrade Information, then select Education. A list of available systems will help you select your particular product: https://www-304.ibm.com/servers/resourcelink/hom03010.nsf/pages/CIUInformation?Ope nDocument Given today's business environment, the growth capabilities provided by the zEC12 are plentiful, including the following benefits: Enabling exploitation of new business opportunities Supporting the growth of dynamic, smart environments Managing the risk of volatile, high-growth, and high-volume applications Supporting 24x365 application availability Enabling capacity growth during lock down periods Enabling planned-downtime changes without availability impacts This chapter discusses the following topics: Upgrade types on page 312 Concurrent upgrades on page 316 MES upgrades on page 322 Permanent upgrade through the CIU facility on page 328 On/Off Capacity on Demand on page 332 Capacity for Planned Event on page 342 Capacity Backup on page 344 Nondisruptive upgrades on page 347 Summary of Capacity on Demand offerings on page 352
1
Registration is required to access Resource Link.
311
8049ch09.fm
9.1 Upgrade types

The types of upgrades for a zEC12 are summarized in this section.
9.1.1 Overview
Upgrades can be categorized as described in the following discussion.
Permanent and temporary upgrades

Depending on the kind of situation, there is a need for different types of upgrades. After some time, depending on your growing workload, you might require more memory, additional I/O cards, or more processor capacity. However, in certain situations, only a short-term upgrade is necessary to handle a peak workload, or to temporarily replace a system that is down during a disaster or data center maintenance. The zEC12 offers the following solutions for such situations: Permanent: Miscellaneous Equipment Specification (MES): The MES upgrade order is always performed by IBM personnel. The result can be either real hardware or installation of Licensed Internal Code Configuration Control (LICCC) to the system. In both cases, installation is performed by IBM personnel. Customer Initiated Upgrade (CIU): Using the CIU facility for a given system requires that the online CoD buying feature (FC 9900) is installed on the system. The CIU facility only supports LICCC upgrades. Temporary: All temporary upgrades are LICCC-based. The one billable capacity offering is On/Off Capacity on Demand (On/Off CoD). The two replacement capacity offerings available are Capacity Backup (CBU) and Capacity for Planned Event (CPE). For descriptions see Terminology related to CoD for zEC12 systems on page 313 Tip: The MES provides system upgrades that can result in more enabled processors, different CP capacity level, as well as in additional books, memory, I/O drawers, and I/O cards (physical upgrade). An MES can also upgrade the zEnterprise BladeCenter Extension. Additional planning tasks are required for nondisruptive logical upgrades. MES is ordered through your IBM representative and delivered by IBM service personnel.
Concurrent and nondisruptive upgrades

Depending on the impact on system and application availability, upgrades can be classified as follows: Concurrent In general, concurrency addresses the continuity of operations of the hardware part of an upgrade. For instance, whether a system (as a box) is not required to be switched off during the upgrade. For details, see 9.2, Concurrent upgrades on page 316. Non-concurrent This type of upgrade requires switching off the hardware that is being upgraded. Examples of such non-concurrent upgrades, include model upgrades from any zEC12 model to the zEC12 HA1 model, as well as certain physical memory capacity upgrades.
312
8049ch09.fm
Disruptive An upgrade is considered disruptive, when resources modified or added to an operating system image, require that the operating system be recycled to configure the newly added resources. Nondisruptive Nondisruptive upgrades do not require the running software or operating system to be restarted for the upgrade to take effect. Thus, even concurrent upgrades can be disruptive to those operating systems or programs that do not support the upgrades while at the same time being nondisruptive to others. For details, see 9.8, Nondisruptive upgrades on page 347.
Terminology related to CoD for zEC12 systems

Table 9-1 lists the most frequently used terms related to Capacity on Demand for zEC12 systems.
Table 9-1 CoD terminology Term Description
Activated capacity Billable capacity Capacity Capacity Backup (CBU) Capacity for planned event CPE) Capacity levels
Capacity that is purchased and activated. Purchased capacity can be greater than the activated capacity. Capacity that helps handle workload peaks, either expected or unexpected. The one billable offering available is On/Off Capacity on Demand. Hardware resources (processor and memory) able to process workload can be added to the system through various capacity offerings. Capacity Backup allows you to replace model capacity or specialty engines to a backup system, in the event of an unforeseen loss of system capacity because of an emergency. Used when temporary replacement capacity is needed for a short term event. CPE activate processor capacity temporarily to facilitate moving machines between data centers, upgrades, and other routine management tasks. CPE is an offering of Capacity on Demand. Can be full capacity or subcapacity. For the zEC12 system, capacity levels for the CP engine are 7, 6, 5, and 4: 199, A0, A1 for capacity level 7nn 120 for capacity levels 6yy, 5yy 020 for capacity levels 4xx. An all IFL or an all ICF system has a capacity level of 400. Derived from the capacity level and the number of processors. For the zEC12 system, the capacity levels are 7nn, 6yy, 5yy, 4xx, where xx, yy or nn indicates the number of active CPs. The number of processors can have a range of: 199, A0, A1 for capacity level 7nn 120 for capacity levels 6yy, 5yy 020 for capacity levels 4xx. An all IFL or an all ICF system has a capacity level of 400. A Web-based facility where you can request processor and memory upgrades by using the IBM Resource Link and the system's remote support facility (RSF) connection. The ability of a computing system to increase or decrease its performance capacity as needed to meet fluctuations in demand. As a component of z/OS Capacity Provisioning, CPM monitors business-critical workloads that are running on z/OS systems on zEC12 systems.
Capacity setting
Customer Initiated Upgrade (CIU) Capacity on Demand (CoD) Capacity Provisioning Manager (CPM)
Chapter 9. System upgrades
313
8049ch09.fm
Term
Description
Customer profile Full capacity CP feature High water mark Installed record Model capacity identifier (MCI)
This information resides on Resource Link and contains customer and machine information. A customer profile can contain information about more than one machine. For zEC12 feature (CP7), provides full capacity. Capacity settings 7nn are full capacity settings. Capacity purchased and owned by the customer. The LICCC record has been downloaded, staged to the SE, and is now installed on the CPC. A maximum of eight different records can be concurrently installed and active. Shows the current active capacity on the system, including all replacement and billable capacity. For the zEC12, the model capacity identifier is in the form of 7nn, 6yy, 5yy, or 4xx, where xx, yy or nn indicates the number of active CPs. nn can have a range of 01 - 99, A0, A1 yy can have a range of 01 - 20 xx can have a range of 00 - 20. An all IFL or an all ICF system has a capacity level of 400. Keeps information about capacity settings active before any temporary capacity was activated.
Model Permanent Capacity Identifier (MPCI) Model Temporary Capacity Identifier (MTCI) On/Off Capacity on Demand (CoD) Permanent capacity Permanent upgrade Purchased capacity Permanent / Temporary entitlement record Replacement capacity Resource Link
Reflects the permanent capacity with billable capacity only, without replacement capacity. If no billable temporary capacity is active, Model Temporary Capacity Identifier equals Model Permanent Capacity Identifier. Represents a function that allows a spare capacity in a CPC to be made available to increase the total capacity of a CPC. For example, On/Off CoD can be used to acquire additional capacity for the purpose of handling a workload peak. The capacity that a customer purchases and activates. This amount might be less capacity than the total capacity purchased. LIC licensed by IBM to enable the activation of applicable computing resources, such as processors or memory, for a specific CIU-eligible machine on a permanent basis. Capacity delivered to and owned by the customer. It can be higher than permanent capacity. The internal representation of a temporary (TER) or permanent (PER) capacity upgrade processed by the CIU facility. An entitlement record contains the encrypted representation of the upgrade configuration with the associated time limit conditions. A temporary capacity used for situations in which processing capacity in other parts of the enterprise is lost during either a planned event or an unexpected disaster. The two replacement offerings available are, Capacity for Planned Events and Capacity Backup. The IBM Resource Link is a technical support website providing comprehensive set of tools and resources available from the IBM Systems technical support site: http://www.ibm.com/servers/resourcelink/ An option, selected by the customer, that a second approver control each Capacity on Demand order. When a secondary approval is required, the request is sent for approval or cancellation to the Resource Link secondary user ID. The point when a record representing a capacity upgrade, either temporary or permanent, has been retrieved and loaded on the Support Element (SE) disk. For the zEC12, CP features (CP4, CP5, and CP6) provide reduced capacity relative to the full capacity CP feature (CP7). An optional capacity that is added to the current system capacity for a limited amount of time. It can be capacity that is owned or not owned by the customer.
Secondary approval
Staged record Subcapacity Temporary capacity
314
8049ch09.fm
Term
Description
Vital product data (VPD)
Information that uniquely defines system, hardware, software, and microcode elements of a processing system.
9.1.2 Permanent upgrades

Permanent upgrades can be obtained as follows: Ordered through an IBM sales representative Initiated by the customer with the Customer Initiated Upgrade (CIU) on the IBM Resource Link Tip: The use of the CIU facility for a given system requires that the online CoD buying feature (FC 9900) is installed on the system. The CIU facility itself is enabled through the permanent upgrade authorization feature code (FC 9898).
Permanent upgrades ordered through an IBM representative

Through a permanent upgrade you can accomplish these tasks: Add processor books Add PCIe drawers and features Add model capacity Add specialty engines Add memory Activate unassigned model capacity or IFLs Deactivate activated model capacity or IFLs Activate channels Activate cryptographic engines Change specialty engine (re-characterization) Add zBX and zBX features: Chassis Racks DataPower Blades Entitlements Attention: Most of the MES can be concurrently applied, without disrupting the existing workload (see 9.2, Concurrent upgrades on page 316 for details). However, certain MES changes are disruptive (for example, model upgrades from any zEC12 model to the zEC12 HA1 model). Memory upgrades that require DIMM changes, can be made nondisruptive if there are multiple books and the flexible memory option is used.
Permanent upgrades initiated through CIU on the IBM Resource Link

Ordering a permanent upgrade by using the CIU application through Resource Link allows you to add capacity to fit within your existing hardware, as follows: Add model capacity Add specialty engines Add memory Activate unassigned model capacity or IFLs Deactivate activated model capacity or IFLs
315
8049ch09.fm
9.1.3 Temporary upgrades

System zEC12 offers three types of temporary upgrades: On/Off Capacity on Demand (On/Off CoD): This offering allows you to temporarily add additional capacity or specialty engines due to seasonal activities, period-end requirements, peaks in workload, or application testing. This temporary upgrade can only be ordered using the CIU application through Resource Link. Capacity Backup (CBU): This offering allows you to replace model capacity or specialty engines to a backup system in the event of an unforeseen loss of system capacity because of an emergency. Capacity for Planned Event (CPE): This offering allows you to replace model capacity or specialty engines due to a relocation of workload during system migrations or a data center move. CBU or CPE temporary upgrades can be ordered by using the CIU application through Resource Link or by calling your IBM sales representative. Temporary upgrades capacity changes can be billable or replacement.
Billable capacity
To handle a peak workload, you can activate up to double the purchased capacity of any PU type temporarily and be charge based on a daily basis. The one billable capacity offering is On/Off Capacity on Demand (On/Off CoD).
Replacement capacity
When a processing capacity is lost in another part of an enterprise, replacement capacity can be activated. It allows you to activate any PU type up to authorized limit. The two replacement capacity offerings are: Capacity Backup Capacity for Planned Event
9.2 Concurrent upgrades

Concurrent upgrades on the zEC12 can provide additional capacity with no system outage. In most cases, with prior planning and operating system support, a concurrent upgrade can also be nondisruptive to the operating system. Given today's business environment, the benefits of the concurrent capacity growth capabilities provided by the zEC12 are plentiful, and include, but are not limited to: Enabling exploitation of new business opportunities Supporting the growth of smart environments Managing the risk of volatile, high-growth, and high-volume applications Supporting 24x365 application availability Enabling capacity growth during lock down or frozen periods Enabling planned-downtime changes without affecting availability
316
8049ch09.fm
This capability is based on the flexibility of the design and structure, which allows concurrent hardware installation and Licensed Internal Code (LIC) control over the configuration. The subcapacity models allow additional configuration granularity within the family. The added granularity is available for models configured with up to 20 CPs and provides 60 additional capacity settings. Subcapacity models provide for CP capacity increase in two dimensions that can be used together to deliver configuration granularity. The first dimension is by adding CPs to the configuration, the second is by changing the capacity setting of the CPs currently installed to a higher model capacity identifier. The zEC12 allows the concurrent and non-disruptive addition of processors to a running logical partition. As a result, you can have a flexible infrastructure, in which you can add capacity without pre-planning. This function is supported by z/OS, z/VM and z/VSE. There are two ways to accomplish this: With planning ahead for the future need of extra processors. In the logical partitions profile reserved processors can be specified. When the extra processor(s) are actually installed, the number of active processors for that LPAR can be increased without the need for an IPL. Another (easier) way is to enable the dynamic addition of CPUs through the z/OS LOADxx member. Parameter DYNCPADD in member LOADxx should be set to ENABLE. The zEC12 supports dynamic CPU addition just as the z196 and z10 do. The operating system has to be z/OS V1R10 or higher. Another function concerns the system assist processor (SAP). When additional SAPs are concurrently added to the configuration, the SAP-to-channel affinity is dynamically re-mapped on all SAPs on the system to rebalance the I/O configuration.
9.2.1 Model upgrades

The zEC12 has a machine type and model, and model capacity identifiers: Machine type and model is 2827-Hvv: The vv can be 20, 43, 66, 89, or A1. The model number indicates how many PUs (vv) are available for customer characterization (A1 stands for 101). Model H20 has one book installed, model H43 contains two books, model H66 contains three books, and models H89 and HA1 contain four books. Model capacity identifiers are 4xx, 5yy, 6yy, or 7nn: The xx is a range of 00 - 202, yy is a range of 01 - 20 and nn is a range of 01 - 99, A0, A1. A1 means 101 decimal. A zEC12 with 101 customer usable processors is a zEC12 7A1. The model capacity identifier describes how many CPs are characterized (xx, yy or nn) and the capacity setting (4, 5, 6, or 7) of the CPs. A hardware configuration upgrade always requires additional physical hardware (books, cages, drawers or all of them3). A system upgrade can change either, or both, of the system model and the model capacity identifier (MCI). Note the following model upgrade information: LICCC upgrade: Does not change the system model 2827-Hvv, because additional books are not added Can change the model capacity identifier, the capacity setting, or both
2 3
zEC12 zero CP Model capacity identifier is 400. This applies to an all IFL or an all ICF systems. I/O cage and the 8-slot I/O drawer cannot be ordered as a MES on zEC12. They are available on carry forward only.
317
8049ch09.fm
Hardware installation upgrade: Can change the system model 2827-Hvv, if additional books are included Can change the model capacity identifier, the capacity setting, or both The system model and the model capacity identifier can be concurrently changed. Concurrent upgrades can be accomplished for both permanent and temporary upgrades. Tip: A model upgrade can be concurrent by using concurrent book add (CBA), except for upgrades to Model HA1.
Licensed Internal Code upgrades (MES ordered)

The LIC Configuration Control (LICCC) provides for system upgrades without hardware changes by activation of additional (previously installed) unused capacity. Concurrent upgrades through LICCC can be done for: Processors (CPs, ICFs, zAAPs, zIIPs, IFLs, and SAPs) if unused PUs are available on the installed books or if the model capacity identifier for the CPs can be increased. Memory, when unused capacity is available on the installed memory cards. Plan-ahead memory and the flexible memory option are available for customers to gain better control over future memory upgrades. See 2.5.6, Flexible Memory Option on page 53, and 2.5.7, Preplanned Memory on page 54 for more details.
Concurrent hardware installation upgrades (MES ordered)

Configuration upgrades can be concurrent when installing additional: Books (which contain processors, memory, and fanouts). Up to three books can be added concurrently on the model zEC12 H20. HCA and PCIe fanouts I/O cards, when slots are still available on the installed PCIe I/O drawers. PCIe I/O drawers. All of zBX and zBX features. However, the upgrade from a zBX model 002 to a zBX model 003 is disruptive. The concurrent I/O upgrade capability can be better exploited if a future target configuration is considered during the initial configuration.
Concurrent PU conversions (MES-ordered)

The zEC12 supports concurrent conversion between all PU types, any-to-any PUs including SAPs, to provide flexibility to meet changing business requirements. Attention: The LICCC-based PU conversions require that at least one PU, either CP, ICF, or IFL, remains unchanged. Otherwise, the conversion is disruptive. The PU conversion generates a new LICCC that can be installed concurrently in two steps: 1. The assigned PU is removed from the configuration. 2. The newly available PU is activated as the new PU type. Logical partitions might also have to free the PUs to be converted, and the operating systems must have support to configure processors offline or online so that the PU conversion can be done nondisruptively.
318
8049ch09.fm
Important: Customer planning and operator action are required to exploit concurrent PU conversion. Consider the following information about PU conversion: It is disruptive if all current PUs are converted to different types. It might require individual logical partition outage if dedicated PUs are converted. Unassigned CP capacity is recorded by a model capacity identifier. CP feature conversions change (increase or decrease) the model capacity identifier.
9.2.2 Customer Initiated Upgrade facility

The Customer Initiated Upgrade (CIU) facility is an IBM online system through which a customer can order, download, and install permanent and temporary upgrades for System z systems. Access to and use of the CIU facility requires a contract between the customer and IBM, through which the terms and conditions for use of the CIU facility are accepted. The use of the CIU facility for a given system requires that the online CoD buying feature code (FC 9900) is installed on the system. Although it can be installed on your zEC12 at any time, we suggest to add it when ordering a zEC12. The CIU facility itself is controlled through the permanent upgrade authorization feature code, FC 9898. After a customer has placed an order through the CIU facility, the customer will receive a notice that the order is ready for download. The customer can then download and apply the upgrade by using functions available through the HMC, along with the remote support facility. After all the prerequisites are met, the entire process, from ordering to activation of the upgrade, is performed by the customer. After download, the actual upgrade process is fully automated and does not require any on-site presence of IBM service personnel.
CIU prerequisites
The CIU facility supports LICCC upgrades only. It does not support I/O upgrades. All additional capacity required for an upgrade must be previously installed. Additional books or I/O cards cannot be installed as part of an order placed through the CIU facility. The sum of CPs, unassigned CPs, ICFs, zAAPs, zIIPs, IFLs, and unassigned IFLs cannot exceed the customer characterizable PU count of the installed books. The total number of zAAPs or zIIPs cannot each exceed the number of purchased CPs.
CIU registration and contract for CIU

To use the CIU facility, a customer must be registered and the system must be set up. After completing the CIU registration, access to the CIU application is available through the IBM Resource Link website: http://www.ibm.com/servers/resourcelink/ As part of the setup, the customer provides one resource link ID for configuring and placing CIU orders and, if required, a second ID as an approver. The IDs are then set up for access to the CIU support. The CIU facility is beneficial by allowing upgrades to be ordered and delivered much faster than through the regular MES process. To order and activate the upgrade, log on to the IBM Resource Link website and invoke the CIU application to upgrade a system for processors, or memory. Requesting a customer order approval to conform to customer operation policies is possible. As previously mentioned, customers can allow the definition of additional IDs to be authorized to access the CIU. Additional IDs can be authorized to enter or approve CIU orders, or only view existing orders.
319
8049ch09.fm
Permanent upgrades
Permanent upgrades can be ordered by using the CIU facility. Through the CIU facility, you can generate online permanent upgrade orders to concurrently add processors (CPs, ICFs, zAAPs, zIIPs, IFLs, and SAPs) and memory, or change the model capacity identifier, up to the limits of the installed books on an existing system.
Temporary upgrades
The base model zEC12 describes permanent and dormant capacity (Figure 9-1) using the capacity marker and the number of PU features installed on the system. Up to eight temporary offerings can be present. Each offering has its own policies and controls and each can be activated or deactivated independently in any sequence and combination. Although multiple offerings can be active at any time, if enough resources are available to fulfill the offering specifications, only one On/Off CoD offering can be active at any time.
Customer defined policy or user commands Manual operations
Orders downloaded from Retain/media
Management Application
HMC Application Query
API
Activation
Enforce terms and conditions and physical model limitations Up to 8 records installed and active
Authorization Layer R1 R2 R3 R4 R5 R6 R7 R8
Dormant capacity Base model Purchased capacity Change permanent capacity through CIU or MES order
Figure 9-1 The provisioning architecture
Temporary upgrades are represented in the system by a record. All temporary upgrade records, downloaded from the remote support facility (RSF) or installed from portable media, are resident on the Support Element (SE) hard drive. At the time of activation, the customer can control everything locally. Figure 9-1 shows a representation of the provisioning architecture. The authorization layer enables administrative control over the temporary offerings. The activation and deactivation can be driven either manually or under control of an application through a documented application program interface (API). By using the API approach, you can customize, at activation time, the resources necessary to respond to the current situation, up to the maximum specified in the order record. If the situation changes, you can add or remove resources without having to go back to the base
320
8049ch09.fm
configuration. This eliminates the need for temporary upgrade specification for all possible scenarios. However, for CPE the ordered configuration is the only possible activation. In addition, this approach enables you to update and replenish temporary upgrades, even in situations where the upgrades are already active. Likewise, depending on the configuration, permanent upgrades can be performed while temporary upgrades are active. Figure 9-2 shows examples of the activation sequence of multiple temporary upgrades.
R1
R2
R3
R4
CBU
CPE OOCoD CBU
Record and associated authorization
R3
CBU
CBU
CPE
R1 R3 R1 R3
CBU CPE
R4 R2
CBU
R4 R2
R3 R2
CBU
CBU
OOCoD
R2
OOCoD
OOCoD
OOCoD
OOCoD
R2
Time
Activation and usage of dormant resources over time
Figure 9-2 Example of temporary upgrade activation sequence
In the case of the R2, R3, and R1 being active at the same time, only parts of R1 can be activated, because not enough resources are available to fulfill all of R1. When R2 is then deactivated, the remaining parts of R1 can be activated as shown. Temporary capacity can be billable as On/Off Capacity on Demand (On/Off CoD), or replacement capacity as Capacity Backup (CBU) or CPE: On/Off CoD is a function that enables concurrent and temporary capacity growth of the system. On/Off CoD can be used for customer peak workload requirements, for any length of time, and has a daily hardware and maintenance charge. The software charges can vary according to the license agreement for the individual products. See your IBM Software Group representative for exact details. On/Off CoD can concurrently add processors (CPs, ICFs, zAAPs. zIIPs, IFLs, and SAPs), increase the model capacity identifier, or both, up to the limit of the installed books of an existing system, and is restricted to twice the currently installed capacity. On/Off CoD requires a contractual agreement between the customer and IBM. You decide whether to either pre-pay or post-pay On/Off CoD. Capacity tokens inside the records are used to control activation time and resources. CBU is a concurrent and temporary activation of additional CPs, ICFs, zAAPs, zIIPs, IFLs, and SAPs, an increase of the model capacity identifier, or both.
321
8049ch09.fm
CBU cannot be used for peak workload management in any form. As stated, On/Off CoD is the right way to do that. A CBU activation can last up to 90 days when a disaster or recovery situation occurs. CBU features are optional and require unused capacity to be available on installed books of the backup system, either as unused PUs or as a possibility to increase the model capacity identifier, or both. A CBU contract must be in place before the special code that enables this capability, can be loaded on the system. The standard CBU contract provides for five 10-day tests4 (the so called CBU test activation) and one 90-day activation over a five-year period. Contact your IBM Representative for details. You can run production workload on a CBU upgrade during a CBU test, provided that at least an equivalent amount of production capacity is shut down for the duration of the CBU test. If you already have existing CBU contracts, you will also need to sign an Amendment (US form #Z125-8145) with IBM. CPE is a concurrent and temporary activation of additional CPs, ICFs, zAAPs, zIIPs, IFLs, and SAPs or an increase of the model capacity identifier, or both. The CPE offering is used to replace temporary lost capacity within a customers enterprise for planned downtime events, for example, with data center changes. CPE cannot be used for peak load management of customer workload or for a disaster situation. The CPE feature requires unused capacity to be available on installed books of the backup system, either as unused PUs or as a possibility to increase the model capacity identifier on a subcapacity system, or both. A CPE contract must be in place before the special code that enables this capability can be loaded on the system. The standard CPE contract provides for one three-day planned activation at a specific date. Contact your IBM representative for details.
9.2.3 Summary of concurrent upgrade functions

Table 9-2 summarizes the possible concurrent upgrades combinations.
Table 9-2 Concurrent upgrade summary Type Name Upgrade Process
Permanent
MES Online permanent upgrade
CPs, ICFs, zAAPs, zIIPs, IFLs, SAPs, book, memory, and I/Os CPs, ICFs, zAAPs, zIIPs, IFLs, SAPs, and memory CPs, ICFs, zAAPs, zIIPs, IFLs, and SAPs CPs, ICFs, zAAPs, zIIPs, IFLs, and SAPs CPs, ICFs, zAAPs, zIIPs, IFLs, and SAPs
Installed by IBM service personnel Performed through the CIU facility Performed through the OOCoD facility Performed through the CBU facility Performed through the CPE facility
Temporary
On/Off CoD CBU CPE
9.3 MES upgrades

Miscellaneous equipment specification (MES) upgrades enable concurrent and permanent capacity growth. MES upgrades allow the concurrent adding of processors (CPs, ICFs, zAAPs, zIIPs, IFLs, and SAPs), memory capacity, and I/O ports as well as hardware and entitlements to the zEnterprise BladeCenter Extension. Regarding subcapacity models, MES upgrades allow the concurrent adjustment of both the number of processors and the capacity
4
zEC12 provides additional improvements in the CBU activation panels. These have been improved to prevent inadvertent CBU activation.
322
8049ch09.fm
level. The MES upgrade can be done using Licensed Internal Code Configuration Control (LICCC) only, by installing additional books, adding I/O cards, or a combination: MES upgrades for processors are done by any of the following methods: LICCC assigning and activating unassigned PUs up to the limit of the installed books LICCC to adjust the number and types of PUs or to change the capacity setting, or both Installing additional books and LICCC assigning and activating unassigned PUs on installed books MES upgrades for memory are done by either of the following methods: Using LICCC to activate additional memory capacity up to the limit of the memory cards on the currently installed books. Plan-ahead and flexible memory features enable you to have better control over future memory upgrades. For details about the memory features, see: 2.5.7, Preplanned Memory on page 54 2.5.6, Flexible Memory Option on page 53 Installing additional books and using LICCC to activate additional memory capacity on installed books Using the enhanced book availability (EBA), where possible, on multibook systems to add or change the memory cards MES upgrades for I/O are done by either of the following methods Installing additional I/O cards and supporting infrastructure if required on PCIe drawers that are already installed, or installing additional PCIe drawers to hold the new cards. MES upgrades for the zEnterprise BladeCenter Extension can only be performed through your IBM customer representative. An MES upgrade requires IBM service personnel for the installation. In most cases, the time required for installing the LICCC and completing the upgrade is short. To better exploit the MES upgrade function, it is highly desirable to carefully plan the initial configuration to allow a concurrent upgrade to a target configuration. The availability of PCIe I/O drawers has improved the flexibility to do unplanned I/O configuration changes concurrently. The store system information (STSI) instruction gives more useful and detailed information about the base configuration and about temporary upgrades. This enables you to more easily resolve billing situations where Independent Software Vendor (ISV) products are in use. The model and model capacity identifier returned by the STSI instruction are updated to coincide with the upgrade. See Store System Information (STSI) instruction on page 349 for more details. Upgrades: The MES provides the physical upgrade, resulting in more enabled processors, different capacity settings for the CPs, additional memory, I/O ports and I/O drawers. Additional planning tasks are required for non-disruptive logical upgrades (see Guidelines to avoid disruptive upgrades on page 352).
9.3.1 MES upgrade for processors

An MES upgrade for processors can concurrently add CPs, ICFs, zAAPs, zIIPs, IFLs, and SAPs to a zEC12 by assigning available PUs that reside on the books, through LICCC. Depending on the quantity of the additional processors in the upgrade, additional books might be required and can be concurrently installed before the LICCC is enabled. With the
323
8049ch09.fm
subcapacity models, additional capacity can be provided by adding CPs, by changing the capacity identifier on the current CPs, or by doing both. Limits: The sum of CPs, inactive CPs, ICFs, zAAPs, zIIPs, IFLs, unassigned IFLs, and SAPs cannot exceed the maximum limit of PUs available for customer use. The number of zAAPs and the number of zIIPs cannot exceed each the number of purchased CPs.
Example of MES upgrade: Figure 9-3 is an example of an MES upgrade for processors,
showing two upgrade steps.
2827-H20
MCI 708
Book 0
8 CPs
CP0
CP1
CP2
CP3
CP4
CP5
CP6
CP7
Spare
Spare
Spare
Spare Spare Spare Spare
Spare Spare Spare Spare
Spare
MES Upgrade +18 CPs (+ 1 Book)
2827-H43 MCI 726
Book 0 20 CPs
Book 1
6 CPs
CP0
CP1
CP2
CP3
CP4
CP5
CP6
CP7
CP8
CP9
CP20
CP21
CP22
CP23
CP24
CP25
Spare
Spare Spare Spare
CP10
CP11
CP12
CP13
CP14
CP15 CP16
CP17
CP18
CP19
Spare
Spare Spare Spare Spare Spare Spare Spare Spare
Spare
CIU Upgrade + 1 CP + 2 IFLs
2827-H43 MCI 727
Book 1
20 CPs
Book 1
7 CPs 2 IFLs
CP0
CP1
CP2
CP3
CP4
CP5
CP6
CP7
CP8
CP9
CP20
CP21
CP22
CP23
CP24
CP25
CP26 Spare
Spare
Spare
CP10 CP11
CP12
CP13
CP14
CP15 CP16
CP17
CP18
CP19
Spare Spare Spare Spare Spare Spare Spare Spare
IFL1
IFL0
Figure 9-3 MES for processor example
A model H20 (one book), model capacity identifier 708 (eight CPs), is concurrently upgraded to a model H43 (two books), with model capacity identifier (MCI) 726 (which is 26 CPs). The model upgrade requires adding a book and assigning and activating eighteen PUs as CPs. Then, model H43, MCI 726, is concurrently upgraded to a capacity identifier 727 (which is 27 CPs) with two IFLs by assigning and activating three more unassigned PUs (one as CP and two as IFLs). If needed, additional logical partitions can be created concurrently to use the newly added processors. Important: Up to 101 logical processors, including reserved processors, can be defined to a logical partition. However, do not define more processors to a logical partition than the target operating system supports. The following table describes the number of processors supported by various z/OS and z/VM releases.
324
8049ch09.fm
Table 9-3 Number of processors supported by operating system Operating System Number of processors supported
z/OS V1R10 w/ PTF z/OS V1R11 w/ PTF z/OS V1R12 w/ PTF z/OS V1R13 w/ PTF z/VM V5R4 - z/VM V6R2 z/VSE z/TPF Linux on System z
64 99 99 99 32 z/VSE Turbo Dispatcher can exploit up to 4 CPs and tolerates up to 10-way LPARs 86 CPs SUSE SLES 10: 64 CPs or IFLs SUSE SLES 11: 64 CPs or IFLs Red Hat RHEL 5: 80 CPs or IFLs Red Hat RHEL 6: 80 CPs or IFLs
Software charges, based on the total capacity of the system on which the software is installed, are adjusted to the new capacity after the MES upgrade. Software products that use Workload License Charge (WLC) might not be affected by the system upgrade, because their charges are based on partition utilization and not based on the system total capacity. For more information about WLC, see 8.12, Software licensing considerations on page 305.
9.3.2 MES upgrades for memory

MES upgrades for memory can concurrently add more memory in the following ways: Enabling, through LICCC, additional capacity up to the limit of the current installed memory cards Concurrently installing additional books and LICCC-enabling memory capacity on the new books. The Preplanned Memory Feature is available to allow better control over future memory upgrades. See 2.5.6, Flexible Memory Option on page 53, and 2.5.7, Preplanned Memory on page 54, for details about plan-ahead memory features. If the zEC12 is a multiple-book configuration, using the enhanced book availability (EBA) feature to remove a book and add memory cards or to upgrade the already-installed memory cards to a larger size and then using LICCC to enable the additional memory is possible. With proper planning, additional memory can be added non-disruptively to z/OS partitions and z/VM partitions. If necessary, new logical partitions can be created non-disruptively to use the newly added memory. Concurrency: Upgrades requiring DIMM changes can be concurrent by using the enhanced book availability (EBA) feature. Planning is required to see whether this is a viable option for your configuration. The use of the flexible memory option and the Preplanned Memory Feature (FC 1996 for 16 GB increment, FC 1990 for 32 GB increment) is the safest way to ensure that EBA can work with the least disruption.
325
8049ch09.fm
The one-book model H20 has a minimum of 80 GB physical installed memory. The customer addressable storage in this case is 32 GB. If you require more than that, an additional memory upgrade can install up to 704 GB of memory for customer use, by changing the existing DIMM sizes and adding additional DIMMs in all available slots in the book. Another possibility is to add memory by concurrently adding a second book with sufficient memory into the configuration and then using LICCC to enable that memory. A logical partition can dynamically take advantage of a memory upgrade if reserved storage has been defined to that logical partition. The reserved storage is defined to the logical partition as part of the image profile. Reserved memory can be configured online to the logical partition by using the LPAR dynamic storage reconfiguration (DSR) function. DSR allows a z/OS operating system image, and z/VM partitions, to add reserved storage to their configuration if any unused storage exists. The nondisruptive addition of storage to a z/OS and z/VM partition necessitates that pertinent operating system parameters have been prepared. If reserved storage has not been defined to the logical partition, the logical partition must be deactivated, the image profile changed, and the logical partition reactivated to allow the additional storage resources to be available to the operating system image.
9.3.3 MES upgrades for I/O

MES upgrades for I/O can concurrently add more I/O ports by one of the following methods: Installing additional I/O cards on an already installed PCIe I/O drawers slot The installed PCIe I/O drawer must provide the number of I/O slots required by the target configuration. Adding a new PCIe I/O drawer to hold the new I/O cards. Attention: Up to one I/O cage and up to two I/O drawers are supported if carried forward on an upgrade from a z196 or z10 EC. For a detailed description of I/O cages, I/O drawers and PCIe I/O drawers see 4.2, I/O system overview on page 126. The following table gives an overview of the number of I/O cages, I/O drawers and PCIe drawers that can be present in a zEC12.
Table 9-4 I/O cage and drawer summary Description New Build Carry Forward MES Add
I/O Cage I/O Drawer PCIe I/O Drawer
0 0 0-5
0-1 0-2 0-5
0 0 0-5
The number of cards that can be in a carry forward, is limited. The following table describes this in more detail.
Table 9-5 Number of I/O cards and I/O Cage / Drawers Number of cards in carry forward Number of I/O drawers Number of Cargo I/O cages
1-8
326
8049ch09.fm
Number of cards in carry forward
Number of I/O drawers
Number of Cargo I/O cages
9 - 16 17 - 28 29 - 36 37 - 44
2 0 1 2
0 1 1 1
Important: The maximum number of legacy I/O cards on a carry forward is 44. Depending on the amount of I/O features carried forward on an upgrade, the configurator will determine the number and mix of I/O cages, I/O drawers, and PCIe I/O drawers. To better exploit the MES for I/O capability, an initial configuration should be carefully planned to allow concurrent upgrades up to the target configuration. If legacy I/O features will be removed from the I/O cage/drawer, the configurator will not physically remove the I/O cage/drawer unless the I/O frame slot(s) are required to install a new PCIe I/O drawer. If an PCIe I/O drawer will be added to an existing zEC12 and some legacy features will need to be physically moved from one I/O cage/drawer to another I/O cage/drawer to completely empty the I/O cage/drawer for removal, legacy card moves are disruptive. I/O cage removal is disruptive. z/VSE, z/TPF,Linux on System z, and CFCC do not provide dynamic I/O configuration support. The installation of the new hardware is performed concurrently, but defining the new hardware to these operating systems requires an IPL.
Note: The zEC12 has a Hardware System Area (HSA) of 32 GB whereas the z196 has 16 GB HSA. It is not part of the customer purchased memory.
9.3.4 MES upgrades for the zBX

The MES upgrades for zBX can concurrently add blade entitlements, if there are any slots available in existing blade chassis. A blade chassis can be added if there is free space in an existing racks. Attention: Physically adding blades in the zBX is the clients responsibility, except for the DataPower blade.
zBX Model 003 upgrade

If a z196 is controlling a zBX Model 002 and the z196 is upgraded to a zEC12, then the zBX Model 002 is upgraded to a zBX Model 003. The zEC12 cannot control a zBX Model 002 (but can connect to it via the 10 GbE ToR switches). Some of the features and functions added by the zBX Model 003 are: Broadband RSF (Remote Support Facility) support. HMC application LIC for zEC12 and zBX Model 3 does not support dial modem use. Increased quantity of System x blades enablement to 56
327
8049ch09.fm
Enables potential of 20 Gb Ethernet bandwidth via link aggregation. Doubled 10 GbE cables between BladeCenter 10 GbE switch and 10 GbE TOR (Top of Rack) switch. Doubled 10 GbE cables between the BladeCenter 10 GbE switches. New version of the Advanced Management Module (AMM) in the BladeCenter chassis. Upgrade hypervisors and other firmware changes.
9.3.5 Summary of Plan-ahead features

A number of plan-ahead features exist for zEC12.The following list provides an overview of those features. Flexible memory Flexible memory has no Feature Code (FC) associated with it. The purpose of Flexible memory is to enable enhanced book availability. In the rare event a book is to be serviced, the flexible memory is activated to accommodate for the storage of the book that is to be taken offline. After the repair action, the memory is taken offline again and is made unavailable for usage. Preplanned memory Preplanned memory provides the ability to plan for non-disruptive memory upgrades. Any hardware required would be pre-plugged, based on a target capacity specified in advance. Pre-plugged hardware is enabled via a LICCC order when the additional memory capacity is needed. FC 1990 represents an amount of 32GB preplanned memory, FC 1996 represents an amount of 16GB preplanned memory. FC 1901 is used to activate previously installed preplanned memory and can activate all the pre-installed memory or subsets of it. Balanced Power Plan Ahead Balanced Power Plan Ahead is designed to anticipate future upgrades power needs on the zEC12. When more books are added to the machine, the power consumption will also rise. If necessary, one or more Bulk Power Regulators (BPRs) will have to be added. This extends the time needed for the upgrade. When ordering this feature, regardless of the configuration, all six BPR pairs will be installed and activated. Balanced Power Plan Ahead has FC 3003. Line Cord plan ahead This option provides the ability to plan ahead for the second set of line cords. It is normally not configured until the addition extra BPRs requires this. A plan ahead option will allow you to plan for a lengthy outage arising out of the need to install additional circuit breakers, power feeds or the routing of under floor cables. Line Cord plan has FC 2000. Tip: Accurate planning and definition of the target configuration will allow you to maximize the value of these plan ahead features.
9.4 Permanent upgrade through the CIU facility

By using the CIU facility (through the IBM Resource Link on the web), you can initiate a permanent upgrade for CPs, ICFs, zAAPs, zIIPs, IFLs, SAPs, or memory. When performed through the CIU facility, you add the resources; IBM personnel do not have to be present at the customer location. You can also unassign previously purchased CPs and IFLs processors through the CIU facility. 328
8049ch09.fm
The capability to add permanent upgrades to a given system through the CIU facility requires that the permanent upgrade enablement feature (FC 9898) be installed on the system. A permanent upgrade might change the system model capacity identifier 4xx, 5yy, 6yy, or 7nn if additional CPs are requested or the capacity identifier is changed as part of the permanent upgrade, but it cannot change the system model. If necessary, additional logical partitions can be created concurrently to use the newly added processors. Attention: A permanent upgrade of processors can provide a physical concurrent upgrade, resulting in more enabled processors available to a system configuration. Thus, additional planning and tasks are required for nondisruptive logical upgrades. See Guidelines to avoid disruptive upgrades on page 352 for more information. Maintenance charges are automatically adjusted as a result of a permanent upgrade. Software charges based on the total capacity of the system on which the software is installed are adjusted to the new capacity in place after the permanent upgrade is installed. Software products that use Workload License Charge (WLC) might not be affected by the system upgrade, because their charges are based on a logical partition utilization and not based on the system total capacity. See 8.12.2, Advanced Workload License Charges (AWLC) on page 307, for more information about WLC. Figure 9-4 illustrates the CIU facility process on the IBM Resource Link.
Customer ibm.com/servers/resourcelink
Internet
Online permanent order
Optional customer secondary order approval
Remote Support Facility

Figure 9-4 Permanent upgrade order example
The following sample sequence on the IBM Resource Link initiates an order: 1. Sign on to Resource Link. 2. Select the Customer Initiated Upgrade option from the main Resource Link page. Customer and system details associated with the user ID are listed. 3. Select the system that will receive the upgrade. The current configuration (PU allocation and memory) is shown for the selected system. 4. Select the Order Permanent Upgrade function. The Resource Link limits options to those that are valid or possible for the selected configuration (system). 5. After the target configuration is verified by the system, accept or cancel the order. An order is created and verified against the pre-established agreement.
329
8049ch09.fm
6. Accept or reject the price that is quoted. A secondary order approval is optional. Upon confirmation, the order is processed. The LICCC for the upgrade should be available within hours. Figure 9-5 illustrates the process for a permanent upgrade. When the LICCC is passed to the remote support facility, you are notified through an e-mail that the upgrade is ready to be downloaded.
EC12
Figure 9-5 CIU-eligible order activation example
The two major components in the process are ordering and retrieval (along with activation).
9.4.1 Ordering
Resource Link provides the interface that enables you to order a concurrent upgrade for a system. You can create, cancel, view the order, and view the history of orders that were placed through this interface. Configuration rules enforce only valid configurations being generated within the limits of the individual system. Warning messages are issued if you select invalid upgrade options. The process allows only one permanent CIU-eligible order for each system to be placed at a time. For a tutorial see: https://www-304.ibm.com/servers/resourcelink/hom03010.nsf/pages/CIUInformation?Ope nDocument
330
8049ch09.fm
Figure 9-6 shows the initial view of the machine profile on Resource Link.
Figure 9-6 Machine profile
The number of CPs, ICFs, zAAPs, zIIPs, IFLs, SAPs, memory size, and unassigned IFLs on the current configuration are displayed on the left side of the web page. Resource Link retrieves and stores relevant data associated with the processor configuration, such as the number of CPs and installed memory cards. It allows you to select only those upgrade options that are deemed valid by the order process. It allows upgrades only within the bounds of the currently installed hardware.
9.4.2 Retrieval and activation

After an order is placed and processed, the appropriate upgrade record is passed to the IBM support system for download. When the order is available for download, you receive an e-mail that contains an activation number. You can then retrieve the order by using the Perform Model Conversion task from the Support Element (SE), or through Single Object Operation to the SE from an HMC.
331
8049ch09.fm
In the Perform Model Conversion panel, select the Permanent upgrades option to start the process. See Figure 9-7.
Figure 9-7 zEC12 Perform Model Conversion panel
The panel provides several possible options. If you select the Retrieve and apply data option, you are prompted to enter the order activation number to initiate the permanent upgrade. See Figure 9-8.
Figure 9-8 Customer Initiated Upgrade Order Activation Number Panel
9.5 On/Off Capacity on Demand

On/Off Capacity on Demand (On/Off CoD) allows you to temporarily enable PUs and unassigned IFLs available within the current model, or to change capacity settings for CPs to help meet your peak workload requirements.
9.5.1 Overview
The capacity for CPs is expressed in MSUs. Capacity for speciality engines is expressed in number of speciality engines. Capacity tokens are used to limit the resource consumption for all types of processor capacity. Capacity tokens are introduced to provide better control over resource consumption when On/Off CoD offerings are activated. Token are represented as follows: 332
8049ch09.fm
For CP capacity, each token represents the amount of CP capacity that will result in one MSU of software cost for one day (an MSU-day token). For speciality engines, each token is equivalent to one speciality engine capacity for one day (an engine-day token). Tokens are by capacity type, MSUs for CP capacity, and number of engines for speciality engines. Each speciality engine type has its own tokens, and each On/Off CoD record has separate token pools for each capacity type. During the ordering sessions on Resource Link, you decide how many tokens of each type should be created in an offering record. Each engine type must have tokens for that engine type to be activated. Capacity that has no tokens cannot be activated. When resources from an On/Off CoD offering record containing capacity tokens are activated, a billing window is started. A billing window is always 24 hours in length. Billing takes place at the end of each billing window. The resources billed are the highest resource usage inside each billing window for each capacity type. An activation period is one or more complete billing windows, and represents the time from the first activation of resources in a record until the end of the billing window in which the last resource in a record is deactivated. At the end of each billing window, the tokens are decremented by the highest usage of each resource during the billing window. If any resource in a record does not have enough tokens to cover usage for the next billing window, the entire record will be deactivated. On/Off CoD requires that the Online CoD Buying feature (FC 9900) be installed on the system that is to be upgraded. On/Off CoD to Permanent Upgrade Option is a new offering, which is an offshoot of On/Off CoD and takes advantage of the aspects of the architecture. The customer is given a window of opportunity to assess capacity additions to their permanent configurations using On/Off CoD. If a purchase is made, the hardware On/Off CoD charges during this window, 3 days or less, are waived. If no purchase is made, then the customer is charged for the temporary use. The resources eligible for temporary use are CPs, ICFs, zAAPs, zIIPs, IFLs, and SAPs. Temporary addition of memory and I/O ports is not supported. Unassigned PUs that are on the installed books can be temporarily and concurrently activated as CPs, ICFs, zAAPs, zIIPs, IFLs, SAPs through LICCC, up to twice the currently installed CP capacity and up to twice the number of ICFs, zAAPs, zIIPs, or IFLs. This means that an On/Off CoD upgrade cannot change the system model. The addition of new books is not supported. However, activation of an On/Off CoD upgrade can increase the model capacity identifier 4xx, 5yy, 6yy, or 7nn.
9.5.2 Ordering
Concurrently installing temporary capacity by ordering On/Off CoD is possible, as follows: CP features equal to the MSU capacity of installed CPs IFL features up to the number of installed IFLs ICF features up to the number of installed ICFs zAAP features up to the number of installed zAAPs zIIP features up to the number of installed zIIPs SAPs up to four for model H20, eight for an H43, twelve for an H66, sixteen for an H89 and HA1. On/Off CoD can provide CP temporary capacity in two ways:
333
8049ch09.fm
By increasing the number of CPs. For subcapacity models, capacity can be added by increasing the number of CPs or by changing the capacity setting of the CPs, or both. The capacity setting for all CPs must be the same. If the On/Off CoD is adding CP resources that have a capacity setting different from the installed CPs, then the base capacity settings are changed to match. On/Off CoD has the following limits associated with its use: The number of CPs cannot be reduced. The target configuration capacity is limited to: Twice the currently installed capacity, expressed in MSUs for CPs. Twice the number of installed IFLs, ICFs, zAAPs, and zIIPs. The number of SAPs that can be activated depends on the model described in 9.2.1, Model upgrades on page 317.
On/Off CoD can be ordered as prepaid or postpaid: A prepaid On/Off CoD offering record contains resource descriptions, MSUs, number of speciality engines, and tokens that describe the total capacity that can be used. For CP capacity, the token contains MSU-days; for speciality engines, the token contains speciality engine-days. When resources on a prepaid offering are activated, they must have enough capacity tokens to allow the activation for an entire billing window, which is 24 hours. The resources remain active until you deactivate them or until one resource has consumed all of its capacity tokens. When that happens, all activated resources from the record are deactivated. A postpaid On/Off CoD offering record contains resource descriptions, MSUs, speciality engines, and can contain capacity tokens describing MSU-days and speciality engine-days. When resources in a postpaid offering record without capacity tokens are activated, those resources remain active until they are deactivated, or until the offering record expires, which is usually 180 days after its installation. When resources in a postpaid offering record with capacity tokens are activated, those resources must have enough capacity tokens to allow the activation for an entire billing window (24 hours). The resources remain active until they are deactivated or until one of the resource tokens are consumed, or until the record expires, usually 180 days after its installation. If one capacity token type is consumed, resources from the entire record are deactivated. As an example, for a zEC12 with capacity identifier 502, two ways to deliver a capacity upgrade through On/Off CoD exist: The first option is to add CPs of the same capacity setting. With this option, the model capacity identifier could be changed to a 503, which would add one additional CP (making it a 3-way) or to a 504, which would add two additional CPs (making it a 4-way). The second option is to change to a different capacity level of the current CPs and change the model capacity identifier to a 602 or to a 702. The capacity level of the CPs is increased but no additional CPs are added. The 502 could also be temporarily upgraded to a 603 as indicated in the table, thus increasing the capacity level and adding another processor. The 420 does not have an upgrade path through On/Off CoD. Preferably, use the Large Systems Performance Reference (LSPR) information to evaluate the capacity requirements according to your workload type. LSPR data for current IBM processors is available at this website:
334
8049ch09.fm
https://www-304.ibm.com/servers/resourcelink/lib03060.nsf/pages/lsprindex The On/Off CoD hardware capacity is charged on a 24-hour basis. There is a grace period at the end of the On/Off CoD day. This allows up to an hour after the 24-hour billing period to either change the On/Off CoD configuration for the next 24-hour billing period or deactivate the current On/Off CoD configuration. The times when the capacity is activated and deactivated are maintained in the zEC12 and sent back to the support systems. If On/Off capacity is already active, additional On/Off capacity can be added without having to return the system to its original capacity. If the capacity is increased multiple times within a 24-hour period, the charges apply to the highest amount of capacity active in the period. If additional capacity is added from an already active record containing capacity tokens, a check is made to control that the resource in question has enough capacity to be active for an entire billing window (24 hours). If that criteria is not met, no additional resources will be activated from the record. If necessary, additional logical partitions can be activated concurrently to use the newly added processor resources. Attention: On/Off CoD provides a concurrent hardware upgrade, resulting in more enabled processors available to a system configuration. Additional planning tasks are required for nondisruptive upgrades. See Guidelines to avoid disruptive upgrades on page 352. To participate in this offering, you must have accepted contractual terms for purchasing capacity through the Resource Link, established a profile, and installed an On/Off CoD enablement feature on the system. Subsequently, you can concurrently install temporary capacity up to the limits in On/Off CoD and use it for up to 180 days. Monitoring occurs through the system call-home facility and an invoice is generated if the capacity has been enabled during the calendar month. The customer will continue to be billed for use of temporary capacity until the system is returned to the original configuration. If the On/Off CoD support is no longer needed, the enablement code must be removed. On/Off CoD orders can be pre-staged in Resource Link to allow multiple optional configurations. The pricing of the orders is done at the time of the order, and the pricing can vary from quarter to quarter. Staged orders can have different pricing. When the order is downloaded and activated, the daily costs are based on the pricing at the time of the order. The staged orders do not have to be installed in order sequence. If a staged order is installed out of sequence, and later an order that was staged that had a higher price is downloaded, the daily cost will be based on the lower price. Another possibility is to store unlimited On/Off CoD LICCC records on the Support Element with the same or different capacities at any given time, giving greater flexibility to quickly enable needed temporary capacity. Each record is easily identified with descriptive names, and you can select from a list of records that can be activated. Resource Link provides the interface that allows you to order a dynamic upgrade for a specific system. You are able to create, cancel, and view the order. Configuration rules are enforced, and only valid configurations are generated based on the configuration of the individual system. After completing the prerequisites, orders for the On/Off CoD can be placed. The order process is to use the CIU facility on Resource Link. You can order temporary capacity for CPs, ICFs, zAAPs, zIIPs, IFLs, or SAPs. Memory and channels are not supported on On/Off CoD. The amount of capacity is based on the amount of owned capacity for the different types of resources. An LICCC record is established and staged to Resource Link for this order. After the record is activated, it has no expiration date.
335
8049ch09.fm
However, an individual record can only be activated once. Subsequent sessions require a new order to be generated, producing a new LICCC record for that specific order. Alternatively the customer can use an auto renewal feature to eliminate the need for a manual replenishment of the On/Off CoD order. The is feature is implemented in Resource Link and the customer will have to check this feature in the machine profile. See Figure 9-9 on page 336 for more details.
Figure 9-9 Order On/Off CoD record panel
9.5.3 On/Off CoD testing

Each On/Off CoD-enabled system is entitled to one no-charge 24-hour test. No IBM charges are assessed for the test, including no IBM charges associated with temporary hardware capacity, IBM software, or IBM maintenance. The test can be used to validate the processes to download, stage, install, activate, and deactivate On/Off CoD capacity. This test can have a maximum duration of 24 hours, commencing upon the activation of any capacity resource contained in the On/Off CoD record. Activation levels of capacity can change during the 24-hour test period. The On/Off CoD test automatically terminates at the end of the 24-hour period. In addition there is a possibility to perform administrative testing, through which no additional capacity is added to the system, but the customer can test all the procedures and automation for the management of the On/Off CoD facility. Figure 9-10 on page 337 is an example of an On/Off CoD order on the Resource Link web page.
336
8049ch09.fm
Figure 9-10 On/Off CoD order example
The example order in Figure 9-10 is a On/Off CoD order for 100% more CP capacity and for one ICF, one zAAP, one zIIP, and one SAP. The maximum number of CPs, ICFs, zAAPs, zIIPs, and IFLs is limited by the current number of available unused PUs of the installed books. The maximum number of SAPs is determined by the model number and the number of available PUs on the already installed books.
9.5.4 Activation and deactivation

When a previously ordered On/Off CoD is retrieved from Resource Link, it is downloaded and stored on the SE hard disk. You can activate the order when the capacity is needed, either manually or through automation. If the On/Off CoD offering record does not contain resource tokens, you must take action to deactivate the temporary capacity. Deactivation is accomplished from the Support Element and is nondisruptive. Depending on how the additional capacity was added to the logical partitions, you might be required to perform tasks at the logical partition level in order to remove the temporary capacity. For example, you might have to configure offline any CPs that had been added to the partition, or deactivate additional logical partitions created to use the temporary capacity, or both. On/Off CoD orders can be staged in Resource Link so that multiple orders are available. An order can only be downloaded and activated one time. If a different On/Off CoD order is required or a permanent upgrade is needed, it can be downloaded and activated without having to restore the system to its original purchased capacity. In support of automation, an API is provided that allows the activation of the On/Off CoD records. The activation is performed from the HMC and requires specifying the order number.
337
8049ch09.fm
With this API, automation code can be used to send an activation command along with the order number to the HMC to enable the order.
9.5.5 Termination
A customer is contractually obligated to terminate the On/Off CoD right-to-use feature when a transfer in asset ownership occurs. A customer can also choose to terminate the On/Off CoD right-to-use feature without transferring ownership. Application of FC 9898 terminates the right to use the On/Off CoD. This feature cannot be ordered if a temporary session is already active. Similarly, the CIU enablement feature cannot be removed if a temporary session is active. Any time the CIU enablement feature is removed, the On/Off CoD right-to-use is simultaneously removed. Reactivating the right-to-use feature subjects the customer to the terms and fees that apply at that time.
Upgrade capability during On/Off CoD

Upgrades involving physical hardware are supported while an On/Off CoD upgrade is active on a particular zEC12. LICCC-only upgrades can be ordered and retrieved from Resource Link and applied while an On/Off CoD upgrade is active. LICCC-only memory upgrades can be retrieved and applied while a On/Off CoD upgrade is active.
Repair capability during On/Off CoD

If the zEC12 requires service while an On/Off CoD upgrade is active, the repair can take place without affecting the temporary capacity.
Monitoring
When you activate an On/Off CoD upgrade, an indicator is set in vital product data. This indicator is part of the call-home data transmission, which is sent on a scheduled basis. A time stamp is placed into call-home data when the facility is deactivated. At the end of each calendar month, the data is used to generate an invoice for the On/Off CoD that has been used during that month.
Maintenance
The maintenance price is adjusted as a result of an On/Off CoD activation.
Software
Software Parallel Sysplex License Charge (PSLC) customers are billed at the MSU level represented by the combined permanent and temporary capacity. All PSLC products are billed at the peak MSUs enabled during the month, regardless of usage. Customers with WLC licenses are billed by product at the highest four-hour rolling average for the month. In this instance, temporary capacity does not necessarily increase the software bill until that capacity is allocated to logical partitions and actually consumed. Results from the STSI instruction reflect the current permanent and temporary CPs. See Store System Information (STSI) instruction on page 349 for more details.
9.5.6 z/OS capacity provisioning

The zEC12 provisioning capability combined with CPM functions in z/OS provides a flexible, automated process to control the activation of On/Off Capacity on Demand. The z/OS provisioning environment is shown in Figure 9-11 on page 339
338
8049ch09.fm
HMC SE zEnterprise EC12
PR/SM z/O image S
z/OS image(s) RMF RMF DDS RMF
Ethernet Switch
WLM
WLM
Provisioning policy
(SNMP)
Capacity Provisioning Manager (CPM)
CIM server CIM server
Capacity Provisioning Control Center (CPCC)

z/OS console(s)
Figure 9-11 The capacity provisioning infrastructure
The z/OS WLM manages the workload by goals and business importance on each z/OS system. WLM metrics are available through existing interfaces and are reported through Resource Measurement Facility (RMF) Monitor III, with one RMF gatherer for each z/OS system. Sysplex-wide data aggregation and propagation occur in the RMF Distributed Data Server (DDS). The RMF Common Information Model (CIM) providers and associated CIM models publish the RMF Monitor III data. The Capacity Provisioning Manager (CPM), a function inside z/OS, retrieves critical metrics from one or more z/OS systems through the Common Information Model (CIM) structures and protocol. CPM communicates to (local or remote) Support Elements and HMCs through the SNMP protocol. CPM has visibility of the resources in the individual offering records, and the capacity tokens. When CPM decides to activate resources, a check is performed to determine whether enough capacity tokens remain for the specified resource to be activated for at least 24 hours. If not enough tokens remain, no resource from the On/Off CoD record is activated. If a capacity token is completely consumed during an activation driven by the CPM, the corresponding On/Off CoD record is deactivated prematurely by the system, even if the CPM has activated this record, or parts of it. You do, however, receive warning messages if capacity tokens are getting close to being fully consumed. You receive the messages five days before a capacity token is fully consumed. The five days are based on the assumption that the consumption will be constant for the 5 days. You will need to put operational
339
8049ch09.fm
procedures in place to handle these situations. You can either deactivate the record manually, let it happen automatically, or replenish the specified capacity token by using the Resource Link application. The Capacity Provisioning Control Center (CPCC), which resides on a workstation, provides an interface to administer capacity provisioning policies. The CPCC is not required for regular CPM operation. The CPCC will over time be moved into the z/OSMF. Parts of the CPCC has been included in z/OSMF V1R13.
Capacity Provisioning Domain

The control over the provisioning infrastructure is executed by the CPM through the Capacity Provisioning Domain (CPD) controlled by the Capacity Provisioning Policy (CPP). The Capacity Provisioning Domain is shown in Figure 9-12.
Figure 9-12 The Capacity Provisioning Domain
The Capacity Provisioning Domain represents the central processor complexes (CPCs) that are controlled by the Capacity Provisioning Manager. The HMCs of the CPCs within a CPD must be connected to the same processor LAN. Parallel Sysplex members can be part of a CPD. There is no requirement that all members of a Parallel Sysplex must be part of the CPD, but participating members must all be part of the same CPD. The Capacity Provisioning Control Center (CPCC) is the user interface component. Administrators work through this interface to define domain configurations and provisioning policies, but it is not needed during production. The CPCC is installed on a Microsoft Windows workstation. CPM operates in four modes, allowing four different levels of automation:
340
8049ch09.fm
Manual mode: Use this command-driven mode when no CPM policy is active. Analysis mode: In analysis mode: CPM processes capacity-provisioning policies and informs the operator when a provisioning or deprovisioning action is required according to policy criteria. The operator determines whether to ignore the information or to manually upgrade or downgrade the system by using the HMC, the SE, or available CPM commands. Confirmation mode: In this mode, CPM processes capacity provisioning policies and interrogates the installed temporary offering records. Every action proposed by the CPM needs to be confirmed by the operator. Autonomic mode: This mode is similar to the confirmation mode, but no operator confirmation is required. A number of reports are available in all modes, containing information about workload and provisioning status and the rationale for provisioning guidelines. User interfaces are provided through the z/OS console and the CPCC application. The provisioning policy defines the circumstances under which additional capacity can be provisioned (when, which, and how). These are the three elements in the criteria: A time condition is when provisioning is allowed, as follows: Start time indicates when provisioning can begin. Deadline indicates that provisioning of additional capacity no longer allowed End time indicates that deactivation of additional capacity should begin. A workload condition is which work qualifies for provisioning. Parameters include: The z/OS systems that can execute eligible work Importance filter indicates eligible service class periods, identified by WLM importance Performance Index (PI) criteria: Activation threshold: PI of service class periods must exceed the activation threshold for a specified duration before the work is considered to be suffering. Deactivation threshold: PI of service class periods must fall below the deactivation threshold for a specified duration before the work is considered to no longer be suffering.
Included service classes are eligible service class periods. Excluded service classes are service class periods that should not be considered. Tip: If no workload condition is specified, the full capacity described in the policy will be activated and deactivated at the start and end times specified in the policy. Provisioning scope is how much additional capacity can be activated, expressed in MSUs. Specified in MSUs, number of zAAPs, and number of zIIPs must be one specification per CPC that is part of the Capacity Provisioning Domain. The maximum provisioning scope is the maximum additional capacity that can be activated for all the rules in the Capacity Provisioning Domain.
341
8049ch09.fm
The provisioning rule is as follows: In the specified time interval, if the specified workload is behind its objective, then up to the defined additional capacity can be activated. The rules and conditions are named and stored in the Capacity Provisioning Policy. For more information about z/OS Capacity Provisioning functions, see z/OS MVS Capacity Provisioning Users Guide, SA33-8299.
Planning considerations for using automatic provisioning

Although only one On/Off CoD offering can be active at any one time, several On/Off CoD offerings can be present on the system. Changing from one to another requires that the active one be stopped before the inactive one can be activated. This operation decreases the current capacity during the change. The provisioning management routines can interrogate the installed offerings, their content, and the status of the content of the offering. To avoid the decrease in capacity, it is best that only one On/Off CoD offering be created on the system by specifying the maximum allowable capacity. The Capacity Provisioning Manager can then, at the time when an activation is needed, activate a subset of the contents of the offering sufficient to satisfy the demand. If, at a later time, more capacity is needed, the Provisioning Manager can activate more capacity up to the maximum allowed increase. Having an unlimited number of offering records pre-staged on the SE hard disk is possible; changing content of the offerings if necessary is also possible. Attention: As previously mentioned, the CPM has control over capacity tokens for the On/Off CoD records. In a situation where a capacity token is completely consumed, the system deactivates the corresponding offering record. Therefore, it is highly desirable to prepare routines for catching the warning messages about capacity tokens being consumed, and have administrative routines in place for such a situation. The messages from the system begin five days before a capacity token is fully consumed. To avoid capacity records from being deactivated in this situation, replenish the necessary capacity tokens before they are completely consumed. In a situation where a CBU offering is active on a system and that CBU offering is 100% or more of the base capacity, activating any On/Off CoD is not possible because the On/Off CoD offering is limited to the 100% of the base configuration. The Capacity Provisioning Manager operates based on Workload Manager (WLM) indications, and the construct used is the performance index (PI) of a service class period. It is extremely important to select service class periods that are appropriate for the business application that needs more capacity. For example, the application in question might be executing through several service class periods, where the first period might be the important one. The application might be defined as importance level 2 or 3, but might depend on other work executing with importance level 1. Therefore, considering which workloads to control, and which service class periods to specify is very important.
9.6 Capacity for Planned Event

Capacity for Planned Event (CPE) is offered with the zEC12 to provide replacement backup capacity for planned down-time events. For example, if a server room requires an extension
342
8049ch09.fm
or repair work, replacement capacity can be installed temporarily on another zEC12 in the customers environment. Attention: CPE is for planned replacement capacity only and cannot be used for peak workload management. The feature codes are: FC 6833 Capacity for Planned Event enablement FC 0116 - 1 CPE Capacity Unit FC 0117 - 100 CPE Capacity Unit FC 0118 - 10000 CPE Capacity Unit FC 0119 - 1 CPE Capacity Unit-IFL FC 0120 - 100 CPE Capacity Unit-IFL FC 0121 - 1 CPE Capacity Unit-ICF FC 0122 - 100 CPE Capacity Unit-ICF FC 0123 - 1 CPE Capacity Unit-zAAP FC 0124 - 100 CPE Capacity Unit-zAAP FC 0125 - 1 CPE Capacity Unit-zIIP FC 0126 - 100 CPE Capacity Unit-zIIP FC 0127 - 1 CPE Capacity Unit-SAP FC 0128 - 100 CPE Capacity Unit-SAP The feature codes are calculated automatically when the CPE offering is configured. Whether using the eConfig tool or the Resource Link, a target configuration must be ordered consisting of a model identifier or a number of speciality engines, or both. Based on the target configuration, a number of feature codes from the list above is calculated automatically and a CPE offering record is constructed. CPE is intended to replace capacity lost within the enterprise because of a planned event such as a facility upgrade or system relocation. CPE is intended for short duration events lasting up to a maximum of three days. Each CPE record, after it is activated, gives the you access to dormant PUs on the system that you have a contract for as described above by the feature codes. Processor units can be configured in any combination of CP or specialty engine types (zIIP, zAAP, SAP, IFL, and ICF). At the time of CPE activation the contracted configuration will be activated. The general rule of one zIIP and one zAAP for each configured CP will be controlled for the contracted configuration. The processors that can be activated by CPE come from the available unassigned PUs on any installed book. CPE features can be added to an existing zEC12 non-disruptively. A one-time fee is applied for each individual CPE event depending on the contracted configuration and its resulting feature codes. Only one CPE contract can be ordered at a time. The base system configuration must have sufficient memory and channels to accommodate the potential requirements of the large CPE-configured system. It is important to ensure that all required functions and resources are available on the system where CPE is activated, including CF LEVELs for coupling facility partitions, memory, cryptographic functions, and including connectivity capabilities. The CPE configuration is activated temporarily and provides additional PUs in addition to the systems original, permanent configuration. The number of additional PUs is predetermined by the number and type of feature codes configured as described above by the feature codes. The number PUs that can be activated is limited by the unused capacity available on the system. For example: A model H43 with 16 CPs, no IFLs, ICFs, or zAAPs, has 27 unassigned PUs available.
343
8049ch09.fm
A model H66 with 28 CPs, 1 IFL, and 1 ICF has 36 unassigned PUs available. When the planned event is over, the system must be returned to its original configuration. You can deactivate the CPE features at any time before the expiration date. A CPE contract must be in place before the special code that enables this capability can be installed on the system. CPE features can be added to an existing zEC12 non-disruptively.
9.7 Capacity Backup

Capacity Backup (CBU) provides reserved emergency backup processor capacity for unplanned situations in which capacity is lost in another part of your enterprise and you want to recover by adding the reserved capacity on a designated zEC12. CBU is the quick, temporary activation of PUs and is available as follows: For up to 90 contiguous days, in case of a loss of processing capacity as a result of an emergency or disaster recovery situation For 10 days for testing your disaster recovery procedures or executing production workload, provided an amount of System z workload capacity equivalent to the CBU Upgrade capacity is shut down or otherwise made unusable during CBU test.5 Attention: CBU is for disaster and recovery purposes only and cannot be used for peak workload management or for a planned event.
9.7.1 Ordering
The CBU process allows for CBU to activate CPs, ICFs, zAAPs, zIIPs, IFLs, and SAPs. To be able to use the CBU process a CBU enablement feature (FC 9910) must be ordered and installed. You must order the quantity and type of PU that you require. The feature codes are: 6805: Additional test activations 6817: Total CBU years ordered 6818: CBU records ordered 6820: Single CBU CP-year 6821: 25 CBU CP-year 6822: Single CBU IFL-year 6823: 25 CBU IFL-year 6824: Single CBU ICF-year 6825: 25 CBU ICF-year 6826: Single CBU zAAP-year 6827: 25 CBU zAAP-year 6828: Single CBU zIIP-year 6829: 25 CBU zIIP-year 6830: Single CBU SAP-year 6831: 25 CBU SAP-year 6832: CBU replenishment The CBU entitlement record (6818) contains an expiration date that is established at the time of order and is dependent upon the quantity of CBU years (6817). You have the capability to
5
All new CBU contract documents contain new CBU Test terms to allow execution of production workload during CBU test. Existing CBU customers will need to execute IBM Customer Agreement Amendment for IBM System z Capacity Backup Upgrade Tests (US form #Z125-8145).
344
8049ch09.fm
extend your CBU entitlements through the purchase of additional CBU years. The number of 6817 per instance of 6818 remains limited to five and fractional years are rounded up to the near whole integer when calculating this limit. For instance, if there are two years and eight months to the expiration date at the time of order, the expiration date can be extended by no more than two additional years. One test activation is provided for each additional CBU year added to the CBU entitlement record. Feature code 6805 allows for ordering additional tests in increments of one. The total number of tests allowed is 15 for each feature code 6818. The processors that can be activated by CBU come from the available unassigned PUs on any installed book. The maximum number of CBU features that can be ordered is 101. The number of features that can be activated is limited by the number of unused PUs on the system. For example: A model H20 with Capacity Model Identifier 410 can activate up to 20 CBU features: ten to change the capacity setting of the existing CPs and ten to activate unused PUs. A model H43 with 15 CPs, four IFLs, and one ICF has twenty three unused PUs available. It can activate up to twenty three CBU features. However, the ordering system allows for over-configuration in the order itself. You can order up to 101 CBU features regardless of the current configuration, however at activation, only the capacity already installed can be activated. Note that at activation, you can decide to activate only a sub-set of the CBU features that are ordered for the system. Subcapacity makes a difference in the way the CBU features are done. On the full-capacity models, the CBU features indicate the amount of additional capacity needed. If the amount of necessary CBU capacity is equal to four CPs, then the CBU configuration would be four CBU CPs. The subcapacity models have multiple capacity settings of 4xx, 5yy, or 6yy; the standard models have capacity setting 7nn. The number of CBU CPs must be equal to or greater than the number of CPs in the base configuration, and all the CPs in the CBU configuration must have the same capacity setting. For example, if the base configuration is a 2-way 402, then providing a CBU configuration of a 4-way of the same capacity setting requires two CBU feature codes. If the required CBU capacity changes the capacity setting of the CPs, then going from model capacity identifier 402 to a CBU configuration of a 4-way 504 would require four CBU feature codes with a capacity setting of 5yy. If the capacity setting of the CPs is changed, then more CBU features are required, not more physical PUs. This means that your CBU contract requires more CBU features if the capacity setting of the CPs is changed. Note that CBU can add CPs through LICCC-only, and the zEC12 must have the proper number of books installed to allow the required upgrade. CBU can change the model capacity identifier to a higher value than the base setting, 4xx, 5yy, or 6yy, but does not change the system model. The CBU feature cannot decrease the capacity setting. A CBU contract must be in place before the special code that enables this capability can be installed on the system. CBU features can be added to an existing zEC12 non-disruptively. For each machine enabled for CBU, the authorization to use CBU is available for a definite number of years of 1-5 years. The alternate configuration is activated temporarily and provides additional capacity greater than the systems original, permanent configuration. At activation time, you determine the capacity required for a given situation, and you can decide to activate only a sub-set of the capacity specified in the CBU contract.
345
8049ch09.fm
The base system configuration must have sufficient memory and channels to accommodate the potential requirements of the large CBU target system. Ensure that all required functions and resources are available on the backup systems, including CF LEVELs for coupling facility partitions, memory, and cryptographic functions, as well as connectivity capabilities. When the emergency is over (or the CBU test is complete), the system must be taken back to its original configuration. The CBU features can be deactivated by the customer at any time before the expiration date. Failure to deactivate the CBU feature before the expiration date can cause the system to degrade gracefully back to its original configuration. The system does not deactivate dedicated engines, or the last of in-use shared engines. Planning: CBU for processors provides a concurrent upgrade, resulting in more enabled processors or changed capacity settings available to a system configuration, or both. You decide, at activation time, to activate a sub-set of the CBU features ordered for the system. Thus, additional planning and tasks are required for nondisruptive logical upgrades. See Guidelines to avoid disruptive upgrades on page 352. For detailed instructions, see the System z Capacity on Demand Users Guide, SC28-6846.
9.7.2 CBU activation and deactivation

The activation and deactivation of the CBU function is a customer responsibility and does not require on-site presence of IBM service personnel. The CBU function is activated/deactivated concurrently from the HMC using the API. On the SE, CBU is be activated either using the Perform Model Conversion task or through the API (API enables task automation).
CBU activation
CBU is activated from the SE by using the Perform Model Conversion task or through automation by using API on the SE or the HMC. In case of real disaster, use the Activate CBU option to activate the 90-day period.
Image upgrades
After the CBU activation, the zEC12 can have more capacity, more active PUs, or both. The additional resources go into the resource pools and are available to the logical partitions. If the logical partitions have to increase their share of the resources, the logical partition weight can be changed or the number of logical processors can be concurrently increased by configuring reserved processors online. The operating system must have the capability to concurrently configure more processors online. If necessary, additional logical partitions can be created to use the newly added capacity.
CBU deactivation
To deactivate the CBU, the additional resources have to be released from the logical partitions by the operating systems. In some cases, this is a matter of varying the resources offline. In other cases, it can mean shutting down operating systems or deactivating logical partitions. After the resources have been released, the same facility on the SE is used to turn off CBU. To deactivate CBU, click the Undo temporary upgrade option from the Perform Model Conversion task on the SE.
CBU testing
Test CBUs are provided as part of the CBU contract. CBU is activated from the SE by using the Perform Model Conversion task. Select the test option to initiate a 10-day test period. A standard contract allows one test per CBU year. However, you can order additional tests in
346
8049ch09.fm
increments of one up to a maximum of 15 for each CBU order. The test CBU has a 10-day limit and must be deactivated in the same way as the real CBU, using the same facility through the SE. Failure to deactivate the CBU feature before the expiration date can cause the system to degrade gracefully back to its original configuration. The system does not deactivate dedicated engines, or the last of in-use shared engine. Testing can be accomplished by ordering a diskette, calling the support center, or using the facilities on the SE. The customer has the possibility of purchasing additional tests.
CBU example
An example of a capacity backup operation could be as follows; 12 CBU features are installed on a backup model H43 with model capacity identifier 708. When a production model H20 with model capacity identifier 708 has an unplanned outage, the backup system can be temporarily upgraded from model capacity identifier 708 to 720, so that the capacity can take over the workload from the failed production system. Furthermore, you can configure systems to back up each other. For example, if you use two models of H20 model capacity identifier 705 for the production environment, each can have five or more features installed. If one system suffers an outage, the other one uses a temporary upgrade to recover approximately the total original capacity.
9.7.3 Automatic CBU enablement for GDPS

The intent of the Geographically Dispersed Parallel Sysplex (GDPS) CBU is to enable automatic management of the PUs provided by the CBU feature in the event of a system or site failure. Upon detection of a site failure or planned disaster test, GDPS will concurrently add CPs to the systems in the take-over site to restore processing power for mission-critical production workloads. GDPS automation does the following tasks: Performs the analysis required to determine the scope of the failure. This minimizes operator intervention and the potential for errors. Automates authentication and activation of the reserved CPs. Automatically restarts the critical applications after reserved CP activation. Reduces the outage time to restart critical workloads from several hours to minutes. The GDPS service is for z/OS only, or for z/OS in combination with Linux on System z.
9.8 Nondisruptive upgrades

Continuous availability is an increasingly important requirement for most customers, and even planned outages are no longer acceptable. Although Parallel Sysplex clustering technology is the best continuous availability solution for z/OS environments, non-disruptive upgrades within a single system can avoid system outages and are suitable to additional operating system environments. The zEC12 allows concurrent upgrades, meaning that dynamically adding more capacity to the system is possible. If operating system images running on the upgraded system do not require disruptive tasks in order to use the new capacity, the upgrade is also non-disruptive. This means that power-on reset (POR), logical partition deactivation, and IPL do not have to take place. If the concurrent upgrade is intended to satisfy an image upgrade to a logical partition, the operating system running in this partition must also have the capability to concurrently
347
8049ch09.fm
configure more capacity online. z/OS operating systems have this capability. z/VM can concurrently configure new processors and I/O devices online, memory can be dynamically added to z/VM partitions. If the concurrent upgrade is intended to satisfy the need for more operating system images, additional logical partitions can be created concurrently on the zEC12 system, including all resources needed by such logical partitions. These additional logical partitions can be activated concurrently. These enhanced configuration options are made available through the separate HSA, which was introduced on the System z196. Linux operating systems in general do not have the capability of adding more resources concurrently. However, Linux, and other types of virtual machines running under z/VM, can benefit from the z/VM capability to non-disruptively configure more resources online (processors and I/O). With z/VM, Linux guests can manipulate their logical processors through the use of the Linux CPU hotplug daemon. The daemon can start and stop logical processors based on the Linux average load value. The daemon is available in Linux SLES 10 SP2 and Red Hat RHEL V5R4.
9.8.1 Components
The following components can be added, depending on considerations described here.
Processors
CPs, ICFs, zAAPs, zIIPs, IFLs, and SAPs can be concurrently added to a zEC12 if unassigned PUs are available on any installed book. The number of zAAPs cannot exceed the number of CPs plus unassigned CPs. The same holds true for the zIIPs. Additional books can also be installed concurrently, allowing further processor upgrades. Concurrent upgrades are not supported with PUs defined as additional SAPs. If necessary, additional logical partitions can be created concurrently to use the newly added processors. The Coupling Facility Control Code (CFCC) can also configure more processors online to coupling facility logical partitions by using the CFCC image operations window.
Memory
Memory can be concurrently added up to the physical installed memory limit. Additional books can also be installed concurrently, allowing further memory upgrades by LICCC, enabling memory capacity on the new books. Using the previously defined reserved memory, z/OS operating system images, and z/VM partitions, can dynamically configure more memory online, allowing nondisruptive memory upgrades. Linux on System z supports Dynamic Storage Reconfiguration.
I/O
I/O cards can be added concurrently if all the required infrastructure (I/O slots and HCAs) is present on the configuration.
348
8049ch09.fm
I/O drawers and PCIe I/O drawers can be added concurrently without preplanning if free space is available in one of the frames and the configuration permits. Dynamic I/O configurations are supported by certain operating systems (z/OS and z/VM), allowing nondisruptive I/O upgrades. However, having dynamic I/O reconfiguration on a stand-alone coupling facility system is not possible because there is no operating system with this capability running on this system.
Cryptographic adapters
Crypto Express4S and Crypto Express3 features can be added concurrently if all the required infrastructure is in the configuration.
9.8.2 Concurrent upgrade considerations

By using MES upgrade, On/Off CoD, CBU, or CPE, a zEC12 can be concurrently upgraded from one model to another, either temporarily or permanently. Enabling and using the additional processor capacity is transparent to most applications. However, certain programs depend on processor model-related information, for example, Independent Software Vendor (ISV) products. You should consider the effect on the software running on a zEC12 when you perform any of these configuration upgrades.
Processor identification
Two instructions are used to obtain processor information: Store System Information instruction (STSI) STSI reports the processor model and model capacity identifier for the base configuration and for any additional configuration changes through temporary upgrade actions. It fully supports the concurrent upgrade functions and is the preferred way to request processor information. Store CPU ID instruction (STIDP) STIDP is provided for purposes of backward compatibility.
Store System Information (STSI) instruction

Figure 9-13 on page 350 shows the relevant output from the STSI instruction. The STSI instruction returns the model capacity identifier for the permanent configuration, and the model capacity identifier for any temporary capacity. This is key to the functioning of Capacity on Demand offerings.
349
8049ch09.fm
Figure 9-13 STSI output on zEC12
The model capacity identifier contains the base capacity, the On/Off CoD, and the CBU. The Model Permanent Capacity Identifier and the Model Permanent Capacity Rating contain the base capacity of the system, and the Model Temporary Capacity Identifier and Model Temporary Capacity Rating contain the base capacity and the On/Off CoD.
Store CPU ID instruction

The STIDP instruction provides information about the processor type, serial number, and logical partition identifier. See Table 9-6. The logical partition identifier field is a full byte to support greater than 15 logical partitions. 350
8049ch09.fm
Table 9-6 STIDP output for zEC12 Description Version code CPU identification number Machine type number Logical partition 2-digit indicator
Bit position Value
0-7 x00a
8 - 15 Logical partition IDb
16 - 31 4-digit number derived from the CPC serial number
32 - 48 x2827
48 - 63 x8000c
a. The version code for zEC12 is x00. b. The logical partition identifier is a two-digit number in the range of 00 - 3F. It is assigned by the user on the image profile through the Support Element or HMC. c. High order bit on indicates that the logical partition ID value returned in bits 8 - 15 is a two-digit value.
When issued from an operating system running as a guest under z/VM, the result depends on whether the SET CPUID command has been used, as follows: Without the use of the SET CPUID command, bits 0 - 7 are set to FF by z/VM, but the remaining bits are unchanged, which means that they are exactly as they would have been without running as a z/VM guest. If the SET CPUID command has been issued, bits 0 - 7 are set to FF by z/VM and bits 8 31 are set to the value entered in the SET CPUID command. Bits 32 - 63 are the same as they would have been without running as a z/VM guest. Table 9-7 lists the possible output returned to the issuing program for an operating system running as a guest under z/VM.
Table 9-7 z/VM guest STIDP output for zEC12 Description Version code CPU identification number Machine type number Logical partition 2-digit indicator
Bit position Without SET CPUID command With SET CPUID command
0-7 xFF
8 - 15 Logical partition ID
16 - 31 4-digit number derived from the CPC serial number
32 - 48 x2827
48 - 63 x8000
xFF
6-digit number as entered by the command SET CPUID = nnnnnn
x2827
x8000
Planning for nondisruptive upgrades

Online permanent upgrades, On/Off CoD, CBU, and CPE can be used to concurrently upgrade a zEC12. However, certain situations require a disruptive task in order to enable the new capacity that was recently added to the system. Some of these situation can be avoided if planning is done in advance. Planning ahead is a key factor for non-disruptive upgrades. The following list describes main reasons for disruptive upgrades. However, by carefully planning and by reviewing Guidelines to avoid disruptive upgrades on page 352, you can minimize the need for these outages: z/OS logical partition processor upgrades when reserved processors were not previously defined are disruptive to image upgrades. Logical partition memory upgrades when reserved storage was not previously defined are disruptive to image upgrades. z/OS and z/VM support this function.
351
8049ch09.fm
Installation of an I/O cage is disruptive. An I/O upgrade when the operating system cannot use the dynamic I/O configuration function is disruptive. Linux, z/VSE, z/TPF, and CFCC do not support dynamic I/O configuration.
Guidelines to avoid disruptive upgrades

Based on the previous list of reasons for disruptive upgrades (Planning for nondisruptive upgrades), here are several guidelines for avoiding or at least minimizing these situations, increasing the possibilities for nondisruptive upgrades: For z/OS logical partitions configure as many reserved processors (CPs, ICFs, zAAPs, and zIIPs) as possible. Configuring reserved processors for all logical z/OS partitions before their activation enables them to be non-disruptively upgraded. The operating system running in the logical partition must have the ability to configure processors online. The total number of defined and reserved CPs cannot exceed the number of CPs supported by the operating system. z/OS V1R10 with PTFs supports up to 64 processors including CPs, zAAPs, and zIIPs. z/OS V1R11, z/OS V1R12 and z/OS V1R13 with PTFs support up to 99 processors including CPs, zAAPs, and zIIPs. z/VM supports up to 32 processors. Configure reserved storage to logical partitions. Configuring reserved storage for all logical partitions before their activation enables them to be non-disruptively upgraded. The operating system running in the logical partition must have the ability to configure memory online. The amount of reserved storage can be above the book threshold limit, even if no other book is already installed. The current partition storage limit is 1TB. z/OS and z/VM support this function. Consider the flexible and plan-ahead memory options. Use a convenient entry point for memory capacity and consider the memory options to allow future upgrades within the memory cards already installed on the books. For details about the offerings, see 2.5.6, Flexible Memory Option on page 53 2.5.7, Preplanned Memory on page 54
Considerations when installing additional books

During an upgrade, additional books can be installed concurrently. Depending on the number of additional books in the upgrade and your I/O configuration, a fanout rebalancing might be desirable for availability reasons.
9.9 Summary of Capacity on Demand offerings

The capacity on demand infrastructure and its offerings are major features introduced with the zEC12 system. The reasons for the introduction of these features are based on numerous customer requirements for more flexibility, granularity, and better business control over the System z infrastructure, operationally and financially. One major customer requirement is to dismiss the necessity for a customer authorization connection to the IBM Resource Link system at the time of activation of any offering. This requirement is being met by the z196 and zEC12. After the offerings have been installed on the zEC12, they can be activated at any time, completely at the customers discretion. No intervention through IBM or IBM personnel is necessary. In addition, the activation of the Capacity Backup does not require a password. 352
8049ch09.fm
The zEC12 can have up to eight offerings installed at the same time, with the limitation that only one of them can be an On/Off Capacity on Demand offering; the others can be any combination. The installed offerings can be activated fully or partially, and in any sequence and any combination. The offerings can be controlled manually through command interfaces on the HMC, or programmatically through a number of APIs, so that IBM applications, ISV programs, or customer-written applications, can control the usage of the offerings. Resource consumption (and thus financial exposure) can be controlled by using capacity tokens in On/Off CoD offering records. The Capacity Provisioning Manager (CPM) is an example of an application that uses the Capacity on Demand APIs to provision On/Off CoD capacity based on the requirements of the executing workload. The CPM cannot control other offerings.
9.10 References
For more information, see the following publications: IBM System z10 Enterprise Class Capacity On Demand, SG24-7504 IBM zEnterprise 196 Capacity on Demand Users Guide, SC28-2605 IBM zEnterprise EC12 Capacity on Demand Users Guide, SCxx-xxxx
353
8049ch09.fm
354
8049ch10.fm
10
Chapter 10.
RAS
This chapter describes a few of the reliability, availability, and serviceability (RAS) features of the IBM zEnterprise EC12. The zEC12 design is focused on providing higher availability by reducing planned and unplanned outages. RAS can be accomplished with improved concurrent replace, repair, and upgrade functions for processors, memory, books, and I/O. RAS also extends to the nondisruptive capability for installing Licensed Internal Code (LIC) updates. In most cases a capacity upgrade can be concurrent without a system outage. As an extension to the RAS capabilities, we discuss environmental controls implemented in the system to help reduce power consumption and cooling requirements. The design of the memory on the zEC12 is implemented based a fully redundant memory infrastructure, Redundant Array of Independent Memory (RAIM), a concept similar to the RAID design used in external disk storage systems. RAIM was first introduced with z196. The zEnterprise CPCs are the only systems in the industry offering this level of memory design. To make the delivery and transmission of microcode (LIC), fixes and restoration/backup files are digitally signed. Any data transmitted to IBM Support is encrypted. The design goal for the zEC12 has been to remove all sources of planned outages. This chapter discusses the following topics: zEC12 availability characteristics on page 356 zEC12 RAS functions on page 357 zEC12 Enhanced book availability on page 360 zEC12 Enhanced driver maintenance on page 368 RAS capability for the HMC and SE on page 369 RAS capability for zBX on page 370 Considerations for PowerHA in zBX environment on page 371 IBM System z Advanced Workload Analysis Reporter (IBM zAware) on page 373 RAS capability for Flash Express on page 374
355
8049ch10.fm
10.1 zEC12 availability characteristics

The following functions include availability characteristics on the zEC12: Enhanced book availability (EBA): EBA is a procedure under which a book in a multi-book machine can be removed and reinstalled during an upgrade or repair action with no impact on the executing workload. Concurrent memory upgrade or replacement: Memory can be upgraded concurrently using LICCC if physical memory is available on the books. If the physical memory cards have to be changed in a multibook configuration, thereby requiring the book to be removed, the enhanced book availability function can be useful. It requires the availability of additional resources on other books or reducing the need for resources during this action. To help ensure that the appropriate level of memory is available in a multiple-book configuration, consider the selection of the flexible memory option for providing additional resources to exploit EBA when repairing a book or memory on a book, or when upgrading memory where larger memory cards might be required. Memory can be upgraded concurrently by using LICCC if physical memory is available as previously explained. The plan-ahead memory function available with the zEC12 provides the ability to plan for nondisruptive memory upgrades by having the system pre-plugged based on a target configuration. Pre-plugged memory is enabled by the customer, by placing an order through LICCC. Enhanced driver maintenance (EDM): One of the greatest contributors to downtime during planned outages is Licensed Internal Code driver updates performed in support of new features and functions. The zEC12 is designed to support the concurrent activation of a selected new driver level. Concurrent fanout addition or replacement: A PCIe, Host Channel Adapter (HCA), or Memory Bus Adapter (MBA) fanout card provides the path for data between memory and I/O using InfiniBand (IFB) cables or PCIe cables. With the zEC12, a hot-pluggable and concurrently upgradable fanout card is available. Up to eight fanout cards are available per book for a total of up to 32 fanout cards when four books are installed. In the event of an outage, a fanout card, used for I/O, can be concurrently repaired while redundant I/O interconnect ensures that no I/O connectivity is lost. Redundant I/O interconnect: Redundant I/O interconnect helps maintain critical connections to devices. The zEC12 allows a single book, in a multibook system, to be concurrently removed and reinstalled during an upgrade or repair, continuing to provide connectivity to the system I/O resources using a second path from a different book. Dynamic oscillator switch-over: The zEC12 has two oscillator cards, a primary and a backup. In the event of a primary card failure, the backup card is designed to transparently detect the failure, switch-over, and provide the clock signal to the system. Cooling improvements: The zEC12 comes with new designed radiator cooling system to replace the modular refrigeration unit (MRU). The radiator cooling system can support all four books simultaneously with redundant design consisting of two pumps and two blowers. One active pump and blower can support the entire system load. Replacement of pump or blower is concurrent with no performance impact. Water cooling system is also an option in zEC12, with water cooling unit (WCU) technology. Two redundant WCUs are running 356
8049ch10.fm
with two independent chilled water feeds. Similarly to the radiator cooling system, one WCU and one water feed can support the entire system load. Both radiator and water cooling systems are backed-up by an air cooling system in the rare event of a cooling system problem.
10.2 zEC12 RAS functions

Hardware RAS function improvements focus on addressing all sources of outages. Sources of outages have three classifications: Unscheduled Scheduled This outage occurs because of an unrecoverable malfunction in a hardware component of the system. This outage is caused by changes or updates that have to be done to the system in a timely fashion. A scheduled outage can be caused by a disruptive patch that has to be installed, or other changes that have to be done to the system. This outage is caused by changes or updates that have to be done to the system. A planned outage can be caused by a capacity upgrade or a driver upgrade. A planned outage is usually requested by the customer and often requires pre-planning. The zEC12 design phase focused on enhancing pre-planning effort to simplify or eliminate planned outages.
Planned
The difference between scheduled outages and planned outages is, perhaps, not very obvious. The general consensus is that scheduled outages are considered to take place somewhere in the near future. The near future is considered to be an approximate 2-week time frame. Planned outages are considered outages planned well in advance and go beyond this approximate 2-week time frame. In this chapter we will not make a difference between scheduled and planned outages. Preventing unscheduled, scheduled, and planned outages has been addressed by the IBM System z system design for many years. The zEC12 introduces a fixed size HSA of 32 GB. This helps eliminate pre-planning requirements for HSA and provides flexibility to dynamically update the configuration. Performing the following tasks dynamically is possible1: Add a logical partition Add a logical channel subsystem (LCSS) Add a subchannel set Add a logical CP to a logical partition Add a cryptographic coprocessor Remove a cryptographic coprocessor Enable I/O connections Swap processor types Add memory Add a physical processor In addition, by addressing the elimination of planned outages, the following tasks are also possible: Concurrent driver upgrades Concurrent and flexible customer-initiated upgrades
1
Some pre-planning considerations may exist, see Chapter 9, System upgrades on page 311 for more information
Chapter 10. RAS
357
8049ch10.fm
For a description of the flexible customer-initiated upgrades see 9.2.2, Customer Initiated Upgrade facility on page 319.
10.2.1 Scheduled outages

Concurrent hardware upgrades, concurrent parts replacement, concurrent driver upgrades, and concurrent firmware fixes, available with the zEC12, all address elimination of scheduled outages. Furthermore, the following indicators and functions that address scheduled outages are included: Double memory data bus lane sparing: This feature reduces the number of repair actions for memory. Single memory clock sparing Double DRAM chipkill tolerance Field repair of the cache fabric bus Power distribution N+2 design: This feature is using Voltage Transformation Module (VTMs) in a highly redundant N+2 configuration. Redundant humidity sensors Redundant altimeter sensors Unified support for the zBX: The zBX will be supported as any other feature on the zEC12. Dual in-line memory module (DIMM) field replaceable unit (FRU) indicators: These indicators imply that a memory module is not error free and might fail sometime in the future. This gives IBM a warning, and the possibility and time to concurrently repair the storage module if the zEC12 is a multibook machine. To do this, first fence-off the book, then remove the book, replace the failing storage module, and then add the book. The flexible memory option might be necessary to maintain sufficient capacity while repairing the storage module. Single processor core checkstop and sparing: This indicator implies that a processor core has malfunctioned and has been spared. IBM has to consider what to do and also take into account the history of the system by asking the question: Has this type of incident happened previously on this system? Point-to-point fabric for the Symmetric MultiProcessing (SMP): Having fewer components that can fail is an advantage. In a two-book or three-book machine, the need to complete the ring connection between all of the books has been removed by having point-to-point connections instead. A book can always be added concurrently. Hot swap InfiniBand (IFB) hub cards: When properly configured for redundancy, hot swapping (replacing) the IFB (HCA2-O (12xIFB) or HCA3-O (12xIFB)) hub cards is possible, thereby avoiding any kind of interruption when the need for replacing these types of cards occurs. Redundant 1 Gbps Ethernet service network with VLAN: The service network in the machine gives the machine code the capability to monitor each single internal function in the machine. This helps to identify problems, maintain the redundancy, and provides assistance in concurrently replacing a part. Through the implementation of the VLAN to the redundant internal Ethernet service network, these 358
8049ch10.fm
advantages are improving even more, as it makes the service network itself easier to handle and more flexible. The PCIe I/O drawer is available for thezEC12. It can be installed concurrently and I/O cards can be added to the PCIe drawers concurrently.
10.2.2 Unscheduled outages

An unscheduled outage occurs because of an unrecoverable malfunction in a hardware component of the system. The following improvements can minimize unscheduled outages: Continued focus on firmware quality: For Licensed Internal Code and hardware design, failures are eliminated through rigorous design rules, design walk-through, peer reviews, element, subsystem and system simulation, and extensive engineering and manufacturing testing. Memory subsystem improvements: z196 introduced the Redundant Array of Independent Memory (RAIM) on System z systems, a concept similar to the known disk industry Redundant Array of Independent Disks (RAID). RAIM design detects and recovers from DRAM, socket, memory channel, or DIMM failures.The RAIM design requires the addition of one memory channel that is dedicated for RAS. The parity of the four data DIMMs are stored in the DIMMs attached to a fifth memory channel. Any failure in a memory component can be detected and corrected dynamically. ThezEC12 inherited this memory architecture. This design takes the RAS of the memory subsystem to another level, making it essentially a fully fault tolerant N+1 design. The memory system on thezEC12 is implemented with an enhanced version of the Reed-Solomon ECC code known as 90B/64B, as well as protection against memory channel and DIMM failures. A very precise marking of faulty chips help assure timely DRAM replacements. The design of thezEC12 further improved this chipmarking technology. The key cache on thezEC12 memory is completely mirrored. For a full description of the memory system on thezEC12, see 2.5, Memory on page 48 Improved thermal, altitude and condensation management Soft-switch firmware: zEC12 is equipped with the capabilities of soft-switching firmware. Enhanced logic in this function ensures that every affected circuit is powered off during soft-switching of firmware components. For example, if you must upgrade the microcode of a FICON feature, enhancements have been implemented to avoid any unwanted side-effects detected on previous systems. STP recovery enhancement: When HCA3-O (12xIFB) or HCA3-O LR (1xIFB) coupling links are used, an unambiguous going away signal will be sent when the system on which the HCA3 is running, is about to enter a failed (check stopped) state. If the going away signal is sent by the Current Time Server (CTS) in an STP-only Coordinated Timing Network (CTN), the receiving end (the Backup Time Server (BTS)) can safely take over as the CTS. In this way, the BTS does not have to rely on the Offline Signal (OLS) in a two-server CTN, or on the Arbiter in a CTN with three or more servers.
Chapter 10. RAS
359
8049ch10.fm
10.3 zEC12 Enhanced book availability

Enhanced Book Availability (EBA) is a procedure under which a book in a multi-book machine can be removed and reinstalled during an upgrade or repair action with no impact on the executing workload. With the EBA procedure, and with careful planning to ensure that all the resources are still available to run critical applications in a (n-1) book configuration, you are able to avoid planned outages. Consider, also, the selection of the flexible memory option to provide additional memory resources when replacing a book. To minimize affecting current workloads, ensure that there are sufficient inactive physical resources on the remaining books to complete a book removal. Also consider non-critical system images, such as test or development logical partitions. After these non-critical logical partitions have been stopped and their resources are freed, you may find sufficient inactive resources to contain critical workloads while completing a book replacement.
10.3.1 EBA planning considerations

To take advantage of the enhanced book availability function, configure enough physical memory and engines so that the loss of a single book does not result in any degradation to critical workloads during the following occurrences: A degraded restart in the rare event of a book failure A book replacement for repair or physical memory upgrade The following configurations especially enable exploitation of the enhanced book availability function. These zEC12 models should have enough spare capacity so that they can accommodate for the resources of the fenced book. This imposes limits on the number of the customer-owned PUs that can be activated when one book within a model is fenced: A maximum of 21 customer PUs are configured on the H43. A maximum of 44 customer PUs are configured on the H66. A maximum of 66 customer PUs are configured on the H89. A maximum of 75 customer PUs are configured on the HA1. No special feature codes are required for PU and model configuration. For all zEC12 models, the number of SAPs in a book are the same. In every book there are 4 SAPs. Flexible memory option, which delivers physical memory so that 100% of the purchased memory increment can be activated even when one book is fenced. The main point here is that the system configuration needs to have sufficient dormant resources on the remaining books in the system for the evacuation of the book that is to be replaced or upgraded. Dormant resources include the following possibilities: Unused PUs or memory are not enabled by LICCC Inactive resources that are enabled by LICCC (memory that is not being used by any activated logical partitions) Memory purchased with the flexible memory option Additional books
360
8049ch10.fm
The I/O connectivity must also support book removal. The majority of the paths to the I/O have redundant I/O interconnect support in the I/O infrastructure (drawers and cages) that enable connections through multiple fanout cards. If sufficient resources are not present on the remaining books, certain non-critical logical partitions might have to be deactivated, and one or more CPs, specialty engines, or storage might have to be configured offline to reach the required level of available resources. Planning that addresses these possibilities can help reduce operational errors. We suggest you do this planning well in advance. EBA function: Single-book systems cannot make use of the EBA procedure. Include the planning as part of the initial installation and any follow-on upgrade that modifies the operating environment. A customer can use the Resource Link machine profile report to determine the number of books, active PUs, memory configuration, and the channel layout. If the zEC12 is installed, you can click the Prepare for Enhanced Book Availability option in the Perform Model Conversion panel of the EBA process on the HMC. This task helps you determine the resources required to support the removal of a book with acceptable degradation to the operating system images. The EBA process determines which resources, including memory, PUs, and I/O paths will have to be freed to allow for the removal of a given book. You can run this preparation on each book to determine which resource changes are necessary; use the results as input to the planning stage to help identify critical resources. With this planning information, you can examine the logical partition configuration and workload priorities to determine how resources might be reduced and allow for the book to be removed. In planning, include the following tasks: Review of the zEC12 configuration to determine the following values: Number of books installed and the number of PUs enabled. Note the following information: Use the Resource Link machine profile or the HMC to determine the model, number and types of PUs (CPs, IFL, ICF, zAAP, and zIIP). Determine the amount of memory, both physically installed and LICCC-enabled. Work with your IBM service personnel to determine the memory card size in each book. The memory card sizes and the number of cards installed for each installed book can be viewed from the SE under the CPC configuration task list, using the view hardware configuration option.
Channel layouts, HCA to channel connections: Use the Resource Link machine profile to review channel configuration including the HCA paths. This is a normal part of the I/O connectivity planning. The alternate paths have to be separated as far into the system as possible. Review the system image configurations to determine the resources for each. Determine the importance and relative priority of each logical partition. Identify the logical partition or workloads and the actions to be taken: Deactivate the entire logical partition. Configure PUs.
Chapter 10. RAS
361
8049ch10.fm
Reconfigure memory, which might require the use of Reconfigurable Storage Units (RSU) value. Vary off of the channels. Review the channel layout and determine whether any changes are necessary to address single paths. Develop the plan to address the requirements. When performing the review, document the resources that can be made available if the EBA is to be used. The resources on the books are allocated during a power-on reset (POR) of the system and can change during a POR. Perform a review when changes are made to the zEC12, such as adding books, CPs, memory, or channels, or when workloads are added or removed, or if the HiperDispatch feature has been enabled and disabled since the last time a POR was done.
10.3.2 Enhanced book availability processing

To use the EBA, first ensure that the following conditions are satisfied: Free the used processors (PUs) on the book that will be removed. Free the used memory on the book. For all I/O domains connected to this book, ensure that alternate paths exist, otherwise place the I/O paths offline. For the EBA process, this is the preparation phase, and is started from the SE, either directly or on the HMC by using the single object operation option in the Perform Model Conversion panel from the CPC configuration task list. See Figure 10-1.
Processor availability
Processor resource availability for reallocation or deactivation is affected by the type and quantity of resources in use, as follows: Total number of PUs that are enabled through LICCC PU definitions in the profiles that can be: Dedicated and dedicated reserved Shared Active logical partitions with dedicated resources at the time of book repair or replacement To maximize the PU availability option, ensure that there are sufficient inactive physical resources on the remaining books to complete a book removal.
Memory availability
Memory resource availability for reallocation or deactivation depends on: Physically installed memory Image profile memory allocations Amount of memory enabled through LICCC Flexible memory option See 2.7.2, Enhanced book availability on page 60.
Fanout card to I/O connectivity requirements

The optimum approach is to maintain maximum I/O connectivity during book removal. The redundant I/O interconnect (RII) function provides for redundant HCA connectivity to all installed I/O domains in the PCIe I/O drawers, I/O cage and I/O drawers.
362
8049ch10.fm
Preparing for enhanced book availability

The Prepare Concurrent Book replacement option validates that enough dormant resources exist for this operation. If enough resources are not available on the remaining books to complete the EBA process, the process identifies the resources and guides you through a series of steps to select and free up resources. The preparation process does not complete until all memory and I/O conditions have been successfully resolved. Preparation: The preparation step does not reallocate any resources. It is only used to record customer choices and to produce a configuration file on the SE that will be used to perform the concurrent book replacement operation. The preparation step can be done in advance. However, if any changes to the configuration occur between the time the preparation is done and when the book is physically removed, you must rerun the preparation phase. The process can be run multiple times, because it does not move any resources. To view results of the last preparation operation, select Display Previous Prepare Enhanced Book Availability Results from the Perform Model Conversion panel (in SE). The preparation step can be run a few times without actually performing a book replacement. It enables you to dynamically adjust the operational configuration for book repair or replacement prior to Customer Engineer (CE) activity. Figure 10-1 shows the Perform Model Conversion panel where you select the Prepare for Enhanced Book Availability option .
Figure 10-1 Perform Model Conversion; select Prepare for Enhanced Book Availability
After you select Prepare for Enhanced Book Availability, the Enhanced Book Availability panel opens. Select the book that is to be repaired or upgraded, then select OK. See Figure 10-2. Only one target book can be selected at a time.
Chapter 10. RAS
363
8049ch10.fm
Figure 10-2 Enhanced Book Availability, selecting the target book
The system verifies the resources required for the removal, determines the required actions, and presents the results for review. Depending on the configuration, the task can take from a few seconds to several minutes. The preparation step determines the readiness of the system for the removal of the targeted book. The configured processors and the memory that is in the selected book are evaluated against unused resources available across the remaining books. The system also analyzes I/O connections associated with the removal of the targeted book for any single path I/O connectivity. If not enough resources are available, the system identifies the conflicts so you can take action to free other resources. Three states can result from the preparation step: The system is ready to perform the enhanced book availability for the targeted book with the original configuration. The system is not ready to perform the enhanced book availability because of conditions indicated from the preparation step. The system is ready to perform the enhanced book availability for the targeted book. However, to continue with the process, processors are reassigned from the original configuration. Review the results of this reassignment relative to your operation and business requirements. The reassignments can be changed on the final window that is presented. However, before making changes or approving reassignments, ensure that the changes have been reviewed and approved by the correct level of support, based on your organizations business requirements.
Preparation tabs
The results of the preparation are presented for review in a tabbed format. Each tab indicates conditions that prevent the EBA option from being performed. Tabs are for processors, memory, and various single path I/O conditions. See Figure 10-3 on page 365. Possible tab selections are: Processors Memory Single I/O Single Domain I/O Single Alternate Path I/O
364
8049ch10.fm
Only the tabs that have conditions that prevent the book from being removed are displayed. Each tab indicates what the specific conditions are and possible options to correct the conditions.
Example panels from the preparation phase

The figures in this section are examples of panels that are displayed, during the preparation phase, when a condition requires further actions to prepare the system for the book removal. Figure 10-3 on page 365 shows the Single I/O tab. The preparation has identified single I/O paths associated with the removal the selected book. The paths have to be placed offline to perform the book removal. After addressing the condition, rerun the preparation step to ensure that all the required conditions have been met.
Figure 10-3 Prepare for EBA: Single I/O conditions
Preparing the system to perform enhanced book availability

During the preparation, the system determines the CP configuration that is required to remove the book. Figure 10-4 shows the results and provides the option to change the assignment on non-dedicated processors.
Chapter 10. RAS
365
8049ch10.fm
Figure 10-4 Reassign Non-dedicated Processors results
Important: Consider the results of these changes relative to the operational environment. Understand the potential impact of making such operational changes. Changes to the PU assignment, although technically correct, can result in constraints for critical system images. In certain cases, the solution might be to defer the reassignments to another time that might have less impact on the production system images. After reviewing the reassignment results, and making adjustments if necessary, click OK. The final results of the reassignment, which include changes made as a result of the review, are displayed. See Figure 10-5. These results will be the assignments when the book removal phase of the EBA is completed.
Figure 10-5 Reassign Non-Dedicated Processors, message ACT37294
Summary of the book removal process steps

This section steps through the process of a concurrent book replacement. To remove a book, the following resources must be moved to the remaining active books: PUs: Enough PUs must be available on the remaining active books, including all types of characterizable PUs (CPs, IFLs, ICFs, zAAPs, zIIPs, and SAPs). Memory: Enough installed memory must be available on the remaining active books.
366
8049ch10.fm
I/O connectivity: Alternate paths to other books must be available on the remaining active books or the I/O path must be taken offline. By understanding both the system configuration and the LPAR allocation for memory, PUs, and I/O, you can make the best decision about how to free necessary resources and allow for book removal. To concurrently replace a book, follow these steps: 1. Run the preparation task to determine the necessary resources. 2. Review the results. 3. Determine the actions to perform in order to meet the required conditions for EBA. 4. When you are ready for the book removal, free the resources that are indicated in the preparation steps. 5. Rerun the step in Figure 10-1 on page 363 (Prepare for Enhanced Book Availability task) to ensure that the required conditions are all satisfied. 6. Upon successful completion (see Figure 10-6), the system is ready for the removal of the book.
Figure 10-6 Preparation completed successfully, message ACT37152
The preparation process can be run multiple times to ensure that all conditions have been met. It does not reallocate any resources. All it does is to produce a report. The resources are not reallocated until the Perform Book Removal process is invoked.
Rules during EBA

Processor, memory, and single I/O rules during EBA are as follows: Processor rules: All processors in any remaining books are available to be used during EBA. This includes the two spare PUs or any available PU that is non-LICCC. The EBA process also allows conversion of one PU type to another PU type. One example is converting a zAAP to a CP for the duration of the EBA function. The preparation for concurrent book replacement task indicates whether any SAPs have to be moved to the remaining books. Memory rules: All physical memory installed in the system, including flexible memory, is available for the duration of the EBA function. Any physical installed memory, whether purchased or not, is available to be used by the EBA function. Single I/O rules: Alternate paths to other books must be available or the I/O path must be taken offline. Review the results. The result of the preparation task is a list of resources that need to be made available before the book replacement can take place.
Chapter 10. RAS
367
8049ch10.fm
Free any resources

At this stage, create a plan to free up these resources. See the following list of resources and actions that are necessary to free them: To free any PUs: Vary the CPs offline, reducing the number of CP in the shared CP pool. Deactivate logical partitions. To free memory: Deactivate a logical partition. Vary offline a portion of the reserved (online) memory. For example, in z/OS issue the command: CONFIG_STOR(E=1),<OFFLINE/ONLINE> This command enables a storage element to be taken offline. Note that the size of the storage element depends on the RSU value. In z/OS, the following command enables you to configure offline smaller amounts of storage than what has been set for the storage element: CONFIG_STOR(nnM),<OFFLINE/ONLINE> A combination of both logical partition deactivation and varying memory offline. Reserved storage: If you plan to use the EBA function with z/OS logical partitions, it is best to set up reserved storage and set an RSU value. Use the RSU value to specify the number of storage units that are to be kept free of long-term fixed storage allocations, allowing for storage elements to be varied offline.
10.4 zEC12 Enhanced driver maintenance

Enhanced Driver Maintenance (EDM) is one more step towards reducing both the necessity and the eventual duration of a scheduled outage. One of the contributors to planned outages is Licensed Internal Code (LIC) Driver updates that are performed in support of new features and functions. When properly configured, the zEC12 supports concurrently activating a selected new LIC Driver level. Concurrent activation of the selected new LIC Driver level is supported only at specifically released sync points. Concurrently activating a selected new LIC Driver level anywhere in the maintenance stream is not possible. There are certain LIC updates where a concurrent update/upgrade might not be possible. Consider the following key points of EDM: The HMC can query whether a system is ready for a concurrent driver upgrade. Previous firmware updates, which require an initial machine load (IML) of zEC12 to be activated, can block the ability to perform a concurrent driver upgrade. An icon on the Support Element (SE) allows you or your IBM support personnel to define the concurrent driver upgrade sync point to be used for an EDM. The ability to concurrently install and activate a new Driver can eliminate or reduce a planned outage. The zEC12 introduces Concurrent Driver Upgrade (CDU) cloning support to other CPC for CDU preload and activate.
368
8049ch10.fm
Concurrent crossover from Driver level N to Driver level N+1, to Driver level N+2 must be done serially; no composite moves are allowed. Disruptive upgrades are permitted at any time and allow for a composite upgrade (Driver N to Driver N+2). Concurrent back off to the previous Driver level is not possible. The driver level must move forward to driver level N+1 after EDM is initiated. Catastrophic errors during an update can result in a scheduled outage to recover. The EDM function does not completely eliminate the need for planned outages for driver-level upgrades. Upgrades might require a system level or a functional element scheduled outage to activate the new LIC. The following circumstances require a scheduled outage: Specific complex code changes might dictate a disruptive driver upgrade. You are alerted in advance so you can plan for the following changes: Design data or hardware initialization data fixes CFCC release level change OSA CHPIDs code changes may require CHPID Vary OFF/ON in order to activate new code.
10.5 RAS capability for the HMC and SE

Backup from HMC and SE
On an scheduled basis the HMC and SE harddisks are backed-up on the HMC backup USB media.
Remote Support Facility (RSF)

The HMC Remote Support Facility (RSF) provides the important communication to a centralized IBM support network for hardware problem reporting and service; see 12.3, Remote Support Facility on page 405
Microcode Change Level (MCL)

Regular installation of MCLs is key for RAS, optimal performance, and new functions. We recommend to install MCLs quarterly at minimum. Hiper MCLs should be reviewed continuously and customers should decide whether to wait for the next scheduled apply session, or schedule one earlier if their risk assessment of hiper MCLs warrants. For further information see 12.5.4, HMC and SE microcode on page 407.
SE
The zEC12 is provided with two Laptops inside the Z Frame. One is always the primary SE and the other is the alternate SE. The primary SE is the active one, while the alternate acts as the backup. Once per day, information is mirrored. For more information, see 12.1, Introduction to HMC and SE on page 398.
HMC in an ensemble
The serviceability function for the components of an ensemble is delivered through the traditional HMC/SE constructs as for earlier System z systems. From a serviceability point of view all the components of the ensemble, including the zBX, are treated as zEC12 features, similar to the treatment of I/O cards and other traditional zEC12 features.
Chapter 10. RAS
369
8049ch10.fm
The zBX receives all of its serviceability and problem management through the HMC/SE infrastructure, and all service reporting, including call home functions are delivered in a similar fashion. The primary HMC for the ensemble is where portions of the Unified Resource Manager routines execute. The Unified Resource Manager is an active part of the ensemble and zEC12 infrastructure. Thus, the HMC is in a stateful state that needs high availability features to assure survival of the system in case of failure. Each ensemble must therefore be equipped with two HMC workstations: a primary and an alternate. While the primary HMC can do all HMC activities (including Unified Resource Manager activities), the alternate can only be the backup and cannot be used for tasks or activities. Failover: The primary HMC and its alternate must be connected to the same LAN segment to allow the alternate HMC to take over the IP address of the primary HMC during failover processing. For more information, see 12.6, HMC in an ensemble on page 422.
10.6 RAS capability for zBX

The zBX Model 003 is based on the BladeCenter and blade hardware offerings that contain IBM certified components. zBX Model 003 BladeCenter and blade RAS features have been considerably extended for IBM System z . This includes: Hardware redundancy at various levels: Redundant power infrastructure Redundant power and switch units in the BladeCenter chassis Redundant cabling for management of zBX and data connections Concurrent to system operations: Install additional blades Hardware repair Firmware fixes and driver upgrades Automated call home for hardware/firmware problems The zBX offering provides extended service capabilities with the zEC12 hardware management structure. The HMC/SE functions of the zEC12 system provide management and control functions for the zBX solution. As mentioned, the zBX has two pairs Top of Rack Switches (TORs). These switches provide N + 1 connectivity for the private networks between the zEC12 system and the zBX for monitoring, controlling, and managing the zBX components. Not only hardware and firmware provide RAS capabilities. Also the operating system can contribute significantly in order to take RAS to a higher level. PowerHA SystemMirror for AIX (PowerHA) supports the zBX PS701 blades2. This enables setting up a PowerHA environment in the zEC12 controlled zBX. The following table provides more detail about PowerHA and the required AIX3 levels needed for a PowerHA environment on zBX.
2 3
PS701 8406-71Y blades AIX 6.1 TL06 SP3 preferred with RSCT 3.1.0.4 (packaged in CSM PTF 1.7.1.10 installed w/ AIX 6.1.6.3) is the preferred baseline for zBX Virtual Servers running AIX.
370
8049ch10.fm
Table 10-1 PowerHA and required AIX levels IBM zBX model 003 PowerHA V5.5 AIX V5.3 AIX V6.2 AIX V7.1
AIX V5.3 TL12 RSCT 2.4.13.0
AIX V6.1 TL05 RSCT 2.5.5.0
PowerHA V5.5 SP8 AIX V7.1 RSCT V3.1.0.3 PowerHA V6.1 SP3 AIX V7.1 RSCT V3.1.0.3 AIX V7.1 RSCT V3.1.0.3
PowerHA V6.1
AIX V5.3 TL12 RSCT 2.4.13.0
AIX V6.1 TL05 RSCT 2.5.5.0
PowerHA V7.1
Not Supported
AIX V6.1 TL06 RSCT V3.1.0.3
zEnterprise BladeCenter Extension (zBX) Model 003 also introduces a new version for the Advanced Management Module (AMM) and major firmware changes compared to the zBX Model 002. This takes the RAS concept of the zBX Model 003 to higher, unprecedented levels.
10.7 Considerations for PowerHA in zBX environment

An application running on AIX can be provided with high availability by the use of the PowerHA System Mirror for AIX (formerly known as HACMP4). PowerHA is easy to configure (menu driven) and provides high availability for applications running on AIX. PowerHA helps defining and managing resources (required by applications) running on AIX, providing service/application continuity through platform resources and application monitoring, and automated actions (start/manage/monitor/restart/move/stop). Note: Resource movement and application restart on the second server is known as FAILOVER. Automating the failover process speeds up recovery and allows for unattended operations thus providing improved application availability. In an ideal situation, an application should be available 24 x 7. Application availability can be measured as the amount of time the service is actually available divided by the amount of time in a year, in percentage. A PowerHA configuration (also known as a cluster) consists of two or more servers5 (up to 32) that have their resources managed by PowerHA cluster services to provide automated service recovery for the applications managed. Servers can have physical or virtual I/O resources, or a combination of both. PowerHA performs the following functions at cluster level: Manage/monitor OS and HW resources Manage/monitor application processes Manage/monitor network resources (service IP addresses) Automate application control (start/stop/restart/move)
4 5
High Availability Cluster Multi-Processing Servers can be also virtual servers; one server = one instance of the AIX Operating System
Chapter 10. RAS
371
8049ch10.fm
The virtual servers defined and managed in zBX use only virtual I/O resources. PowerHA can manage both physical and virtual I/O resources (virtual storage and virtual network interface cards). PowerHA can be configured to perform automated service recovery for the applications running in virtual servers deployed in zBX. PowerHA automates application failover from one virtual server in a System p blade to another virtual server in a different System p blade with similar configuration. Failover protects service (masks service interruption) in case of unplanned or planned (scheduled) service interruption. During failover, clients might experience a short service unavailability, while resources are configured by PowerHA on the new virtual server. Power HA configuration for zBX environment is similar to standard Power environments, with the particularity that it uses only virtual I/O resources. Currently, PowerHA for zBX support is limited to failover inside the same zBX. PowerHA configuration must include the following tasks: Network planning (VLAN and IP configuration definition and for server connectivity). Storage planning (shared storage must be accessible to all blades that provide resources for a PowerHA cluster). Application planning (start/stop/monitoring scripts and OS, CPU and memory resources). PowerHA SW installation and cluster configuration. Application integration (integrating storage, networking, and application scripts). PowerHA cluster testing and documentation. A typical PowerHA cluster is shown in Figure 10-7 on page 373
372
8049ch10.fm
Figure 10-7 Typical PowerHA cluster diagram
For more information about IBM PowerHA System Mirror for AIX, see this website: http://www-03.ibm.com/systems/power/software/availability/aix/index.html
10.8 IBM System z Advanced Workload Analysis Reporter (IBM zAware)

IBM zAware provides a smart solution for detecting and diagnosing anomalies in z/OS systems by analyzing software logs and highlighting abnormal events. It represents a first in a new generation of smart monitoring products with pattern based message analysis. IBM zAware runs as firmware virtual appliance in a zEC12 LPAR. It consists of an integrated set of analytic applications and creates a model of normal system behavior based on prior system data, and uses pattern recognition techniques to identify unexpected messages in current data from the z/OS systems that it is monitoring. This analysis of events provides nearly real-time detection of anomalies that can then be easily viewed through a graphical user interface (GUI). Statement of Direction: IBM plans to provide new capability within the Tivoli Integrated Service Management family of products designed to leverage analytics information from IBM zAware, and to provide alert and event notification. IBM zAware improves overall RAS capability of zEC12 by helping to: Identify when and where to look for a problem
Chapter 10. RAS
373
8049ch10.fm
Drill down to identify the cause of the problem Improve problem determination in near real time Reduce problem determination efforts significantly For detailed information on IBM zAware, see Appendix A, IBM zAware on page 429.
10.9 RAS capability for Flash Express

Flash Express cards come in pairs for availability and reside exclusively in PCIe I/O drawers. Similar to other PCIe I/O cards, redundant PCIe paths to Flash Express cards are provided by redundant IO interconnect. Unlike other PCIe I/O cards, they can only be accessed by the host using a unique protocol. In each Flash Express card, data is stored in four solid state disks in a RAID configuration. If a solid state disk fails, data is reconstructed dynamically. The cards in a pair mirror each other over a pair of cables, in a RAID 10 configuration. If either card fails, the data is available on the other card. Card replacement is concurrent with no disruption to customer operations. The data is always stored encrypted with a volatile key and the card is only usable on the system with the key that encrypted it. For key management, both the Primary and Alternate Support Elements (SE) have a smart card reader installed. Flash Express cards support concurrent firmware upgrades. Figure 10-8 shows the various components supporting Flash Express RAS functions
Figure 10-8 Flash Express RAS components
374
8049ch11.fm
11
Chapter 11.
Environmental requirements
This chapter briefly describes the environmental requirements for the IBM zEnterprise EC12. It lists the dimensions, weights, power and cooling requirements as an overview of what is needed to plan for the installation of a IBM zEnterprise EC12 and zEnterprise BladeCenter Extension. There are a number of options for the physical installation of the server, including air or water cooling, installation on raised floor, or non-raised floor, I/O and power cables exiting under the raised floor, or off the top of the server frames, and the possibility of having a high-voltage DC power supply as an alternative to the usual AC power supply. For comprehensive physical planning information, see Installation Manual - Physical Planning 2827, GC28-6914. In this chapter, the following topics are discussed: zEC12 power and cooling on page 376 IBM zEnterprise EC12 physical specifications on page 381 IBM zEnterprise EC12 physical planning on page 382 zBX environmental requirements on page 385 Energy management on page 390
375
8049ch11.fm
11.1 zEC12 power and cooling

The zEC12 is always a two-frames system. The frames are shipped separately and are bolted together during the installation procedure. The zEC12 supports installation on a raised floor or non-raised floor, but zEC12 with water cooling feature must be installed on raised floor, optional water hoses arriving to the server from underneath the raised floor. Power and I/O cables also exit from the bottom of the server frames unless the Top Exit I/O Cabling feature code (FC 7942) or Top Exit Power feature code (FC 7901) are installed, so that I/O cables and power cables can exit directly from the top of the server into overhead cabling rails.
11.1.1 Power consumption

The system operates with two fully redundant power supplies. Each power supply has individual line cord or pair of line cords depending on the configuration. For redundancy, the server should have two power feeds. Each power feed is either one or two line cords. The number of line cords required depends on system configuration. Line cords attach to either 3 phase, 50/60 Hz, 200 to 480 V AC power, or 380 to 520 V DC power. There is no impact to system operation with the total loss of one power feed. For ancillary equipment such as the Hardware Management Console, its display and switch, additional single-phase outlets are required. The power requirements depend on the cooling facility installed and on number of books, as well as the number of I/O units installed. I/O power units are values for I/O cages (equals two I/O units) and I/O drawers and/or PCIe drawers (both drawer types equal one I/O unit). Heat output expressed in kBTU per hour can be derived from multiplying the table entries by a factor of 3.4. Table 11-1 lists the absolute maximum power requirements for the air cooled models, warm room (>= 28 degrees centigrade).
Table 11-1 Power requirements: Air cooled models Number of I/O units Power requirement kVA 0 1 2 3 4 5 6
H20 H43 H66 H89 / HA1
5.8 9.7 13.2 17.6
7.7 11.6 15.0 19.5
9.7 13.4 16.9 21.4
11.3 15.2 18.8 23.2
13.2 17.1 20.5 24.9
13.4 19.0 22.4 26.8
13.4 19.8 23.3 27.6
Note: Power will be lower in normal ambient temperature room and for configuration which do not have every I/O slot plugged, maximum memory and maximum configured processors. Power will also be slightly lower for DC input voltage. Actual power for any configuration, power source and room condition can be obtained using the power estimation tool (see Resource Link). Table 11-2 lists the maximum power requirements for the water cooled models.
376
8049ch11.fm
Table 11-2 Power requirements: Water cooled models Number of I/O units Power requirement kVA 0 1 2 3 4 5 6
H20 H43 H66 H89 / HA1
5.5 9.1 12.4 17.7
7.4 10.9 14.3 18.6
9.4 12.7 16.1 20.4
11.0 14.6 18.0 22.3
12.9 16.4 19.7 24.1
13.1 18.3 21.6 25.8
13.4 19.1 22.5 26.6
Table 11-3 Bulk Power Regulator (BPR) requirements for books and I/O units. Note that a second pair of line cords will be installed if the number of BPR pairs is 4 or higher. If your initial configuration needs one line cord pair, but for growth would need a second pair, you can order the Line Cord Plan Ahead feature (FC 2000), which will install four line cords at the initial configuration. Also, if Balanced Power Plan Ahead (FC 3003) is ordered, four line cords are shipped and all 12 possible BPRs will be installed. Finally, if the zEC12 is configured with the Internal Battery Feature (IBF), Balanced Power Plan Ahead will automatically supply the maximum number of batteries, six IBFs, with the system.
Table 11-3 Number of BPRs requirements Number of I/O units Number of BPRs per side 0 1 2 3 4 5 6
H20 H43 H66 H89 / HA1
1 2 3 4
1 3 3 4
1 3 4 5
2 3 4 5
3 3 4 5
3 4 4 5
3 4 4 5
Systems that specify two line cords can be brought up with one line cord and will continue to run. The larger machines, that is a minimum of 4 BPR pairs are installed, four line cords are installed. Four line cords offer power redundancy, so that when a line cord fails, the remaining cords deliver sufficient power to keep the system up and running.
11.1.2 Internal Battery Feature

The optional Internal Battery Feature (IBF) provides sustained system operations for a relatively short period of time, allowing for orderly shutdown. In addition, an external uninterrupted power supply system can be connected, allowing for longer periods of sustained operation. The IBF is capable of providing emergency power for the estimated periods of time listed in Table 11-4. The Batteries are only connected to the BPRs associated with the section one, so one pair of batteries is connected to BPR 1 and BPR 2 respectively. See Table 11-3 for the number of BPRs installed in relation to I/O units and number of books.
Table 11-4 Battery hold-up times Internal battery hold-up times in minutes Number of I/O units 0 1 2 3 4 5 6
H20
7.7
5.0
4.0
7.9
11.1
11.0
11.0
Chapter 11. Environmental requirements
377
8049ch11.fm
Internal battery hold-up times in minutes
Number of I/O units 0 1 2 3 4 5 6
H43 H66 H89 / HA1
9.7 11.1 7.6
13.6 9.3 6.8
11.0 8.0 6.1
9.1 7.1 5.0
7.9 6.4 4.4
7.0 5.2 4.0
6.7 4.9 3.8
Note: System holdup time given above assumes both sides functional and fresh batteries under normal room ambient conditions. Holdup times will be greater for configurations which do not have every I/O slot plugged, maximum memory and maximum configured processors. Holdup times for actual configurations are given in the power estimation tool (see Resource Link)
11.1.3 Emergency power-off

On the front of frame A is an emergency power off switch that, when activated, immediately disconnects utility and battery power from the server. This causes all volatile data in the server to be lost. If the server is connected to a machine rooms emergency power-off switch, and the Internal Battery Feature is installed, the batteries take over if the switch is engaged. To avoid take over, connect the machine room emergency power off switch to the server power off switch. Then, when the machine room emergency power off switch is engaged, all power will be disconnected from the line cords and the Internal Battery Features. However, all volatile data in the server will be lost.
11.1.4 Cooling requirements

The zEC12 cooling system is a combination of air cooling system and water cooling system. In normal working conditions, zEC12 MCM is cooled by water cooling system with radiator or WCU. I/O drawers, power enclosures and books are cooled by chilled air with blowers.
Air cooling system requirements

The air cooling system requires chilled air to fulfill the air cooling requirements. Normal air exhaust from front to rear of frames. The chilled air is usually provided through perforated floor panels in front of machine. Figure 11-1 on page 379 does not represent any particular server machine type, and is intended only to show hot and cold airflow and the arrangement of server aisles. Typically, the zEC12 air cooled models use chilled air, provided from under the raised floor, to cool the system. As shown below, rows of servers must face front-to-front. Chilled air is usually provided through perforated floor panels placed in rows between the fronts of servers (the cold aisles shown in the figure). Perforated tiles generally are not placed in the hot aisles. (If your particular computer room causes the temperature in the hot aisles to exceed limits of comfort for activities like system service, you may add as many perforated tiles as necessary to create a satisfactory comfort level.) Heated exhaust air exits the computer room above the computing equipment. The requirements for air cooling options are indicated in detail in Installation Manual Physical Planning 2827, GC28-6914. 378
8049ch11.fm
Figure 11-1 Hot and cold aisles
Water cooling system requirements

The water cooling system requires chilled building water to be supplied to the zEC12 water cooling units (WCUs). The zEC12 will require four connections to the facility water, two feeds and two returns. These connections are made using hoses that are fixed to the facility plumbing and are routed up through the front tailgate of the machine and terminate using quick connect couplings. Following are some general conditions for water cooled systems that your facility must meet prior to installation of the EC12: Total Hardness must not exceed 200 mg/L as calcium carbonate. pH must be between 7 and 9. Turbidity must be less than 10 NTU (Nephelometric Turbidity Unit). Bacteria must be less than 1000 CFUs (Colony Forming Unit)/ml. Water to be as free of particulate matter as feasible.
379
8049ch11.fm
Allowable system inlet water temperature range is 6-20 C (43-68 F), using standard building chilled water (BCW). A special water system is typically not required. Required flow rate to the frame is 3.7 - 79.4 lpm (1- 21 gpm), depending on inlet water temperature and the number of nodes populated in the server. Colder inlet water temperatures require less flow then warmer water temperatures. Fewer nodes require less flow then maximum populated processors. Minimum water pressure required across the IBM hose ends is 0.34 - 2.32BAR (5 - 33.7 psi), depending on the minimum flow required. Maximum water pressure supplied at the IBM hose connections to the customer water supply should not exceed 6.89 BAR (100 psi). The requirements for water cooling options are indicated in detail in Installation Manual Physical Planning 2827, GC28-6914.
Supply Hoses (can be pre-ordered as M/T 2819-W00)

For zEC12 water cooling system, we provide 4.2 m (14 ft) water hoses, one set hoses include one supply and one return for using with under-floor water supply connections. Multiple sets of hoses can be ordered based on your requirements, see Figure 11-2. The customer end of the hoses are left open, allows your ability to cut-to-length to match installation requirements. Insulation clamp is provided to secure the insulation and protective sleeving once the hose has been cut to the desired length and installed onto your facilities plumbing. Using of shut-off valves in front of the hoses is recommended. Allows for removal of hoses for a service procedure or relocation. Valves are not included in the order. Stainless steel fitting is available for ordering. Fitting is barbed on one side and has a 25.4 mm (1 in) male NPT. Not supplied with original ship group, must be ordered separately.
S upply Return Supply Facilities Valves (Optional) Return
pl t y Su p F acili
F ac il
urn ity R et
Quick Disconnect
Figure 11-2 WCU water supply connections.
380
8049ch11.fm
11.2 IBM zEnterprise EC12 physical specifications

This section describes weights and dimensions of the zEC12. zEC12 can be installed on raised floor, or non-raised floor. In the Installation Manual Physical Planning 2827, GC28-6914, weight distribution and floor loading tables are published, to be used together with the maximum frame weight, frame width, and frame depth to calculate the floor loading. Table 11-5 indicates the maximum system dimension and weights for the HA1 model. The weight ranges are based on configuration models with five PCIe I/O drawers, and with and without IBFs.
Table 11-5 System dimensions and weights << need to update >> Maximum
A and Z frames with IBF (FC 3212)
A and Z frames with IBF (3212) and Top Exit Cabling Features (FC 7942 + FC 7901)
Air cooled servers
Weight kg (lbs) Width mm (in) Depth mm (in) Height mm (in) Height Reduction mm (in) 1568 (61.7) 1806 (71.1) 2013 (79.3) 1803 (71.0)
Water cooled servers
1847 (72.7) 1806 (71.1) 2154 (84.8) 1803 (71.0)
Weight kg (lbs) Width mm (in) Depth mm (in) Height mm (in) Height Reduction mm (in) 1568 (61.7) 1908 (75.1) 2013 (79.3) 1809 (71.2) 1847 (72.7) 1908 (75.1) 2154 (84.8) 1809 (71.2)
Notes: 1. Weight includes covers. Width, depth, and height are indicated without covers. 2. Weight is based on maximum system configuration, not the addition of the maximum weight of each frame.
381
8049ch11.fm
11.3 IBM zEnterprise EC12 physical planning

This section describes various floor mounting options, as well as power and I/O cabling options.
11.3.1 Raised floor or non-raised floor

zEC12 can be installed on raised floor, or non-raised floor environment. But for water cooled models, raised floor is required. Raised floor If zEC12 server is installed on raised floor environment, either air cooled models or water cooled models are supported. Typically, I/O cables, power cables, and water hoses arriving to the server from underneath the raised floor. Anyway, for zEC12, you have an option to select top exit features to route I/O cables and/or power cables from the top frame of zEC12 server. Following options are available for zEC12: Top Exit I/O Cabling feature code (FC 7942) only, or Top Exit Power feature code (FC 7901) only, or Top Exit I/O Cabling feature code (FC 7942) and Top Exit Power feature code (FC 7901) together, or None of above. Figure 11-3 shows top exit feature options of zEC12 on raised floor environment.
On raised floor, either air cooled models or water cooled models are supported.
None T op Exit features
NO Top exit support for cooling water supply & return.

RF Tailg ate
Top Exit Power only
RF Tailg ate
RF Tailgate
RF Tailga te
Line Co rd s I/O Cables Opt Water T op Exit I/O and Power
Top Exit I/O only
RF Tail gate
RF Tail gate
RF Tailgate
RF Ta lgate i
Figure 11-3 Raised Floor Options
Note: No top exit feature support of water hoses for water cooled models, water hoses must go through underneath the raised floor. Non-raised floor
382
8049ch11.fm
If you want to install zEC12 server on non-raised floor environment, you can only select air cooled models and Non-Raised Floor Support feature code (FC 7998) is required. Top Exit I/O Cabling feature code (FC 7942) and Top Exit Power feature code (FC 7901) need to be ordered also. All cables should exit from the top frame of zEC12 server, no cables may exit at floor level. See Figure 11-4
Li ne Cords I/O C ab les Top Exit I/O and Power
Figure 11-4 Non-raised floor options
11.3.2 Top Exit Power

Top exit power is a new feature of zEC12. Top exit power feature (FC 7901) is designed to provide you with an additional option. Instead of all of your power cables exiting under the raised floor, you now have the flexibility to choose the option to route your power cables from the top of the frame to meet the requirements of your data center. Top exit power feature (FC 7901) will be shipped separately with machine and installed on site. It will be installed on the top of z frame and increase the height of frame from 7 to 12 inches based on what kind of line cords selected. Two types of line cords offered in this feature, plugged connectors and cut cords. For the cut cords, it is14 ft (4.3m) long from the exit point of the frame with an attached mount bracket that you can fix power cords on the top of frame, see Figure 11-5. For the plugged cords, the concept will be all new for zEC12. In z114, the plugged cords are 6 ft (1.8m) long from the frame exit and must be plugged within 6 to 8 in (152 to 203 mm) from the top of the frame. Since these cords have 30A fittings which are not difficult to plug, this is a reasonable solution. For zEC12, the fittings are 60A and require much more force to plug successfully. For the 60A plugs, the line cord will be a short connection from the power enclosure to the top of the frame and the plug will be rigidly fixed to the frame.
383
8049ch11.fm
The customer drop must come down to the frame to meet the system input plug. These concepts are illustrated in the Figure 11-5.
Figure 11-5 Top Exit Power
cut cords plugged connections
A frame Z frame Z frame
A frame
11.3.3 Top Exit I/O Cabling

Same as z196, zEC12 also supports the Top Exit I/O Cabling feature (FC 7942). With this feature, you can route all coupling links and all I/O cables, including 1000BASE-T Ethernet cable from I/O cages or drawers, through four additional frame extensions, out the top of the frame. As Figure 11-6 shows, the frame extensions, we call it Chimneys, are installed to each corner of the frames (A frame and Z frame) when the Top Exit I/O Cabling feature (FC 7942) is ordered. Only Coupling Link, Ethernet and Fiber Cabling may enter the machine through the Chimneys. The bottom of the Chimney is closed with welded sheet metal The Top Exit I/O Cabling feature will add 6 in. (153mm) to the width of each frame and about 95 lbs (43 Kg) to the weight of each frame. In zEC12, the Top Exit I/O Cabling feature (FC 7942) is available for both air cooled models and water cooled models.
Figure 11-6 Top Exit I/O Cabling
384
8049ch11.fm
11.3.4 Weight distribution plate

The weight distribution plate is designed to distribute the weight of a frame onto two floor panels in a raised-floor installation. As Table 11-5 shows, the weight of a frame can be substantial, causing a concentrated load on a caster or leveling foot to be half of the total frame weight. In a multiple system installation, one floor panel can have two casters from two adjacent systems on it, potentially inducing a highly concentrated load on a single floor panel. The weight distribution plate distributes the weight over two floor panels. The weight distribution kit is ordered and delivered through feature code 9970. Always consult the floor tile manufacturer to determine the load rating of the tile and pedestal structure. Additional panel support might be required to improve the structural integrity, because cable cutouts significantly reduce the floor tile rating.
11.3.5 3-in-1 bolt down kit for raised floor

A bolt-down kit for raised floor environments can be ordered for the EC12 frames. The kit provides hardware to enhance the ruggedness of the frames and to tie down the frames to a concrete floor beneath a raised floor of 228-912 mm (936 inches). The kit is offered in the following configurations: The Bolt-Down Kit for an air cooled machine (FC 8000) provides frame stabilization and bolt-down hardware for securing a frame to a concrete floor beneath the raised floor. The Bolt-Down Kit for a water cooled machine (FC 8001) provides frame stabilization and bolt-down hardware for securing a frame to a concrete floor beneath the raised floor. Each server needs two features, one for each of the frames. The kits help secure the frames and their contents from damage when exposed to shocks and vibrations such as those generated by a seismic event. The frame tie-downs are intended for securing a frame weighing less than 1632 kg (3600 lbs) per frame.
11.4 zBX environmental requirements

The following sections discuss the environmental requirements in summary for zEnterprise BladeCenter Extension (zBX). For a full description of the environmental requirements for the zBX, see zBX Installation Manual for Physical Planning 2458-003, GC27-2619.
11.4.1 zBX configurations

The zBX can have from one to four racks. The racks are shipped separately and are bolted together at installation time. Each rack can contain up to two BladeCenter chassis, and each chassis can contain up to fourteen single-wide blades. The number of blades required determines the actual components required for each configuration. The number of BladeCenters and racks are generated by the quantity of blades, see Table 11-6.
Table 11-6 zBX configurations Number of blades Number of BladeCenters Number of racks
7 14 28
1 1 2
1 1 1
385
8049ch11.fm
Number of blades
Number of BladeCenters
Number of racks
42 56 70 84 98 112
3 4 5 6 7 8
2 2 3 3 4 4
A zBX can be populated by up to 112 Power 701 blades. A maximum of 28 IBM BladeCenter HX5 blades can be installed in a zBX. For DataPower blades the maximum number is 28. Note that the DataPower blade is double-wide.
11.4.2 zBX power components

The zBX has its own power supplies and cords, independent of the zEC12 server power. Depending on the configuration of the zBX, up to 16 customer supplied power feeds might be required. A fully configured four-rack zBX has 16 power distribution units (PDUs). The zBX operates with: 50/60Hz AC power Voltage (240V) Both single-phase and three-phase wiring
PDUs and power cords

The Power Distribution Units (PDU) options available for the zBX are as follows: FC 0520 - 7176 Model 3NU with attached Line-cord (US) FC 0521 - 7176 Model 2NX (WW) The power cord options available for the zBX are as follows: FC 0531 - 4.3 meter, 60A/208V, US Line-cord, Single Phase FC 0532 - 4.3 meter, 63A/230V, non-US Line-cord, Single Phase FC 0533 - 4.3 meter, 32A/380V-415V, non-US Line-cord, Three Phase. Note that 32A WYE 380V or 415V gives you 220V or 240V line to neutral, respectively. This ensures that the BladeCenter maximum of 240V is not exceeded.
Power installation considerations

Each zBX BladeCenter operates from two fully-redundant power distribution units (PDUs) installed in the rack with the BladeCenter. These PDUs each have their own line-cords, see Table 11-7, allowing the system to survive the loss of customer power to either line-cord. If power is interrupted to one of the PDUs, the other PDU will pick up the entire load and the BladeCenter will continue to operate without interruption.
Table 11-7 Number of BladeCenter power cords Number of BladeCenters Number of power cords
1 2 3
2 4 6
386
8049ch11.fm
Number of BladeCenters
Number of power cords
4 5 6 7 8
8 10 12 14 16
For the maximum availability, the line-cords on each side of the racks should be powered from different building power distribution units. Actual power consumption is dependent on the zBX configuration in terms of the number of BladeCenters and blades installed. Input power in kVA is equal to the out power in kW. Heat output expressed in kBTU per hour is derived by multiplying the table entries by a factor of 3.4. For 3-phase installations, phase balancing is accomplished with the power cable connectors between the BladeCenters and the PDUs.
11.4.3 zBX cooling

The individual BladeCenter configuration is air cooled with two hot swap blower modules. The blower speeds vary depending on the ambient air temperature at the front of the BladeCenter unit and the temperature of internal BladeCenter components. If the ambient temperature is 25C (77F) or below, the BladeCenter unit blowers will run at their minimum rotational speed, increasing their speed as required to control internal BladeCenter temperature. If the ambient temperature is above 25C (77F), the blowers will run faster, increasing their speed as required to control internal BladeCenter unit temperature. If a blower fails, the remaining blower will run full speed and continues to cool the BladeCenter unit and blade servers.
Heat released by configurations

Table 11-8 shows the typical heat released by the different zBX solution configurations.
Table 11-8 zBX power consumption and heat output Number of blades Max utility power (kW) Heat output (kBTU/hour)
7 14 28 42 56 70 84 98
7.3 12.1 21.7 31.3 40.9 50.5 60.1 69.7
24.82 41.14 73.78 106.42 139.06 171.70 204.34 236.98
387
8049ch11.fm
Number of blades
Max utility power (kW)
Heat output (kBTU/hour)
112
79.3
269.62
Optional Rear Door Heat eXchanger - FC 0540

For data centers that have limited cooling capacity, using the Rear Door Heat eXchanger (see Figure 11-7 on page 388) is a more cost effective solution than adding another air conditioning unit. Attention: The Rear Door Heat eXchanger is not a requirement for BladeCenter cooling. It is a solution for clients that cannot upgrade a data centers air conditioning units due to space, budget, or other constraints. The Rear Door Heat eXchanger has the following features: A water cooled heat exchanger door is designed to dissipate heat generated from the back of the computer systems before it enters the room. An easy-to-mount rear door design attaches to client-supplied water, using industry standard fittings and couplings. Up to 50,000 BTUs (or approximately 15 kW) of heat can be removed from air exiting the back of a zBX rack. The IBM Rear Door Heat eXchanger details are shown in Figure 11-7.
zBX rack
Real Door Heat eXchanger
Heatrise
Cold air leaves the rack Computer Room Water Conditioner (CRWC)
Heatrise
CL
Building chilled water
Figure 11-7 Rear Door Heat eXchanger (left) and functional diagram
The IBM Rear Door Heat eXchanger also offers a convenient way to handle hazardous hot spots, which can help you lower the total energy cost of the data center.
388
8049ch11.fm
11.4.4 zBX physical specifications

The zBX solution is delivered either with one (Rack B) or four racks (Rack B, C, D, and E). Table 11-9 shows the physical dimensions of the zBX minimum and maximum solutions.
Table 11-9 Dimensions of zBX racks Racks with covers Width mm (in) Depth mm (in) Height mm (in)
B B+C B+C+D B+C+D+E
648 (25.5) 1296 (51.0) 1994 (76.5) 2592 (102)
1105 (43.5) 1105 (43.5) 1105 (43.5) 1105 (43.5)
2020 (79.5) 2020 (79.5) 2020 (79.5) 2020 (79.5)
Height Reduction FC 0570

This feature is required if it is necessary to reduce the shipping height for the zBX. Select this feature when it has been deemed necessary for delivery clearance purposes. Order it if you have doorways with openings less than 1941 mm (76.4 inches) high. It accommodates doorway openings as low as 1832 mm (72.1 inches).
zBX weight
Table 11-10 shows the maximum weights of fully populated zBX racks and BladeCenters.
Table 11-10 Weights of zBX racks Rack description Weight kg (lbs.)
B with 28 blades B + C full B + C + D full B + C + D + E full
740 (1630) 1234 (2720) 1728 (3810) 2222 (4900)
Rack weight: A fully configured Rack B is heavier than a fully configured Rack C, D, or E because Rack B has the TOR switches installed.
For a complete view of the physical requirements, see zBX Installation Manual for Physical Planning 2458-003, GC27-2619.
389
8049ch11.fm
11.5 Energy management

This section discusses the elements of energy management in areas of tooling to help you understand the requirement for power and cooling, monitoring and trending, and reducing power consumption. The energy management structure for the server is shown in Figure 11-8.
Enterprise Managem ent

Active Energy Manager IBM Director Other apps Other apps Other apps
( e .g. Ca pa c ity P rov is ioning M a a ge r) n (e .g. Ca pa c ity P rov is ionin g M a na ge r) ( e .g. Ca pa c ity P rov is ioning M a a ge r) n
Act ive Energy Manager
GUI
API
Energy M anagement HMC/SE Energy M oring Dashboard onit Hard ware Man agemen t Console
GUI
Energy Management
EC12
Supp ort Element
power code power and ambi ent temperat ure snapshots read ambient temp read V, I set V set f Power Su b-system Con l ler tro System z server
AMM
Bl adeCenter
Figure 11-8 zEC12 Energy Management
The hardware components in the zEC12 and the optional zBX are monitored and managed by the Energy Management component in the Support Element (SE) and HMC. The GUIs of the SE and HMC provide views, for instance with the System Activity Display or Monitors Dashboard. Through an SNMP API energy information is available to, for instance Active Energy Manager, a plug-in of IBM Systems Director, see 11.5.4, IBM Systems Director Active Energy Manager on page 392 for more information. When Unified Resource Manager features are installed (see 12.6.1, Unified Resource Manager on page 422), several monitoring and control functions can be used to perform Energy Management. More details are discussed in section 11.5.5, Unified Resource Manager: Energy Management on page 393. A few aids are available to plan and monitor the power consumption and heat dissipation of the zEC12. This section summarizes the tools that are available to plan and monitor the energy consumption of the zEC12. Power estimation tool Query maximum potential power System activity display and Monitors Dashboard IBM Systems Director Active Energy Manager
390
8049ch11.fm
11.5.1 Power estimation tool

The power estimation tool for System z servers is available through the IBM Resource Link website: http://www.ibm.com/servers/resourcelink The tool provides an estimate of the anticipated power consumption of a machine model given its configuration. You input the machine model, memory size, number of I/O cages, I/O drawers, and quantity of each type of I/O feature card. The tool outputs an estimate of the power requirements for that configuration. If you have a registered machine in Resourcelink you can access the Power Estimator tool through the machine information page of that particular machine. In the Tools section of Resourcelink you also can enter the Power Estimator and enter any system configuration you would like to calculate its power requirements. This tool helps with power and cooling planning for installed/planned System z servers.
11.5.2 Query maximum potential power

The maximum potential power used by the system is less then the Label Power, as depicted in atypical power usage report in Figure 11-9. The Query maximum potential power function shows what your systems maximum power usage and heat dissipation can be, so you are able to allocate the proper power and cooling resources. The output values of this function for Maximum potential power and Maximum potential heat load are displayed on the Energy Management tab of the CPC Details view of the HMC. This function enables operations personnel with no System z knowledge to query the maximum possible power draw of the system. The implementation helps avoid capping enforcement through dynamic capacity reduction. The customer controls are implemented in the HMC, the SE, and the Active Energy Manager. Use this function in conjunction with the Power Estimation tool that allows for pre-planning for power and cooling requirements. See Power estimation tool on page 391.
Label Power
Power typically allocated to a server
Allocation Model of Server

Over Allocated Power not converted into compute cycles
Max Potential Power
Wasted Power / Power budget not converted into Cooling compute cycles Allocation
Power (watts)
Proper Power/ Cooling Allocation
Power budget converted into compute cycles
Time
Trending (weeks, months)
Figure 11-9 Maximum potential power
391
8049ch11.fm
11.5.3 System Activity Display and Monitors Dashboard

The System Activity Display present you, among other information, with the current power usage:
Figure 11-10 Power usage on the System Activity Display (SAD)
The Monitors Dashboard of the HMC allows you to display power and other environmental data and also allows you to start a Dashboard Histogram Display, where you can trend a particular value of interest, such as the power consumption of a blade or the ambient temperature of the zEC12.
11.5.4 IBM Systems Director Active Energy Manager

IBM Systems Director Active Energy Manager is an energy management solution building block that returns true control of energy costs to the customer. Active Energy Manager is an industry leading cornerstone of the IBM energy management framework. Active Energy Manager Version 4.3.1 is a plug-in to IBM Systems Director Version 6.2.1 and is available for installation on Linux on System z. It can also run on Windows, Linux on IBM System x, and AIX and Linux on IBM Power Systems. For more specific information, see Implementing IBM Systems Director Active Energy Manager 4.1.1, SG24-7780, supports IBM zEnterprise EC12 and its optional attached zBX. Use Active Energy Manager to monitor the power and environmental values of resources, not only System z, also other IBM products like IBM Power Systems, IBM System x or devices and hardware acquired from another vendor. You can view historical trend data for resources, calculate energy costs and savings, view properties and settings for resources, and view active energy-related events. AEM is not directly connected to the System z servers but is attached through a LAN connection to the HMC, see Figure 11-8 on page 390 and 12.2, HMC and SE connectivity on page 401. AEM discovers the HMC managing the server using a discovery profile, specifying the HMC's IP address and the SNMP Credentials for that System z HMC. As the system is discovered, the System z servers managed by the HMC will also be discovered.
392
8049ch11.fm
Active Energy Manger is a management software tool that can provide a single view of the actual power usage across multiple platforms as opposed to the benchmarked or rated power consumption. It can effectively monitor and control power in the data center at the system, chassis, or rack level. By enabling these power management technologies, data center managers can more effectively power manage their systems while lowering the cost of computing. The following data is available through Active Energy Manager: System name, machine type, model, serial number and firmware level of System z servers and optional zBX attached to IBM zEnterprise Systems or IBM zEnterprise EC12. Ambient temperature Exhaust temperature Average power usage Peak power usage Limited status and configuration information. This information helps explain changes to the power consumption, called Events, which can be: Changes in fan speed Radiator/WCU failures Changes between power off, power on, and IML complete states Number of books and I/O cages CBU records expiration(s)
IBM Systems Director Active Energy Manager provides customers with the intelligence necessary to effectively manage power consumption in the data center. Active Energy Manager, which is an extension to IBM Director systems management software, enables you to meter actual power usage and trend data for any single physical system or group of systems. Active Energy Manager uses monitoring circuitry, developed by IBM, to help identify how much actual power is being used and the temperature of the system.
11.5.5 Unified Resource Manager: Energy Management

This section discuss the energy management capabilities of Unified Resource Manager.
Choice of suites
The energy management capabilities for Unified Resource Manager that can be used in an ensemble depends on which suite is installed in the ensemble: Manage suite (feature code 0019) Automate/Advanced Management suite (feature code 0020)
Manage suite
For energy management the manage suite focuses on the monitoring capabilities. Energy monitoring can help better understand the power and cooling demands of the zEC12 system, by providing complete monitoring and trending capabilities for the zEC12 and the zBX using one or more of the following options: Monitor dashboard Environmental Efficiency Statistics Details view
393
8049ch11.fm
Automate/Advanced Management suite

The Unified Resource Manager offers multiple energy management tasks as part of the automate/advanced management suite. These tasks allow you to actually change the systems behavior for optimized energy usage and energy saving. Power Cap Group Power Cap Power Save Group Power Save Depending on the scope that is selected inside the Unified Resource Manager GUI, different options are presented.
Set Power Cap

The Set Power Cap function can be used to limit the maximum amount of energy used by the ensemble. If enabled, it enforces power caps for the hardware by actually throttling the processors in the system. The Unified Resource Manager shows all components of an ensemble in the power cap window, as seen in Figure 11-11. Because not all components used in a specific environment might support power capping, only those marked as enabled can actually perform power capping functions. A zEC12 does not support power capping, as opposed to specific Blades, that can be power capped. When capping is enabled for a zEC12, this capping level is used as a threshold for a warning message that informs you that the CPC went above the set cap level. The lower limit of the cap level is equal to the maximum potential power value (see 11.5.2, Query maximum potential power on page 391).
Figure 11-11 Set Power Cap panel
394
8049ch11.fm
Static power saving mode

The server has a mechanism to vary frequency and voltage, originally developed to help avoid interruptions due to cooling failures. The mechanism can be used to reduce the energy consumption of the system in periods of low utilization and to partially power off systems designed mainly for disaster recovery: CBU systems. The mechanism is under full customer control, there is no autonomous function to perform changes under the covers. The customer controls are implemented in the HMC, in the SE, and in the Active Energy Manager with one power saving mode. The expectation is that the frequency reduction is 17%, the voltage reduction is 9%, and the total power savings is from 10% to 20% depending on the configuration. Figure 11-12, Set Power Saving panelshows the Set Power Saving panel.
Figure 11-12 Set Power Saving panel
More information about Energy Management with Unified Resource Manager is available in the Redbooks publication, IBM zEnterprise Unified Resource Manager, SG24-7921.
395
8049ch11.fm
396
8049ch12.fm
12
Chapter 12.
Hardware Management Console and Support Element

The Hardware Management Console (HMC) supports many functions and tasks to extend the management capabilities of zEC12. When tasks are performed on the HMC, the commands are sent to one or more Support Elements (SEs) which then issue commands to their CPCs or zEnterprise BladeCenter Extension (zBX). This chapter discusses the HMC and SE in general, and adds relevant information for HMCs that manage ensembles with the zEnterprise Unified Resource Manager. In this chapter, the following topics are discussed: Introduction to HMC and SE on page 398 Remote Support Facility on page 405 HMC and SE remote operations on page 406 HMC and SE key capabilities on page 406 HMC in an ensemble on page 422
397
8049ch12.fm
12.1 Introduction to HMC and SE

The Hardware Management Console (HMC) is a stand-alone computer that runs a set of management applications. The HMC is a closed system, which means that no other applications can be installed on it. The HMC is used to set up, manage, monitor, and operate one or more System z CPCs. It manages System z hardware, its logical partitions, and provides support applications. At least one HMC is required to operate a IBM System z . An HMC can manage multiple IBM System z CPCs and can be located at a local or a remote site. If the zEC12 is defined as a member of an ensemble, a pair of HMCs is required (a primary and an alternate). When a zEC12 is defined as a member of an ensemble, certain restrictions apply, see 12.6, HMC in an ensemble on page 422. The Support Elements (SEs) are two integrated Laptops that are supplied with the zEC12. One is always the primary SE and the other is the alternate SE. The primary SE is the active one, while the alternate act as the backup. The SEs are closed systems, just as the HMCs, and no other applications can be installed on them. When tasks are performed at the HMC, the commands are routed to the active SE of the System z system which issues commands to their CPC and controlled zBX (if any). The microcode for the System z and zBX is managed at the HMC. One HMC can control up to 100 SEs and one SE can be controlled by up to 32 HMCs. Some functions are only available on the SE. With Single Object Operations (SOO) these functions can be used from the HMC. See SE access on page 406 for further details. The HMC Remote Support Facility (RSF) provides the important communication to a centralized IBM support network for hardware problem reporting and service; see 12.3, Remote Support Facility on page 405.
SE driver support with new HMC

The driver of the HMC and SE is always equivalent to a specific HMC/SE version. For example: Driver 12 is equivalent to version 2.12.0 Driver 86 is equivalent to version 2.11.0 Driver 79 is equivalent to version 2.10.2 At the time of this writing an zEC12 is shipped with HMC version 2.12.0, which is capable of supporting different System z types. Some functions that are available on version 2.12.0 and later are only supported when connected to a zEC12. Table 12-1 on page 399 shows a summary of the SE driver/version which are supported by the new HMC version 2.12.0 (driver 12).
398
8049ch12.fm
Table 12-1 zEC12 HMC - System z support summary System z family name Machine type Minimum SE driver Minimum SE version
zEC12 z114 z196 z10 BC z10 EC z9 BC z9 EC z890 z990
2827 2818 2817 2098 2097 2096 2094 2086 2084
12 93 86 79 79 67 67 55 55
2.12.0 2.11.1 2.11.0 2.10.2 2.10.2 2.9.2 2.9.2 1.8.2 1.8.2
HMC FC 0091 changes

New build feature code (FC) 0091 is an HMC that contains 16 GB memory. Previous FC 0091 can be carry forward, but a HMC for zEC12 needs 16 GB memory. When driver 12 is ordered for an existing FC 0091 HMC, the additional needed 8 GB memory will be provided. Older HMCs than FC 0091 are not supported for zEC12 and driver 12.
12.1.1 HMC and SE enhancements and changes

The zEC12 comes with the new HMC application Version 2.12.0. We encourage you to use the Whats New Wizard to explore new features available for each release. For a complete list of HMC functions, see Hardware Management Console Operations Guide (V2.12.0), SC28-6919 and Support Element Operations Guide (V2.12.0), SC28-6920. The HMC and SE has several enhancements and changes for zEC12: There is a new section in the System Management for Flash Express. More information about Flash Express can be found in Appendix C, Flash Express on page 445.A new task panel for Flash Status and Control was added where you can manage your Flash Express as in the example Figure 12-1:
Figure 12-1 Flash Status and Controls
For the IBM zAware management, tasks and panels are implemented to configure and support it as the example shows in the following Figure 12-2 on page 400:
Chapter 12. Hardware Management Console and Support Element
399
8049ch12.fm
Figure 12-2 zAware configuration panel example
For details see Appendix A, IBM zAware on page 429 and IBM zAware Concept Guide , SG24-8070. STP NTP broadband security Authentication is added to the HMCs NTP communication with NTP time servers. Refer to HMC NTP broadband authentication support for zEC12 on page 415 for details. The environmental task has usability improvements regarding the time frame. See Environmental Efficiency Statistic Task on page 412. Crypto Function Integration in Monitors Dashboard See The Monitors Dashboard task on page 410. Removal of modem support from the HMC This change impacts customers that may have set up the modem for Remote Support Facility (RSF) or for STP NTP access. For details see 12.3, Remote Support Facility on page 405 and 12.5.8, NTP client/server support on HMC on page 414 Install and activate by MCL bundle target. Microcode installation by MCL bundle target on page 409 provides further informations. A confirmation panel before processing an Alt-Ctrl-Delete request is added. Note: If an HMC needs to be rebooted, always use the Shutdown and Restart task on the HMC to avoid any file corruption. The capability to modify the time of the SE mirror scheduled operation is added. It is now possible to allow mass delete of messages from the Operating System Messages task. The Network Settings task is updated to clearly show the ordering of the routing table entries. 400
8049ch12.fm
Remove support for the Coprocessor Group Counter Sets In zEC12 each physical processor has its own crypto coprocessor, and no longer has to share this coprocessor with another PU. The Coprocessor Group Counter Sets of the counter facilities will not be available. All of the necessary crypto counter information will be available in the crypto activity counter sets directly. The check-box selection for the Coprocessor Group Counter Sets was removed from the Image profile definition and the Change Logical Partition Security task.
12.1.2 HMC media support

The HMC provides a DVD-RAM drive and, with HMC version 2.11.0, a USB flash memory drive (UFD) was introduced as an alternative. The tasks that require access to a DVD-RAM drive now have the ability to access a UFD. There can be more than one UFD inserted into the HMC.
12.1.3 Tree Style User Interface and Classic Style User Interface
Two User Interface styles are provided with an HMC. The Tree Style User Interface (default) uses a hierarchical model popular in newer operating systems and features context-based task launching. The Classic Style User Interface uses the drag and drop interface style. Tutorials: The IBM Resource Linka provides tutorials which demonstrate how to change from the Classic to the Tree Style Interface, and will introduce you to the function of the Tree Style Interface on the HMC: https://www-304.ibm.com/servers/resourcelink/hom03010.nsf/pages/education?Op enDocument
a. Registration is required to access the IBM Resource Link
12.2 HMC and SE connectivity

The HMC has two Ethernet adapters which are supported by HMC version 2.12.0 to have up to two different Ethernet LANs. The SEs on the zEC12 are connected to the Bulk Power Hub (BPH). The HMC to BPH communication is only possible trough an Ethernet switch in between. Other System z and HMC can also be connected to the switch. To provide redundancy for the HMCs, two switches are suggested. Only the switch (and not the HMC directly) can be connected to the BPH ports J02 and J01 for the customer network 1 and 2. Table 12-1 on page 399 shows the connectivity between the HMC and the SE.
401
8049ch12.fm
HMC
HMC HMC Customer Network 1
HMC
H MC
HMC
HMC Customer Network 2 J0 1 J05 J05 J02 J01 J02
SE
SE
SE
z2120 BPH-B Rear Z-Frame A-Frame BPH -A Front
Figure 12-3 HMC to SE connectivity
Various methods are available for setting up the network. It is the customer responsibility to plan and conceive the HMC and SE connectivity. The method that should be selected depends on the customer connectivity and security requirements. Security: Configuration of network components, such as routers or firewall rules, is beyond the scope of this document. Any time networks are interconnected, security exposures can exist. Network security is a clients responsibility. The document IBM System z HMC Security provides information about HMC security. It is available on the IBM Resource Linka: https://www-304.ibm.com/servers/resourcelink/lib03011.nsf/pages/zHmcSecurity/$f ile/zHMCSecurity.pdf The Hardware Management Console Operations Guide (V2.12.0), SC28-6919 covers the possibilities to set on the HMC regarding access and security.
a. Registration is required to access the IBM Resource Link.
The HMC and SE network connectivity should be planned carefully to allow for current and future use. Many of the System z capabilities benefit from the various network connectivity options available. For example, functions available to the HMC that depend on the HMC connectivity include: LDAP support, that can be used for HMC user authentication NTP client/server support Remote Support Facility (RSF) via broadband HMC remote web browser
402
8049ch12.fm
Enablement of the SNMP and CIM APIs to support automation or management applications such as IBM System Director Active Energy Manager (AEM) The above examples are shown in Figure 12-4:
Figure 12-4 HMC connectivity examples
More information can be found in the following documentation: Hardware Management Console Operations Guide (V2.12.0), SC28-6919 11.5.4, IBM Systems Director Active Energy Manager on page 392 Installation Manual - Physical Planning 2827, GC28-6914
12.2.1 Hardware prerequisites news

The following two news regarding the HMC are important for the zEC12.
No HMC LAN switch available as a feature

It is no longer possible to order Ethernet switches required by the HMCs to connect to the zEC12. The customer must provide the Ethernet switches. Existing supported switches can be used by the customer. Typical Ethernet switch/hub characteristics: 16 auto-negotiation ports 10/100/1000 Mbps data rate Full or half duplex operation Auto-MDIX on all ports Port Status LEDs
403
8049ch12.fm
Remote Support Facility (RSF) only possible via broadband, no modem

RSF through modem is not supported on the zEC12 HMC. Broadband is needed in order to have hardware problem reporting and service. See 12.3, Remote Support Facility on page 405 for more details.
12.2.2 TCP/IP Version 6 on HMC and SE

The HMC and SE can communicate using IPv4, IPv6, or both. Assigning a static IP address to an SE is unnecessary if the SE only has to communicate with HMCs on the same subnet. An HMC and SE can use IPv6 link-local addresses to communicate with each other. IPv6 link-local addresses have the following characteristics: Every IPv6 network interface is assigned a link-local IP address. A link-local address is for use on a single link (subnet) and is never routed. Two IPv6-capable hosts on a subnet can communicate by using link-local addresses, without having any other IP addresses (v4) assigned.
12.2.3 Assigning addresses to HMC and SE

An HMC can have the following IP configurations: Statically assigned IPv4 or statically assigned IPv6 addresses. DHCP assigned IPv4 or DHCP assigned IPv6 addressees. Autoconfigured IPv6: Link-local is assigned to every network interface. Router-advertised, which is broadcast from the router, can be combined with a MAC address to create a unique address. Privacy extensions can be enabled for these addresses as a way to avoid using MAC address as part of address to ensure uniqueness. An SE can have the following IP addresses: Statically assigned IPv4 or statically assigned IPv6. Autoconfigured IPv6 as link-local or router-advertised. IP addresses on the SE cannot be dynamically assigned through DHCP to ensure repeatable address assignments. Privacy extensions are not used. The HMC uses IPv4 and IPv6 multicasting to automatically discover SEs. The HMC Network Diagnostic Information task can be used to identify the IP addresses (IPv4 and IPv6) that are being used by the HMC to communicate to the CPC SEs. IPv6 addresses are easily identified. A fully qualified IPV6 address has 16 bytes, written as eight 16-bit hexadecimal blocks separated by colons, as shown in the following example: 2001:0db8:0000:0000:0202:b3ff:fe1e:8329 Because many IPv6 addresses are not fully qualified, shorthand notation can be used. This is where the leading zeros can be omitted and a series of consecutive zeros can be replaced with a double colon. The address in the previous example can also be written as follows: 2001:db8::202:b3ff:fe1e:8329
404
8049ch12.fm
For remote operations using a web browser, if an IPv6 address is assigned to the HMC, navigate to it by specifying that address. The address must be surrounded with square brackets in the browsers address field: https://[fdab:1b89:fc07:1:201:6cff:fe72:ba7c] Using link-local addresses must be supported by browsers.
12.3 Remote Support Facility

The HMC Remote Support Facility (RSF) provides the important communication to a centralized IBM support network for hardware problem reporting and service. The types of communication provided include: Problem reporting and repair data Microcode Change Level (MCL) delivery Hardware inventory data On-demand enablement Note: RSF through modem is not supported on the zEC12 HMC. Broadband connectivity is needed in order to have hardware problem reporting and service. Future HMC hardware will not include modem hardware. Modems on installed HMC FC 0091 hardware will not work with the HMC version 2.12.0 required to support zEC12. The following security characteristics are in effect: Remote Support Facility requests are always initiated from the HMC to IBM. An inbound connection is never initiated from the IBM Service Support System. All data transferred between the HMC and the IBM Service Support System is encrypted in a high-grade Transport Layer Security (TLS)/Secure Sockets Layer (SSL) encryption. When initializing the SSL/TLS encrypted connection the HMC validates the trusted host by its digital signature issued for the IBM Service Support System. Data sent to the IBM Service Support System consists of hardware problems and configuration data. Broadband RSF connection setup: The following document, available on the IBM Resource Linka, introduces the benefits of Broadband RSF, SSL/TLS secured protocol, and has a sample configuration for the Broadband RSF connection: https://www-304.ibm.com/servers/resourcelink/lib03011.nsf/pages/zHmcBroadbandRs fOverview
IPv4 address support for connection to IBM

Only the following IPv4 addresses are supported with zEC12 to connect to the IBM support network:
129.42.26.224 :443 129.42.34.224 :443 129.42.42.224 :443
405
8049ch12.fm
12.4 HMC and SE remote operations

There are two ways to perform remote manual operations on the HMC: Using a remote HMC: A remote HMC is an physical HMC that is on a different subnet from the SE, therefore the SE can not be automatically discovered with IP multicast. A remote HMC requires TCP/IP connectivity to each SE to be managed. Therefore, any existing customer-installed firewalls between the remote HMC and its managed objects must permit communications between the HMC and SE. For service and support, the remote HMC also requires connectivity to IBM, or to another HMC with connectivity to IBM via RSF (see 12.3, Remote Support Facility on page 405). Using a web browser to connect to an HMC: The zEC12 HMC application simultaneously supports one local user and any number of remote users. The user interface in the web browser is the same as the local HMC and has the same functionality. Some functions are not available, as for example USB flash media drive (UFD) access needs physical access or you can not shutdown or restart the HMC from remote. Logon security for a web browser is provided by the local HMC user logon procedures. Certificates for secure communications are provided, and can be changed by the user. A remote browser session to the primary HMC that is managing an ensemble allows a user to perform ensemble-related actions.
SE access
It is not necessary to be physically close to an SE in order to use it. The HMC can be used to access the SE remotely via the Single Object Operations (SOO). The same interface as in the SE is provided. For details please consult the Hardware Management Console Operations Guide (V2.12.0), SC28-6919.
12.5 HMC and SE key capabilities

The HMC and SE have many capabilities. This section covers the key ones. For a complete list of capabilities refer to the following documentation: Hardware Management Console Operations Guide (V2.12.0), SC28-6919 Support Element Operations Guide (V2.12.0), SC28-6920
12.5.1 CPC management

The HMC is the primary place for central processor complex (CPC) control. For example the IOCDS contains definitions of logical partitions, channel subsystems, control units and devices and their accessibility from logical partitions. IOCDS can be created and put into production from the HMC. The zEC12 is powered on and off from the HMC. The HMC is used to initiate power-on reset (POR) of the server. During the POR, among other things, PUs are characterized and placed into their respective pools, memory is put into a single storage pool and the IOCDS is loaded and initialized into the hardware system area.
406
8049ch12.fm
The Hardware messages task displays hardware-related messages at CPC level, at logical partition level, SE level, or hardware messages related to the HMC itself.
12.5.2 LPAR management

Use the HMC to define logical partition properties, such as how many processors of each type, how many are reserved, or how much memory is assigned to it. These parameters are defined in logical partition profiles and they are stored on the SE. Because PR/SM has to manage logical partition access to processors and initial weights of each partition, weights are used to prioritize partition access to processors. A Load task on the HMC enables you to IPL an operating system. It causes a program to be read from a designated device and initiates that program. The operating system can be IPLed from disk, the HMC DVD-RAM drive, the USB flash memory drive (UFD), or an FTP server. When a logical partition is active and an operating system is running in it, you can use the HMC to dynamically change certain logical partition parameters. The HMC provides an interface to change partition weights, add logical processors to partitions, and add memory. LPAR weights can be also changed through a scheduled operation. Use the HMCs Customize Scheduled Operations task to define the weights that will be set to logical partitions at the scheduled time. Channel paths can be dynamically configured on and off, as needed for each partition, from an HMC. The Change LPAR Controls task for the zEC12 has the ability to export the Change LPAR Controls table data to a .csv formatted file. This support is available to a user when connected to the HMC remotely by a web browser. Partition capping values can be scheduled and are specified on the Change LPAR Controls scheduled operation support. Viewing of Details about an existing Change LPAR Controls schedule operation is available on the SE.
12.5.3 Operating system communication

The Operating System Messages task displays messages from a logical partition. You can also enter operating system commands and interact with the system. This task is especially valuable to enter Coupling Facility Control Code (CFCC) commands. The HMC also provides integrated 3270 and ASCII consoles so an operating system can be accessed without requiring other network or network devices (such as TCP/IP or control units).
12.5.4 HMC and SE microcode

The microcode for the HMC, SE, CPC, and zBX is included in the driver/version. The HMC provides the management of the driver upgrade (EDM, see Chapter 10.4, zEC12 Enhanced driver maintenance on page 368) and installation of latest functions and patches (MCLs).
Microcode Change Level (MCL)

Regular installation of MCLs is key for reliability, availability, and serviceability (RAS), optimal performance, and new functions. We recommend to install MCLs quarterly at minimum. Hiper
407
8049ch12.fm
MCLs should be reviewed continuously and customers should decide whether to wait for the next scheduled apply session, or schedule one earlier if their risk assessment of hiper MCLs warrants. Note: The following link in IBM Resource Linka provides access to the machine information for your System z according the scheduled sent system availability data. There you can find further information about the MCL status of your zEC12: https://www-304.ibm.com/servers/resourcelink/svc0303a.nsf/fwebsearchstart?openf orm
Microcode terms
The driver contains EC streams. Each EC stream covers the code for a specific component from the zEC12. It has a specific name and an ascending number. The EC stream name and a specific number is one Microcode Change Level (MCL). MCLs from the same EC stream must be installed in sequence. MCLs can have installation dependencies with other MCLs. Combined MCLs from one or more EC streams are in one bundle. A MCL contains one or more Microcode Fixes (MCF). Figure 12-5 shows how the driver, bundle, EC stream, MCL, and MCF interact with each other.
Figure 12-5 Microcode terms and interaction
408
8049ch12.fm
Microcode installation by MCL bundle target

A bundle is a set of MCLs grouped during testing and released as a group on the same date. The System Information panel is enhanced to show a summary bundle level for the activated level (see Figure 12-6).
Figure 12-6 System Information - bundle level
12.5.5 Monitoring
This section discusses monitoring considerations.
Monitor Task Group

The task group Monitor on the HMC and SE holds monitoring related tasks for the zEC12 (see Figure 12-7 on page 410).
409
8049ch12.fm
Figure 12-7 HMC Monitor Task Group
Customize Activity Profiles

Use Customize Activity Profiles to set profiles depending on monitoring requirements. Multiple activity profiles can be defined.
The Monitors Dashboard task

The Monitors Dashboard supersedes the System Activity Display (SAD). In zEC12 the Monitors Dashboard task in the Monitor task group provides a tree-based view of resources. Multiple graphical ways are available for displaying data, such as history charts. The task Open Activity (known as SAD) monitors processor and channel usage and includes power monitoring information, the power being consumed, and the air input temperature for the server. See Figure 12-8 on page 411) shows an example of the Monitors Dashboard.
410
8049ch12.fm
Figure 12-8 Monitors Dashboard
With zEC12 the monitor dashboard has been enhanced with an adapters table. The crypto utilization percentage is displayed on the monitors dashboard according to the PCHID number. The associated crypto number (Adjunct Processor Number) for this PCHID is also shown in the table. It provides information about utilization rate in a system wide basis, not per logical partition, as shown in Figure 12-9.
Figure 12-9 Monitors Dashboard - Crypto function integration
Also for Flash Express there is a new panel, shown in Figure 12-10 on page 412.
411
8049ch12.fm
Figure 12-10 Monitors Dashboard - Flash Express function integration
Environmental Efficiency Statistic Task

The Environmental Efficiency Statistic Task (see Figure 12-11) is part of the Monitor task group, provides historical power consumption and thermal information for the zEnterprise CPC, and is available on the HMC. The data is presented in table form, graphical (histogram) form and it can also be exported to a .csv formatted file so that it can be imported into spreadsheet. Prior to zEC12, when the data is first shown (default being one day), the chart displayed from midnight of the prior day to midnight of the current day. In zEC12 the initial chart display shows the 24 hours preceding the current time so that a full 24 hours of recent data is displayed. The panel was also enhanced with the ability to specify a Starting time.
Figure 12-11 Environmental Efficiency Statistics
12.5.6 Capacity on Demand support

All Capacity on Demand upgrades are performed from the SE Perform a model conversion task. Use the task to retrieve and activate a permanent upgrade, and to retrieve, install, 412
8049ch12.fm
activate and deactivate a temporary upgrade. The task help managing all installed or staged LICCC records by showing a list of them. It also shows a history of recorded activities. HMC for IBM zEnterprise EC12 CoD capabilities include: SNMP API support: API interfaces for granular activation and deactivation API interfaces for enhanced Capacity On Demand query information API Event notification for any Capacity On Demand change activity on the system Capacity On Demand API interfaces (such as On/Off CoD and CBU) SE panel features (accessed through HMC Single Object Operations): Panel controls for granular activation and deactivation History panel for all Capacity On Demand actions Descriptions editing of Capacity On Demand records HMC/SE version 2.12.0 provides CoD information such as: MSU and processor tokens shown on panels Last activation time shown on panels Pending resources are shown by processor type instead of just a total count Option to show details of installed and staged permanent records More details for the Attention! state on panels (by providing seven additional flags) New in SE version 2.12.0: Some panels preselected defaults are removed. Specifying each selection in the panel is required. HMC and SE are an integral part of the z/OS Capacity Provisioning environment. The Capacity Provisioning Manager (CPM) communicates with the HMC through System z APIs and enters CoD requests. For this reason, SNMP must be configured and enabled on the HMC. For additional information about using and setting up CPM, see the publications: z/OS MVS Capacity Provisioning Users Guide, SC33-8299 zEnterprise System Capacity on Demand Users Guide, SC28-2605
12.5.7 Server Time Protocol support

With the Server Time Protocol (STP) functions, the role of the HMC has been extended to provide the user interface for managing the Coordinated Timing Network (CTN). The zEC12 relies solely on STP for time synchronization, and continues to provide support of a Pulse per Second (PPS) port. It is possible to have a zEC12 server as a Stratum 2 or Stratum 3 server in a Mixed CTN linked to z10s (STP configured) attached to the Sysplex Timer operating as Stratum 1 servers. In such a configuration two Stratum 1 servers are highly recommended to provide redundancy and avoid a single point of failure. The zEC12 can not be in the same Mixed CTN with a System z9 (n-2) or earlier systems. Figure 12-12 shows what is supported with zEC12 and previous System z regarding Sysplex and STP.
413
8049ch12.fm
z9 EC or earlier - CF & OS
EC12
(Sysplex and STP)
Not Supported
Support for N-2 Only !
Not Supported
IMPORTANT
z2120 can be in sam e STP CTN as z196, z114 and z10 servers but NOT in a STP CTN with z9 servers
z9 BC or earlier - CF and OS
(Sysplex and STP)
Figure 12-12 Parallel Sysplex System z coexistence
In an STP-only CTN, the HMC can be used to perform the following tasks: Initialize or modify the CTN ID. Initialize the time, manually or by contacting an NTP server. Initialize the time zone offset, daylight saving time offset, and leap second offset. Assign the roles of preferred, backup, and current time servers, as well as arbiter. Adjust time by up to plus or minus 60 seconds. Schedule changes to the offsets listed. STP can automatically schedule daylight saving time, based on the selected time zone. Monitor the status of the CTN. Monitor the status of the coupling links initialized for STP message exchanges. For diagnostic purposes the Pulse per Second port state on a zEC12 can be displayed and fenced ports can be reset individually.
STP recovery enhancement

STP recovery has been enhanced since zEnterprise. See Chapter , STP recovery enhancement on page 166. For additional planning and setup information, see the following publications: Server Time Protocol Planning Guide, SG24-7280 Server Time Protocol Implementation Guide, SG24-7281 Server Time Protocol Recovery Guide, SG24-7380
12.5.8 NTP client/server support on HMC

The Network Time Protocol (NTP) client support allows an STP-only Coordinated Timing Network (CTN) to use an NTP server as an External Time Source (ETS). 414
8049ch12.fm
Note: External Time Source (ETS) connection through modem is not supported on the zEC12 HMC. This capability addresses the following requirements: Customers who want time accuracy for the STP-only CTN Customers using a common time reference across heterogeneous platforms NTP server becomes the single time source, ETS for STP, as well as other servers that are not System z (such as AIX, Windows, and others) that have NTP clients. The HMC can act as an NTP server. With this support, the zEC12 can get time from the HMC without accessing other than the HMC/SE network. When the HMC is used as an NTP server, it can be configured to get the NTP source from the Internet. For this type of configuration, a LAN separate from the HMC/SE LAN can be used.
HMC NTP broadband authentication support for zEC12

The possibility to use HMC NTP authentication is added with HMC level 2.12.0. The SE NTP support is unchanged. To use this option on the SE, configure the HMC with this option as an NTP server for the SE.
Authentication support with a proxy

Some customer configurations use a proxy to access outside the corporate data center. NTP requests are UDP socket packets and cannot pass through the proxy. The proxy must be configured as an NTP server to get to target servers on the web. Authentication can be set up on the customers proxy to communicate to the target time sources.
Authentication support with a firewall

Some customers use a firewall. HMC NTP requests can pass through the firewall. Customers in this configuration should use the HMC authentication to ensure un-tampered time stamps.
Symmetric key and autokey authentication

With the symmetric key and autokey authentication the highest level of NTP security is available. Level 2.12.0 provides panels that accept and generate key information to be configured into the HMC NTP configuration, and offer the possibility to issue NTP commands, as shown in Figure 12-13 on page 416.
415
8049ch12.fm
Figure 12-13 HMC NTP broadband authentication support
Symmetric key (NTP V3-V4) authentication Symmetric key authentication as described in RFC-1305 (made available in NTP Version 3). Symmetric key encryption uses same key for both encryption and decryption. Users exchanging data keep this key to themselves. Messages encrypted with a secret key can be only decrypted with the same secret key. Symmetric does support network address translation (NAT). Symmetric key autokey (NTP V4) authentication Autokey which uses public key cryptography as described in RFC-5906 (made available in NTP Version 4). The key generation for the HMC NTP is performed by clicking the Generate Local Host Key button on the Autokey Configuration panel. Pressing this button will issue the ntp-keygen command to generate the specific key and certificate for this system. Autokey authentication is not available with NAT firewall. Issue NTP commands NTP Command support is also added to display the status of remote NTP servers and the current NTP server (HMC). For additional planning and setup information for STP and NTP, see the following publications: Server Time Protocol Planning Guide, SG24-7280 Server Time Protocol Implementation Guide, SG24-7281 Server Time Protocol Recovery Guide, SG24-7380
Time coordination for zBX components

Network Time Protocol (NTP) clients, running on blades in the zBX, can synchronize their time to the SEs Battery Operated Clock (BOC) The SE's BOC is synchronized to the zEC12 416
8049ch12.fm
Time-of-Day (TOD) clock every hour and allows the SE's clock to maintain a time accuracy of 100 milliseconds to an NTP server configured as the External Time Source in an STP-only CTN. This is shown on Figure 12-14.
Figure 12-14 Time coordination for zBX components
12.5.9 Security and user ID management

This section discusses security and user ID management considerations.
HMC/SE security audit improvements

With the Audit & Log Management task, audit reports can be generated, viewed, saved, and off-loaded. The Customize Scheduled Operations task allows for scheduling of audit report generation, saving, and off loading. The Monitor System Events task allows for Security Logs to result in e-mail notifications using the same type of filters and rules that are used for both hardware and operating system messages. With zEC12, you have the ability to off load the following HMC and SE log files for Customer Audit: Console Event Log Console Service History Tasks Performed Log Security Logs System Log Full log off load as well as delta log off load (since last off load request) is provided. Off loading to removable media as well as to remote locations by FTP is available. The off loading can be manually initiated by the new Audit & Log Management task or scheduled by the Scheduled Operations task. The data can be off-loaded in the HTML and XML formats.
417
8049ch12.fm
HMC User ID Templates and LDAP User Authentication

LDAP User Authentication and HMC User ID templates enable adding/removing HMC users according to your own corporate security environment, by using an LDAP server as the central authority. Each HMC User ID template defines the specific levels of authorization levels for the tasks/objects for the user mapped to that template. The HMC User is mapped to a specific User ID template by User ID pattern matching and/or obtaining the name of the User ID template from content in the LDAP Server schema data.
View Only User IDs/Access for HMC/SE

With HMC and SE User ID support, users can be created who have View Only access to selected tasks. Support for View Only user IDs is available for the following purposes: Hardware Messages Operating System Messages Customize/Delete Activation Profiles Advanced Facilities Configure On/Off
HMC and SE Secure FTP support

You can use a secure FTP connection from a HMC/SE FTP client to a customer FTP server location. This is implemented using the SSH File Transfer Protocol, which is an extension of the Secure Shell protocol (SSH). The Manage SSH Keys console action (available to both HMC and SE) allows you to import public keys associated with a host address. Secure FTP infrastructure allows HMC/SE applications to query if a public key is associated with a host address as well as to utilize the Secure FTP interface with the appropriate public key for a given host. Tasks utilizing FTP now provide a selection for the Secure Host connection. When selected, the task verify that a public key is associated with the specified host name, and if none is provided a message box appears to point to the Manage SSH Keys task to input one. Tasks that provide this support include: Import/Export IOCDS Advanced Facilities FTP ICC Load Audit and Log Management (Scheduled Operations Only)
12.5.10 System Input/Output Configuration Analyzer on the SE and HMC

A System Input/Output Configuration Analyzer task is provided that supports the system I/O configuration function. The information necessary to manage a system's I/O configuration has to be obtained from many separate sources. The System Input/Output Configuration Analyzer task enables the system hardware administrator to access, from one location, the information from these many sources. Managing I/O configurations then becomes easier, particularly across multiple servers. The System Input/Output Configuration Analyzer task performs the following functions: Analyzes the current active IOCDS on the SE. Extracts information about the defined channel, partitions, link addresses, and control units. Requests the channels node ID information. The FICON channels support remote node ID information, which is also collected.
418
8049ch12.fm
The System Input/Output Configuration Analyzer is a view-only tool. It does not offer any options other than viewing options. With the tool, data is formatted and displayed in five different views, various sort options are available, and data can be exported to a USB flash memory drive (UFD) for a later viewing. The following five views are available: PCHID Control Unit View, which shows PCHIDs, CSS, CHPIDs and their control units. PCHID Partition View, which shows PCHIDS, CSS, CHPIDs and the partitions they are in. Control Unit View, which shows the control units, their PCHIDs, and their link addresses in each CSS. Link Load View, which shows the Link address and the PCHIDs that use it. Node ID View, which shows the Node ID data under the PCHIDs.
12.5.11 Automated operations

As an alternative to manual operations, an application can interact with the HMC and SE through an application programming interface (API). The interface allows a program to monitor and control the hardware components of the system in the same way a human can monitor and control the system. The HMC APIs provide monitoring and control functions through SNMP and CIM. These APIs provide the ability to get and set a managed objects attributes, issue commands, receive asynchronous notifications, and generate SNMP traps. The HMC supports the Common Information Model (CIM) as an additional systems management API. The focus is on attribute query and operational management functions for System z, such as CPCs, images, and activation profiles. The zEC12 contains a number of enhancements to the CIM systems management API. The function is similar to that provided by the SNMP API. For additional information about APIs, see the System z Application Programming Interfaces , SB10-7030.
12.5.12 Cryptographic support

This section lists the cryptographic management and control functions available in Hardware Management Console and Support Element.
Cryptographic hardware
The IBM zEnterprise EC12 includes both standard cryptographic hardware and optional cryptographic features for flexibility and growth capability. The HMC/SE interface provides the following capabilities: Define the cryptographic controls Dynamically add a Crypto feature to a partition for the first time Dynamically add a Crypto feature to a partition already using Crypto Dynamically remove Crypto feature from a partition The Crypto Express4S, a new Peripheral Component Interconnect Express (PCIe) Cryptographic Coprocessor, was introduced. It is an optional and zEC12 exclusive feature. This provides a secure programming and hardware environment wherein crypto processes are performed. Each Crypto Express4S adapter can be configured by the installation as a Secure IBM CCA coprocessor, as a Secure IBM Enterprise PKCS #11 (EP11) coprocessor or as an accelerator.
419
8049ch12.fm
When EP11 mode is selected, a unique Enterprise PKCS #11 firmware is loaded into de cryptographic coprocessor. It is separate from the CCA firmware currently loaded when CCA coprocessor is selected. CCA firmware and PKCS #11 firmware cannot coexist at same time in a card. Trusted Key Entry (TKE) Workstation with smart card reader feature is required to support the administration of the Crypto Express4S when configured as an Enterprise PKCS #11 coprocessor. Crypto Express3 is also available in a carry forward only basis when upgrading from earlier generations tozEC12. In order to support the new Crypto Express4S card, the Cryptographic Configuration panel was changed to support the card modes as follows: Accelerator mode (CEX4A) CCA Coprocessor mode (CEX4C) PKCS #11 Coprocessor mode (CEX4P) Other Cryptographic Configuration panel updates include: Support for a Customer Initiated Selftest (CIS) for Crypto running EP11 Coprocessor mode. TKE commands always permitted for EP11 mode. The Test RN Generator function was modified/generalized to also support CIS, depending on the mode of the crypto card. Crypto Details panel changed to display the crypto part number. Support is now provided for up to 4 UDX files. Only UDX CCA is currently supported for zEC12. UDX import now only supports importing from DVD-ROM The next Figure 12-15 shows an example of the Cryptographic Configuration.
Figure 12-15 Cryptographic Configuration panel
A Usage Domain Zeroize task is provided to clear the appropriate partition crypto keys for a given usage domain when removing a crypto card from a partition. Crypto Express4S in EP11 mode will be configured to the standby state after Zeroize.
420
8049ch12.fm
For detailed set-up information, see IBM zEnterprise EC12 Configuration Setup, SG24-8034.
Digitally signed firmware

One critical issue with firmware upgrades is security and data integrity. Procedures are in place to use a process to digitally sign the firmware update files sent to the HMC, the SE, and the TKE. Using a hash-algorithm, a message digest is generated that is then encrypted with a private key to produce a digital signature. This operation ensures that any changes made to the data will be detected during the upgrade process by verifying the digital signature. It helps ensure that no malware can be installed on System z products during firmware updates. It enables, with other existing security functions,zEC12 CPACF functions to comply with Federal Information Processing Standard (FIPS) 140-2 Level 1 for Cryptographic Licensed Internal Code (LIC) changes. The enhancement follows the System z focus of security for the HMC and the SE.
12.5.13 z/VM virtual machine management

The HMC can be used for basic management of z/VM and its virtual machines. The HMC exploits the z/VM Systems Management Application Programming Interface (SMAPI) and provides a graphical user interface (GUI)-based alternative to the 3270 interface. Monitoring the status information and changing the settings of z/VM and its virtual machines are possible. From the HMC interface, virtual machines can be activated, monitored, and deactivated. Authorized HMC users can obtain various status information, such as these: Configuration of the particular z/VM virtual machine z/VM image-wide information about virtual switches and guest LANs Virtual Machine Resource Manager (VMRM) configuration and measurement data The activation and deactivation of z/VM virtual machines is integrated into the HMC interface. You can select the Activate and Deactivate tasks on CPC and CPC image objects, and for virtual machines management. An event monitor is a trigger that is listening for events from objects managed by HMC. When z/VM virtual machines change their status, they generate such events. You can create event monitors to handle the events coming from z/VM virtual machines. For example, selected users can be notified by an e-mail message if the virtual machine changes status from Operating to Exception, or any other state. In addition, in z/VM V5R4 (or later releases), the APIs can perform the following functions: Create, delete, replace, query, lock, and unlock directory profiles. Manage and query LAN access lists (granting and revoking access to specific user IDs). Define, delete, and query virtual CPUs, within an active virtual image and in a virtual image's directory entry. Set the maximum number of virtual processors that can be defined in a virtual image's directory entry.
12.5.14 Installation support for z/VM using the HMC

Starting with z/VM V5R4 and System z10, Linux on System z can be installed in a z/VM virtual machine from the HMC workstation drive. This Linux on System z installation can exploit the existing communication path between the HMC and the SE, where no external network and no additional network setup is necessary for the installation.
421
8049ch12.fm
12.6 HMC in an ensemble

An ensemble is a platform systems management domain consisting of up to eight zEC12 or zEnterprise nodes. Each node comprises a zEnterprise CPC and its optional attached IBM zEnterprise BladeCenter Extension (zBX). The ensemble provides an integrated way to manage virtual server resources and the workloads that can be deployed on those resources. The IBM zEnterprise System (zEnterprise) is a workload optimized technology system that delivers a multi-platform, integrated hardware system; spanning System z, System p, and System x blade server technologies. Management of the ensemble is provided by the IBM zEnterprise Resource Manager. Note: The ensemble HMC mode is only available for managing zEC12 and IBM zEnterprise Systems.
12.6.1 Unified Resource Manager

The ensemble is provisioned and managed through the Unified Resource Manager residing in the HMC. The Unified Resource Manager provides a large set of functions for system management.
Hy pe rv is or M an age m ent Hyp ervi sors (except z/VM) shipped and serviced as firmware. Integrate d depl oyment and config uration of hypervisors Mana ge and control communication betw en virtu al server operating e systems and the hypervisor.
Ene rgy M a nage m ent Moni toring and trend reporting of energy efficiency. Ab ility to query maximum pote ntial power. Po wer saving Po wer capping
Hype rvisors Ope r ationa l C ontr ols HMC provi des a si ngle con sol idate d and consiste nt vi ew of resources Auto-discove ry and configuration supp ort for ne w resource s. Cross platform hardware probl em detection, reporti ng and call home. Physical hardware configurati on, backup and re store. Delivery of system acti vi ty u sin g new use r.
Energy
Operations
Performance
Network s
Virtual Serve rs
Work load A war e nes s a nd Platf orm Pe rfor m anc e Ma nage m ent Wizard-driven man agemen t of resources in acco rda nce with specified business servi ce level ob jectives HMC p rovid es a sin gle consoli dated and consistent view of resources Monitor resource use withi n the context of a busine ss worklo ad Define workloads and associated performance po licies
Ne twor k M ana ge me nt Creation of vitual networks Man ageme nt of vi rtual networks including access control
Key
Manag e suite
Automate suite
Figure 12-16 Unified Resource Manager functions and suites
V ir tua l S er ve r Life c yc le M ana ge me nt Sin gle vie w of virtu alization across platforms. Abi lity to deplo y mul tip le, cro ss-platform virtua l servers within mi nutes Manag ement of virtual networks includ ing access contro l
Overview
Unified Resource Manager provides the following functions:
422
8049ch12.fm
Hypervisor Management: Provides tasks for managing hypervisor life-cycle, managing storage resources, performing RAS (reliability, availability, and serviceability) and FFDC (First Failure Data Capture) features and monitoring the supported hypervisors. Ensemble membership management: Provides tasks for creating an ensemble and controlling membership of the ensemble. Storage Management: Provides a common user interface for allocation and deallocation of physical and virtual storage resources for an ensemble. Virtual server management: Provides life-cycle management to create, delete, activate, deactivate, and modify definitions of virtual servers. Virtual network management: Provides for the management of networking resources for an ensemble. Performance management: Provides a global performance view of all the virtual servers supporting workloads deployed in an ensemble. The virtual server workload performance goal is like a simplified z/OS WLM policy: You can define, monitor, report, and manage the performance of virtual servers based on workload performance policies. Policies are associated to the workload: From the overall Workload performance health report, you can review contributions of individual virtual servers. You can manage resources across virtual servers within a hypervisor instance.
Energy management: Monitors energy usage and control power-saving settings, accessed through the new monitors dashboard. Monitoring of virtual server resources for CPU use and delays, with capability of creating a graphical trend report Unified Resource Manager supports different levels of system management. A feature determines these management functions and operational controls that are available for a zEnterprise mainframe and any attached zBX. These named features are Manage and Automate/Advanced Management: Manage suite Provides Unified Resource Managers function for core operational controls, installation, and energy monitoring. It is configured by default and activated when an ensemble is created. workload definition and performance policy monitoring and reporting. The Automate functionality adds, on top of the Advanced Management functionality, goal oriented resource monitoring management and energy management for CPC components, IBM Smart Analytic Optimizer, POWER7 Blade, and DataPower XI50z.
Automate/Advanced Advanced Management functionality for IBM System x Blades delivers Management suite
423
8049ch12.fm
Table 12-2 lists the feature codes that must be ordered for enabling Unified Resource Manager. To get ensemble membership, make sure that you also order FC 0025 for zEC12.
Table 12-2 Unified Resource Manager feature codes and charge indicator. Unified Resource Manager managed component Managea - per connection Advanced Managementa - per connection Automatea - per connection
base features IFL POWER7 Blade DataPower Blade IBM System x Blades
0019b - N/C N/C 0041 - Yes 0040 - Yes 0042 - Yes
N/A N/A N/A N/A 0046 - Yes
0020c - N/C 0052 - Yes 0045 - Yes 0044 - N/C N/A
a. Yes = charged feature, N/C = no charge, N/A = not applicable. All components are either managed through the
Manage suite, or the Automate/Advanced Management suite. The Automate/Advanced Management suite contains the functionality of the Managed suite. b. Feature code 0019 is a prerequisite for feature codes 0020, 0039, 0040, 0041, and 0042. c. Feature code 0020 is a prerequisite for feature codes 0043, 0044, 0045, 0046, and 0052.
APIs for IBM zEnterprise Unified Resource Manager

Statement of Direction: IBM intends to offer APIs for IBM zEnterprise Unified Resource Manager. These APIs are designed to provide access to the same underlying functions that support the Unified Resource Manager user interface and can be exploited to enable discovery, monitoring, and provisioning use cases. Note: These APIs enable management of Unified Resource Manager from external tools, like IBM Systems Director, IBM Tivoli or ISV systems management products. IBMs priority scenario is, in the following sequence, to first provide Discovery, Monitoring, and then access to Provisioning functions of Unified Resource Manager. For more information regarding the Unified Resource Manager, see the Redbooks publication, IBM zEnterprise Unified Resource Manager, SG24-7921; and the product documentation, IBM zEnterprise System Introduction to Ensembles, SC27-2609.
12.6.2 Ensemble definition and management

The ensemble starts with a pair of HMCs that are designated as the primary and alternate HMCs, and are assigned an ensemble identity. The zEnterprise CPCs and zBXs are then added to the ensemble through an explicit action at the primary HMC.
Feature code
Feature code 0025 (Ensemble Membership Flag) is associated with an HMC when a zEC12 is ordered. This feature code is required on the controlling zEC12 to be able to attach a zBX. A new task called Create Ensemble will allow the Access Administrator to create an ensemble that contains CPCs, Images, workloads, virtual networks and storage pools, either with or without an optional zBX. If a zEC12 has been entered into an ensemble, then the CPC details task on the SE and HMC will reflect the ensemble name.
424
8049ch12.fm
Unified Resource Manager actions for the ensemble are conducted from a single primary HMC. All other HMCs connected to the ensemble will be able to perform system management tasks (but not ensemble management tasks) for any CPC within the ensemble. The primary HMC can also be used to perform system management tasks on CPCs that are not part of the ensemble, such as Load, Activate, and so on. The ensemble-specific managed objects include: Ensemble Members Blades BladeCenters Hypervisors Storage Resources Virtual Servers Workloads When another HMC accesses an ensemble nodes CPC, the HMC can do the same tasks as if the CPC were not a part of an ensemble. A few of those tasks have been extended to allow you to configure certain ensemble-specific properties (such as setting the virtual network associated with OSAs for a LPAR). Showing ensemble-related data in certain tasks is allowed. Generally, if the data affects the operation of the ensemble, then the data is read-only on another HMC. The tasks that show ensemble-related data on another HMC are as follows: Scheduled operations: Displays ensemble introduced scheduled operations, but you can only view these scheduled operations. User role: Shows ensemble tasks and you can modify and delete those roles. Event monitoring: Displays ensemble-related events, but you cannot change or delete the event.
HMC considerations when used to manage an ensemble

Here we list considerations for using Unified Resource Manager to manage an ensemble: All HMCs at the supported code level are eligible to create an ensemble. Only HMCs FC 0091 are capable of being primary or alternate HMCs for zEC12. The primary and the alternate HMC must be the same machine type/feature code. There is a single HMC pair managing the ensemble: primary HMC and alternate HMC. Only one primary HMC manages an ensemble, which can consist of a maximum of eight CPCs. The HMC that performed the Create Ensemble wizard becomes the primary HMC. An alternate HMC is elected and paired with the primary. Primary Hardware Management Console (Version 2.12.0 or later) and Alternate Hardware Management Console (Version 2.12.0 or later) will appear on the HMC banner. When the ensemble is deleted, the titles will resort to default. A primary HMC is the only HMC that can perform ensemble-related management tasks (create virtual server, manage virtual networks, create workload, and so on) A zEnterprise ensemble can have a maximum of eight nodes and is managed by one primary HMC and its alternate. Each node comprises a zEnterprise CPC and its optional attached IBM zEnterprise BladeCenter Extension (zBX). Any HMC can manage up to 100 CPCs. The primary HMC can perform all non-ensemble HMC functions on CPCs that are not members of the ensemble. The primary and alternate HMCs must be on the same LAN segment. 425
8049ch12.fm
The alternate HMCs role is to mirror ensemble configuration and policy information from the primary HMC. When failover happens, the alternate HMC will become the primary HMC. This behavior is the same as the current primary and alternate Support Elements.
12.6.3 HMC availability

The HMC is attached to the same LAN as the servers support element (SE). This LAN is referred to as the Customer Managed Management Network. The HMC communicates with each CPC, and optionally to one or more zEnterprise BladeCenter Extensions (zBXs), through the SE. If the zEC12 node is defined as a member of an ensemble, the primary HMC is the authoritative controlling (stateful) component for Unified Resource Manager configuration and policies that have a scope that spans all of the managed CPCs/SEs in the ensemble. The managing HMC has an active role in ongoing system monitoring and adjustment. This requires the HMC to be configured in an primary/alternate configuration and cannot be disconnected from the managed ensemble members. Failover: The primary HMC and its alternate must be connected to the same LAN segment to allow the alternate HMC to take over the IP address of the primary HMC during failover processing.
12.6.4 Considerations for multiple HMCs

Customers often deployed multiple HMC instances to manage an overlapping collection of systems. Until the emergence of ensembles all of the HMCs were peer consoles to the managed systems, and all management actions are possible to any of the reachable systems, while logged into a session on any of the HMCs (subject to access control). With the Unified Resource Manager, this paradigm has changed. On ensemble is managed by one primary and alternate HMC pair. Multiple ensembles will require an equal number of multiple primary and alternate HMC pairs to manage. In this environment, if a zEC12 or zEnterprise System node has been added to an ensemble, management actions targeting that system can only be done from the managing (primary) HMC for that ensemble.
12.6.5 HMC browser session to a primary HMC

A remote HMC browser session to the primary HMC that is managing an ensemble allows a user currently logged onto another HMC or a workstation to perform ensemble-related actions.
426
8049ch12.fm
12.6.6 HMC ensemble topology

The system management functions that pertain to an ensemble, exploit the virtual server resources and the intraensemble management network (IEDN). They are provided by the HMC/SE by the internode management network (INMN). Figure 12-17 depicts an ensemble with two zEC12 and a zBX that are managed by the Unified Resource Manager residing in the primary and alternate HMCs. CPC1 controls the zBX, while CPC2 is a stand-alone CPC.
Primary
Client Network
A lternate
1 HMC
O S M
HMC EC12 CPC1 Controls zBX
O S M
EC12 CPC2
2
1. HMC to SE Network 2. INMN (EC12) 3. INMN (zBX) 4. IEDN 5. FC Disk Storage (zBX)
OSX
zBX
5
Fibre Channel Switch
Figure 12-17 Ensemble example - with primary and alternate HMCs
For the stand-alone CPC ensemble node (CPC2), two OSA-Express4S 1000BASE-T ports (CHPID type OSM) connect to the Bulk Power Hubs (port J07) with 3.2 meter Category 6 Ethernet cables. The HMCs also communicate with all the components of the ensemble by the BPHs in the CPC. The OSA-Express4S 10 GbE ports (CHPID type OSX) are plugged with client provided 10 GbE cables (either SR or LR, depending on the OSA feature). Details for zBX can be found in Chapter 7, zBX zEnterprise BladeCenter Extension Model 003 on page 211.
427
8049ch12.fm
428
8049ax01.fm
Appendix A.
IBM zAware
Instead of trying to produce a programme to simulate the adult mind, why not rather try to produce one which simulates the child's? If this were then subjected to an appropriate course of education one would obtain the adult brain.
Computing Machinery and Intelligence by Alan Turing
In this appendix we introduce IBM System z Advanced Workload Analysis Reporter (IBM zAware), the next generation of system monitoring. It is a new feature that is designed to offer a near real-time, continuous learning, diagnostics and monitoring capability. IBM zAware helps you pinpoint and resolve potential problems quickly enough to minimize impacts to your business. In this appendix, the following topics are discussed: A.1, Troubleshooting in Complex IT Environments on page 430 A.2, Introducing the IBM zAware on page 431 A.3, IBM zAware Technology on page 432 A.4, IBM zAware PreRequisites on page 438 A.5, Configuring and using the IBM zAware virtual appliance on page 439 For comprehensive information about IBM zAware, see IBM zAware Concept Guide , SG24-8070 and Advanced Workload Analysis Reporter (IBM zAware), SC27-2623.
429
8049ax01.fm
A.1 Troubleshooting in Complex IT Environments

In a 24 x 7 operating environment, a system problem or incident can drive up operations costs and disrupt service to the clients for hours -- even days. Todays IT environments cannot afford recurring problems or outages that take too long to repair; as it may result in external reputation damage and limit the ability to remain competitive in the marketplace. However, as the systems become more complex errors can occur anywhere; some incidents begin early with symptoms that go undetected for long periods of time. Systems often experience soft failures (sick but not dead) which are much more difficult or unusual to detect. Moreover, problems can grow, cascade and snowball. Many everyday activities can introduce system anomalies and initiate either hard or soft failures in complex, integrated data centers; these activities include: Increased volume of business activity Application modifications to comply with changing regulatory requirements IT efficiency efforts like consolidating images Standard operational changes, such as: Adding, upgrading hardware (Server, Storage, etc.) Adding, upgrading software (Operating Systems, Middleware, Independent Software Vendor products, applications, etc.) Modifying network configurations Moving workloads (provisioning, balancing, deploying, DR testing, etc.) Although using a combination of existing system management tools helps to diagnose problems, they usually do little to quickly identify messages preceding system problems or cannot detect every possible combination of change and failure. So even when using these tools, it might be still needed to look through message logs to understand the underlying issue. But the number of messages makes it not only too challenging and skills-intensive task, but also an error prone task.
A.1.1 Smarter computing needs smarter monitoring

To meet IT service challenges and to effectively sustain high levels of availability, a proven way is needed to identify, isolate and resolve system problems quickly. Information and insight become vital in order to understand typical system behavior along with possible deviations, reduce the time to diagnose problems, address them quickly and accurately. Todays complex, integrated data centers require a team of experts to monitor systems and perform real time diagnosis of real time events. But it is not always possible to afford such high skill demand considering that: A z/OS sysplex might produce more than 40GB of message traffic per day for its images and components alone Application messages can significantly increase that number There are more than 40,000 unique message IDs defined in z/OS and the IBM software that runs on z/OS. ISV or client messages can increase that number
430
8049ax01.fm
A.2 Introducing the IBM zAware

IBM zAware is an integrated expert solution which contains sophisticated analytics, IBM insight into the problem domain, and web browser based visualization. It is an adaptive, smart analytics solution that learns your unique system characteristics and helps you to detect and diagnose unusual behavior of z/OS images in near real time, accurately and rapidly. Statement of Direction: IBM plans to provide new capability within the Tivoli Integrated Service Management family of products designed to leverage analytics information from IBM zAware, and to provide alert and event notification. Running on a client-visible logical partition as a virtual appliance and providing out-of-band monitoring, IBM zAware converts data into information and provides visualization to help a human develop insight into the behavior of a complex systems like a z/OS sysplex. It reduces problem determination time and improve service availability even beyond what it is in z/OS today.
A.2.1 Value of IBM zAware

Early detection and focused diagnosis can help improving time to recover from complex z/OS problems (including cross sysplex, across a set of System z servers, and beyond CPC boundaries). IBM zAware delivers sophisticated detection and diagnostic capabilities identifying when and where to look for a problem, the cause of the anomalies that would be hard to spot. High speed analytics on large quantities of log data reduces problem determination and isolation efforts, time to repair, impact to service levels and provides system awareness for a smarter monitoring. IBM zAware provides an easy to use graphical user interface with quick drill-down capabilities where you can view analytical data that indicates which system is experiencing deviations in behavior, when the anomaly occurred, and whether the message was issued out of context. IBM zAware GUI fits into existing monitoring structure, and can also feed other processes or tools to take corrective action for faster problem resolution.
A.2.2 IBM z/OS Solutions to Improve Problem Diagnostics

In order to position IBM zAware as a smarter monitoring tool among all other problem diagnostic solutions for IBM z/OS, it is useful to review Table A-1 on page 431 comparing options from several aspects.
Table A-1 Positioning IBM zAware Solutions Available Rules based Analytics / Statistical model Examines message traffic Self Learning Method
z/OS Health Checkera
Checks configurations Programmatic, applies to IBM and ISV tools Can escalate notifications
Yes
Rules based
Appendix A. IBM zAware
431
8049ax01.fm
Solutions Available
Rules based
Analytics / Statistical model
Examines message traffic
Self Learning
Method
z/OS PFAa z/OS RTDa IBM zAware
Trending analysis of z/OS system resources, and performance Can invoke z/OS RTD Real time diagnostics of specific z/OS system issues Pattern based message analysis Self learning Provides aid in diagnosing complex z/OS problems, including cross sysplex Yes
Yes
Yes
Early detection Rules based
Yes Yes Yes Yes
Diagnosis
a. Included in z/OS
Any large and complex z/OS installation with mission-critical applications and middleware, is suggested to exploit the IBM zAware along with z/OS included problem diagnosis solutions.
A.3 IBM zAware Technology

The IBM zAware runs analytics in firmware and intelligently examines OPERLOG data for potential deviations, inconsistencies, or variations from the norm. It automatically manages the creation of the behavioral model which is used to compare current message log data from the connected z/OS system(s). Historical data, machine learning, mathematical modeling, statistical analysis and cutting edge pattern recognition techniques combine to uncover unusual patterns and to understand the nuances of your unique environment. Figure A-1 on page 433 depicts basic components of a IBM zAware environment;
432
8049ax01.fm
Vie w IB M zA wa re re sults
z /OSM F
IBM zAware GUI

z EC12 IBM zAware host IBM zAware m onitored cli ent
me sto Cu
o rk e tw rn
z/OS
I BM zAware Host Parti ti on
z/OS
z/OS
z/OS
z/OS
Pe rsiste nt Stora ge
File Syste m
T IP
n un
el
Manage IB M zA wa re Firmwa re partition (simila r to CF)
Figure A-1 Elements of a IBM zAware configuration
IBM zAware runs in a logical partition as firmware. IBM zAware; requires the zEC12 configuration to have a priced feature code. needs processor, memory, disk, and network resources to be assigned to the LPAR it runs Shows similarity with Coupling Facility in many ways is updated like all other firmware, with a separate Engineering Change stream is loaded from the Support Element Hard Disk demonstrates out of band monitoring with minimal effect on z/OS product workloads Figure A-2 on page 434 shows IBM zAware Image Profile on HMC
433
8049ax01.fm
Figure A-2 HMC Image Profile for a IBM zAware LPAR
IBM zAware analyzes massive amounts of OPERLOG messages (including all z/OS console messages, ISV and application generated messages) to build Sysplex and detailed views in the IBM zAware GUI. Figure A-3 on page 435 and Figure A-4 on page 436 show samples for both Sysplex and Detailed views consecutively.
434
8049ax01.fm
Figure A-3 IBM zAware Sysplex view; showing all connected managed z/OS clients
435
8049ax01.fm
Figure A-4 IBM zAware detailed view; drilled down to a single z/OS image
The analytics creates a statistical model of the normal message traffic generated by each individual z/OS. Using this model which is stored in a database, unexpected messages and patterns of messages are identified. Using a sliding ten minute interval which is updated every two minutes, a current score for the interval is created based on how unusual the message traffic is. Stable system requires lower interval score to be marked as interesting or rare Unstable system requires larger interval score to be marked as interesting or rare For each interval IBM zAware provides details of all of the unique and unusual message IDs found within interval including how many, how rare, how much they contributed to the intervals score (anomaly score, interval contribution score, rarity score, appearance count), when they first appeared. IBM zAware also handles burst of messages and identifies; if it is one component emitting unusual message IDs, if the message is a critical z/OS kernel message, if the messages are related to changes like new software levels (operating system, middleware, applications) or updated system settings/configurations. The choice of unique message IDs is embedded in the domain knowledge of IBM zAware. IBM zAware detects things typical monitoring systems miss due to Message suppression (message too common) - useful for long-term health issues 436
8049ax01.fm
Uniqueness (message not common enough) - useful for real-time event diagnostics IBM zAware decides on the color of an interval using distribution of interval score: Normal: Blue color - interval score between 1- 99.5 Interesting: Orange color - interval score between 99.5 - 100 Rare: Red color - interval score 101
Training Period
The IBM zAware server starts receiving current data from the z/OS system logger running on z/OS monitored clients. However, the server cannot use this data for analysis until a model of normal system behavior exists. The estimated amount of data for building the most accurate models is 90 days of data for each client. By default, training automatically runs every 30 days. Your installation can modify the number of days required for this training period, based on your knowledge of the workloads running on z/OS monitored clients. This training period applies for all monitored clients; different training period for each client cannot be defined.
Priming IBM zAware

Instead of waiting for the IBM zAware server to collect data over the course of the training period, you can prime the server by transferring prior data for monitored clients, and request the server to build a model for each client from the transferred data.
IBM zAware Graphical User Interface

IBM zAware creates XML data with the status of the z/OS image and with details of the message traffic. This data is rendered by the web server running as a part of IBM zAware and is available using a standard web browser (Internet Explorer 8, Mozilla Firefox or Chrome). As shown in Figure A-5 on page 438, IBM zAware provides a easy to use browser based GUI with relative weighting and color coding.
437
8049ax01.fm
Figure A-5 IBM zAware Graphical User Interface
For IBM messages; IBM zAware GUI has a link to the message description which often includes a recommended action to correct the issue highlighted by the message. There also is a z/OSMF link on the navigation bar.
IBM zAware is complementary to your existing tools Compared to existing tools, IBM zAware works out of the box with relatively little
customization. It does not depend on other solutions or manual coding of rules and is always enabled to watch your system. The XML output built by IBM zAware can be queued up to existing system monitoring tools like Tivoli via published API.
A.4 IBM zAware PreRequisites

This section describes both hardware and software requirements in order to configure IBM zAware.
zEC12 Configuration Requirements:

IBM zAware is orderable as an environment which consists of the quantity of Processor Books present on the host machine plus the quantity of Processor Books present in the client machine(s). IBM zAware related feature codes are: Feature Code 0011 - IBM zAware Feature Code 0101 - IBM zAware Connection Count Feature Code 0102 - IBM zAware Connection Count - DR Machine (Optional) 438
8049ax01.fm
FC 0101 is a priced feature code that represents the quantity (in decimal) of client/host connections. IBM zAware must be ordered from the host to select both host and client connections, so there is no need to order FC 0101 for client machines. Minimum quantity available would be the quantity of Processor Books present on the host (ordering) machine. For instance, if thezEC12 to host IBM zAware is a model H49, minimum quantity will be 2. Maximum quantity that can be ordered as FC 0101 is 99. A Disaster Recovery option is also available and will represent that IBM zAware is installed on a Disaster Recovery zEC12 server. In this case, zero priced feature FC 0102 will be assigned to represent the quantity of connections (in decimal). Note: zEC12 resource requirements are dependent on the number of monitored clients, amount of message traffic, length of time data retained. Processors General Purpose CP or IFL that can be shared with other LPARs in zEC12. Usage estimates between a partial engine to two engines depending on the size of the configuration Memory Minimum 4GB initial memory + 200MB for each z/OS LPAR being monitored Flash Express is not supported Direct Access Storage Devices 250-300 GB persistent DASD storage Only ECKD format, FCP devices are not supported IBM zAware manages its own data store and uses Logical Volume Manager (LVM) to aggregate multiple physical devices into a single logical device Network (for both instrumentation data gathering and outbound alerting/communications) HiperSockets for the z/OS LPARs running on the same zEC12 as the IBM zAware LPAR OSA ports for the z/OS LPARs running on a different CPC than where IBM zAware LPAR runs z/OS LPARs running on other System z servers (IBM zEnterprise 196, IBM System z10 Business Class, etc.) can be monitored clients by sending log files via TCP/IP network to host zEC12 server as long as they fulfill z/OS requirements
z/OS Requirements:
z/OS V1.13 with additional PTFs 90 days historical SYSLOG or formatted OPERLOG data to prime IBM zAware
A.5 Configuring and using the IBM zAware virtual appliance

Following checklist provide a task summary for configuring and using IBM zAware: Phase 1: Planning Plan the configuration of the IBM zAware environment. Plan the LPAR characteristics of the IBM zAware partition. Plan the network connections required for the IBM zAware partition and each z/OS monitored client Plan the security requirements for the IBM zAware server, its monitored clients, and users of the IBM zAware graphical user interface (GUI). Plan for using the IBM zAware GUI Phase 2: Configuring the IBM zAware partition
439
8049ax01.fm
Verify that your installation meets the prerequisites for using the IBM zAware virtual appliance Configure network connections for the IBM zAware partition through the Hardware Configuration Definition (HCD) or the Input/Output Configuration Program (IOCP). Configure persistent storage for the IBM zAware partition through the HCD or IOCP. Define the LPAR characteristics of the IBM zAware partition through the Hardware Management Console (HMC). Define network settings for the IBM zAware partition through the HMC. Activate the IBM zAware partition through the HMC. Phase 3: Configuring the IBM zAware server and its monitored clients: Assign storage devices for the IBM zAware server through the IBM zAware GUI. (Optional) Replace the self-signed Certificate Authority (CA) certificate that is configured in the IBM zAware server. (Optional) Configure an LDAP directory or local file-based repository for authenticating users of the IBM zAware GUI. (Optional) Authorize users or groups to access the IBM zAware GUI. (Optional) Modify the configuration values that control IBM zAware analytics operation. Configure a network connection for each z/OS monitored client through the TCP/IP profile. If necessary, update firewall settings. Verify that each z/OS system meets the sysplex configuration and OPERLOG requirements for IBM zAware virtual appliance monitored clients. Configure the z/OS system logger to send data to the IBM zAware virtual appliance server. Prime the IBM zAware server with prior data from monitored clients. Build a model of normal system behavior for each monitored client. The IBM zAware server uses these models for analysis.
440
8049ax02.fm
Appendix B.
Channel options
The following two tables describe all channel attributes, the required cable types, the maximum unrepeated distance, and the bit rate for the zEC12. For all optical links the connector type is LC Duplex, except the 12xIFB connection is established with an MPO connector. The electrical Ethernet cable for the OSA connectivity are connected through an RJ45 jack. Table B-1 lists the attributes of the channel options supported on zEC12.
Statement of Direction: The zEC12 is the last IBM System z server to support ISC-3 Links. The zEC12 is the last IBM System z server to support Ethernet half-duplex operation and 10 Mbps link data rate on 1000BASE-T Ethernet features. The FICON Express8S features, supporting a 2, 4, or 8 Gbps link data rate, are planned to be the last FICON features to support a 2 Gbps link data rate.
Table B-1 zEC12 channel feature support Channel feature Feature codes Bit rate Cable type in Gbps (or stated) Maximum unrepeated distancea Ordering information
Fiber Connection (FICON)
FICON Express4 10KM LX FICON Express8 10KM LX
3321 3325
1, 2, or 4 2, 4, or 8 1, 2, or 4 2, 4, or 8 OM1,OM2, OM3 See Table B-2 on page 442. SM 9 m 10 km
Carry forward Carry forward New build Carry forward Carry forward New build
FICON Express8S 10KM LX 0409 FICON Express4 SX FICON Express8 SX FICON Express8S SX 3322 3326 0410
Open Systems Adapter (OSA)
441
8049ax02.fm
Channel feature
Feature codes
Bit rate Cable type in Gbps (or stated)
Maximum unrepeated distancea
Ordering information
OSA-Express3 GbE LX OSA-Express4S GbE LX OSA-Express3 GbE SX OSA-Express4S GbE SX
3362 0404 3363 0405 1 1
SM 9 m MCP 50 m
5 km 550 m (500)
Carry forward New build Carry forward New build
MM 62.5 m 220 m (166) 275 m (200) MM 50 m 550 m (500)
OSA-Express3 1000BASE-T OSA-Express4 1000BASE-T OSA-Express3 10 GbE LR OSA-Express4S 10 GbE LR OSA-Express3 10 GbE SR
3367 0408 3370 0406 3371
10, 100, or 1000 Mbps 10
Cat 5, Cat 6 copper SM 9 m 10 km
Carry forward New build Carry forward New build Carry forward New build
MM 62.5 m 33 m (200) 10 MM 50 m 300 m (2000) 82 m (500)
OSA-Express4S 10 GbE SR 0407
Parallel Sysplex
IC ISC-3 (peer mode) ISC-3 (RPQ 8P2197 Peer mode at 1 Gbps)b HCA2-O (12x IFB) HCA2-O LR (1x IFB) HCA3-O (12x IFB) HCA3-O LR (1x IFB)
Cryptography
N/A 2 0217 0218 0219 0163 0168 0171 0170 1 6 GBps 2.5 or 5 Gbps 6 GBps 2.5 or 5 Gbps
N/A SM 9 m MCP 50 m SM 9 m OM3 SM 9 m OM3 SM 9 m
N/A 10 km 550 m (400) 20 km 150 m 10 km 150 m 10 km
N/A Carry forward Carry forward Carry forward Carry forward New build New build
Crypto Express3 Crypto Express4s
0864 0865
N/A N/A
N/A N/A
N/A N/A
Carry forward New build
a. were applicable, Minimum fiber bandwidth distance product in MHzkm for multi-mode fiber optic links are included in parentheses. b. RPQ 8P2197 enables the ordering of a daughter card supporting 20 km unrepeated distance for 1 Gbps peer mode. RPQ 8P2262 is a requirement for that option, and other than the normal mode, the channel increment is two (that is, both ports (FC 0219) at the card must be activated).
Table B-2 Maximum unrepeated distance for FICON SX features Cable type\bit rate 1 Gbps 2 Gbps 4 Gbps 8 Gbps
OM1 (62,5 m at 200 MHzkm)
300 meters 984 feet
150 meters 492 feet
70 meters 230 feet
21 meters 69 feet
442
8049ax02.fm
Cable type\bit rate
1 Gbps
2 Gbps
4 Gbps
8 Gbps
OM2 (50 m at 500 MHzkm) OM3 (50 m at 2000 MHzkm)
500 meters 1640 feet 860 meters 2822 feet
Appendix B. Channel options
443
8049ax02.fm
444
8049ax03 - Flash Express.fm
Appendix C.
Flash Express
This appendix introduces the IBM Flash Express feature available on the zEC12 server. Flash memory is a non-volatile computer storage technology. It was introduced on the market some decades ago. Flash memory is commonly used today in memory cards, USB Flash drives, Solid State Drives (SSDs), and similar products for general storage and transfer of data. Until recently, the high cost per gigabyte and limited capacity of SSDs restricted deployment of these drives to specific applications. Recent advances in SSD technology and economies of scale have driven down the cost of SSDs, making them a viable storage option for I/O intensive enterprise applications. A Solid State Drive (SSD), sometimes called a solid-state disk or electronic disk, is a data storage device that uses integrated circuit assemblies as memory to store data persistently. SSD technology uses electronic interfaces compatible with traditional block I/O hard disk drives. SSDs do not employ any moving mechanical components, which distinguishes them from traditional magnetic disks such as hard disk drives (HDDs), which are electromechanical devices containing spinning disks and movable read/write heads. With no seek time or rotational delays, SSDs can deliver substantially better I/O performance than HDDs. Flash SSDs demonstrate latencies that are at least 10 to 50 times lower than the fastest hard disk drives (HDDs), often enabling dramatically improved I/O response times.
445
C.1 Flash Express overview

Flash Express introduces Solid State Drive (SSD) technology to the IBM zEnterprise EC12 server, implemented using Flash SSDs mounted in PCIe Flash Express feature cards. Flash Express is an innovative solution available on zEC12 designed to help improve availability and performance for getting a higher level of quality of service. It is designed to automatically improve availability for key workloads at critical processing times and drive performance improvements for critical business z/OS workloads that cannot afford page. It can also reduce latency time during diagnostic collection (dump operations). Flash Express introduces a new level in the zEC12 storage hierarchy as showed in Figure C-1 on page 446.
Acces s time
CPU Cache Random Access Memory (RAM) Flash Express Solid State Drive (SSD) Storage Spinning Dis k Driv e WORM, Tape Library
< 20 ns
< 200 ns
5-20 s 1-3 ms
< 10 ms seconds
Figure C-1 zEC12 storage hierarchy
Flash Express is as optional PCIe card feature available on zEC12 servers. Flash Express cards are supported in PCIe I/O drawers and can be mixed with other PCIe I/O cards like FICON Express8S, Crypto Express4S or OSA Express4S cards. You must order a minimum of two features (FC 0408) and a maximum of eight. The cards are ordered in increments of two. Flash Express cards are assigned one PCHID even though they have no ports. There is no HCD/IOCP definition required for Flash Express installation. Flash uses sub-channels that are allocated from the .25K reserved in sub-channel set 0. Similar to other PCIe I/O cards, redundant PCIe paths to Flash Express cards are provided by redundant I/O interconnect. Unlike other PCIe I/O cards, they can only be accessed to the host by a unique protocol. A Flash Express PCIe adapter card integrates four SSD cards of 400 GB each for a total of 1.6 TB of usable data per card as showed in Figure C-2.
446
Figure C-2 Flash Express PCIe adapter card
Each card is installed in a PCIe I/O drawer in two different I/O domains. A maximum of two pairs are installed in a drawer with only one flash card per domain only. Greater than two pairs will require a second PCIe I/O drawer. Cards first installed in the front of the installed drawers on slots 1 and 14, before using the rear slots 25 and 33. Each pair of card must formatted prior to utilization. Figure C-3 shows a PCIe I/O cage fully populated with Flash Express cards.
Top View
Card Slot 01 Card Slot 02 Card Slot 03 Card Slot 04 PCI-IN 0 Card Slot 06 Card Slot 07 Card Slot 08 Card Slot 09 FSP Card Slot 11 Card Slot 12 Card Slot 13 Card Slot 14 PCI-IN 2 D o m a i n 2 D o m a i n 0 Card Slot 38 D Card Slot 37 o m Card Slot 36 a Card Slot 35 i PCI-IN 1 n Card Slot 33 1 Card Slot 32 Card Slot 31 Card Slot 30 FSP Card Slot 28 D Card Slot 27 o Card Slot 26 m a Card Slot 25 i n 3 PCI-IN 3 Card Slot 23 Card Slot 22 Card Slot 21 Card Slot 20
Front
Rear
2 interconnect Card Slot 16 Card Slot 17 cables

Card Slot 18 Card Slot 19
2 interconnect cables
Figure C-3 PCIe I/O cage fully populated with Flash Express cards.
For higher resiliency and high availability, Flash Express cards are always installed by pairs. A maximum of four pairs are supported in a zEC12 system providing with a maximum of 6.4 TB of storage. In each Flash Express card, data is stored in a RAID configuration. If a solid state disk fails, data is reconstructed dynamically. The cards pair are mirror to each other over a pair of cables in RAID 10 configuration mixing mirroring and stripping RAID capabilities. If either card fails, the data is available on the other card. Card replacement is concurrent with customer operations. In addition, Flash Express support concurrent firmware upgrades and card replacement is concurrent with customer operations.
Appendix C. Flash Express
447
The data written on the Flash Express cards is always stored encrypted with a volatile key and the card is only usable on the system with the key that encrypted it. For key management, both the primary and alternate Support Element (SE) have a smart card installed. Smart Card contains both a unique key personalized for each system and a small Crypto engine that can perform a set of security functions within the card.
C.2 Using Flash Express

Flash Express has been designed to improve availability and latency from batch to interactive processing in z/OS environments such as start of day. It helps accelerate start of day processing when there is a heavy application activity. Flash Express also helps in improving snapshots of diagnostics like SVC dump, and stand alone dump. In z/OS, Flash Express memory is accessed using the new System z Extended Asynchronous Data Mover (EADM) architecture) and initiated with a Start subchannel instruction. The Flash Express PCIe cards are shareable across LPARs. Flash Express memory can be assigned to z/OS LPARs like the central storage. It is dedicated to each logical partition and is dynamically configurable, when increasing the amount of Flash Express memory allocated to a logical partition. Flash Express is supported by z/OS 1.13 plus PTFs for the z/OS paging activity and SVC dumps. Using Flash Express memory, 1MB large pages become pageable. It is expected to provide applications with substantial improvement in SVC dump data capture time. Flash Express is expected to provide the applications with improved resiliency and speed, with pageable large pages being introduced. Other software subsystems may exploit Flash Express in the future. Table C-1 gives the minimum support requirements for Flash Express.
Table C-1 Minimum support requirements for Flash Express Operating system Support requirements
z/OS
z/OS V1R13a
a. Web delivery and PTFs required
On the SE or HMC by using the Flash Express allocation panels, you can define the initial and maximum amount of Flash Express available to a logical partition. The maximum memory allocated to a LPAR can be dynamically changed. From the SE or HMC, additional Flash memory up to the maximum can be configured online to a logical partition. On z/OS this can also be done using an operator command. Flash memory can also be configured offline to a logical partition. Figure C-4 on page 449 gives a sample SE/HMC panel interface to used for Flash Express allocation.
448
Figure C-4 Sample SE/HMC panel for Flash Express allocation to LPAR
The new SE user interface for Flash Express provides four new types of actions you can perform: Flash status and control: displays the list of adapters that are installed in the system and their state Manage Flash allocation: display the amount of Flash memory on the system View Flash allocations: displays a table of Flash information for one partition View Flash: display information for one pair of flash adapters. Physical Flash Express PCIe cards are fully virtualized across logical partitions. Each logical partition can be configured with its own SCM address space. The size of Flash Express memory allocated to a partition is done by amount, not by card size. The hardware supports error isolation, transparent mirroring, centralized diagnostic, hardware logging, recovery, independently from the software. At IPL, z/OS detects if flash is assigned to the partition. z/OS automatically uses Flash Express for paging unless specified otherwise via the new z/OS PAGESCM=NONE parameter. All paging data can reside on Flash Express memory. The function is easy to use, and there is no need for capacity planning or placement of data on Flash Express cards. Figure C-5 gives an example of Flash Express allocation between two z/OS LPARs.
449
Stora ge Cla ss Mem ory (S CM)
Stor age C la ss Me mor y (SCM )
SC M SPACE
z/OS
Main Memory
z/OS
Main Memory
SC M SPACE
Pa rtition Ma xim um
Pa rtition M ax im um
L P1
LP2
D ata transfer be tween Main Memory and Sto rage Class Memory is via EAD MF (4KB or 1MB blocks)
Pa rtition Initia l Va lue
Pa rtition Initial Va lue
Figure C-5 Flash Express allocation in z/OS LPARs
Flash Express memory is a faster paging device as compared to HDD. It is not replacing memory, but replacing disks. It is suitable for workloads that can tolerate paging and will not benefit to workloads that cannot afford to page. The z/OS design for Flash Express memory does not completely remove the virtual constraints created by a paging spike in the system.z/OS paging subsystem works with a mix of internal Flash Express and external disks. Flash Express improves paging performance. Currently 1 MB large pages are not pageable. With the introduction of Flash Express, 1 MB large pages can reside on Flash and becomes then pageable. Table C-2 on page 450 introduces, for a few z/OS data types supported by Flash Express the choice criteria for data placement on Flash Express or on disk.
Table C-2 Flash Express z/OS supported data types Data type Data page placement
Pageable Link Pack Area (PLPA) VIO
At IPL/NIP time PLPA pages will be placed both on flash and disk. VIO data will always be placed on disk (First to VIO accepting data sets with any spillover flowing to non VIO data sets) If flash space is available, all virtual pages belonging to a HyperSwap Critical Address Space will be placed on flash memory. If flash space is not available, these pages will be kept in memory and only paged to disk when the system is real storage constrained and no other alternatives exist If contiguous flash space is available, pageable large pages will be preferentially written to flash. If available space exists on both flash and disk then make a selection based on response time.
HyperSwap Critical Address Space data
Pageable Large Pages All other data
450
Flash Express is used by the Auxiliary Storage Manager (ASM) in conjunction with paging data sets to satisfy page-out and page-in requests received from the Real Storage Manager (RSM). It supports 4KB and 1MB page sizes. ASM determines where to write a page based on space availability, data characteristics, and performance metrics. ASM still requires definition of a PLPA, Common, and at least one local paging data set. VIO pages are only written to DASD (persistence needed for warm starts). A new PAGESCM keyword in IEASYSxx member defines the minimum amount of flash to be reserved for paging. Value may be specified in units of MB, GB, or TB. NONE indicates do not used flash for paging. ALL (default) indicates all flash defined to the partition is available for paging. New messages issued during z/OS IPL indicate the status of SCM: IAR031I USE OF STORAGE-CLASS MEMORY FOR PAGING IS ENABLED - PAGESCM=ALL, ONLINE=00001536M or IAR032I USE OF STORAGE-CLASS MEMORY FOR PAGING IS NOT ENABLED - PAGESCM=NONE. The D ASM and D M commands are enhanced to display flash-related information/status: D ASM lists SCM status along with paging data set status. D ASM,SCM displays summary of SCM usage. D M=SCM display SCM online/offline and increment information. D M=SCM(DETAIL) displays detailed increment-level information. The CONFIG ONLINE command is enhanced to allow bringing additional SCM online: CF SCM (amount), ONLINE.
C.3 Security on Flash Express

Data stored on Flash Express is encrypted by a strong encryption symmetric key (known as the Flash encryption key / authentication key) which resides in a file on the Support Element hard disk. The firmware management of the Flash Express adapter has the ability to generate an asymmetric transport key in which the flash encryption key / authentication key is wrapped while in transit from the Support Element to the firmware management of the Flash Express adapter. The Support Element has an integrated card reader into which one smart card at a time may be inserted. When a Support Element is locked down, removing the smart card is not an option unless the person doing the removal has the physical key to the physical lock.
C.3.1 Integrated Key Controller

The Support Element (SE) initializes the environment by invoking APIs within the Integrated Key Controller (IKC). The IKC loads an applet to a smart card inserted in the integrated card reader. The smart card applet, as part of its installation, creates an RSA key pair, the private component of which never leaves the smart card, but the public key is exportable. The applet also creates two Advanced Encryption Standard (AES) symmetric keys. One of these AES keys is known as the Key-Encrypting-Key (KEK) which is retained on the smart card. The KEK may also be exported. The other AES key becomes the Flash encryption key / authentication key and is encrypted by the KEK. A buffer is allocated containing the KEK-encrypted flash encryption key / authentication key and the unique serial number of the Support Element. The buffer is padded per Public-Key
451
Cryptography Standards #1 (PKCS #1) and then encrypted by the smart card RSA public key. The encrypted content is then written to a file on the Support Element hard disk. This design defines a tight-coupling of the file on the Support Element to the smart card which ensures that any other Support Element will not be able to share the file or the smart card associated with a given SE. It guarantees that the encrypted files are unique and all such smart cards are uniquely tied to their Support Elements. All key generation, encryption, and decryption take place on the smart card. Keys are never in clear. The truly sensitive key the flash encryption key / authentication key only resides in the file on the Support Element until be served to the firmware management of the Flash Express adapter. Figure C-6 shows the cryptographic keys involved in creating this tight-coupling design.
Support Element (SE)
SE
Integrated Key Controller
Hard Disk
Flash Encryption Key / Authentication Key
Support Element Serial Number
Keys Generated in the Smart Card
AES Key-Encrypting Key
AES Flash Encryption Key / Authentication Key
RSA Public Key
RSA Private Key
Figure C-6 Integrated Key Controller
The flash encryption key / authentication key can be served to the firmware management of the Flash Express adapter either upon request from the firmware at IML time or from the Support Element as the result of a request to change or roll the key. During the alternate Support Element initialization, APIs are called to initialize the alternate smart card in it with the applet code and create the RSA public/private key pair. The API will return the public key of the smart card associated with the alternate Support Element to be used to encrypt the KEK and the Flash encryption key / authentication key from the primary Support Element, sending the resulting encrypted file to the alternate SE for redundancy.
452
C.3.2 Key Serving Topology

In a key serving topology, the Support Element (SE) is the key server and the Integrated Key Controller (IKC) is the key manager. The Support Element is connected to the firmware management of the Flash Express adapter through a secure communications line and the firmware manages the transportation of the Flash encryption key / authentication key through internal system paths. Data in the adapter cache memory is backed up by a flash-backed DRAM module which has the ability to encrypt the data with the Flash encryption key / authentication key. The firmware management of the Flash Express adapter generates its own transport RSA asymmetric key pair which is used to wrap the Flash encryption key / authentication key while in transit between the Support Element and the firmware code.
Support Element (SE)

SE Hard Disk
Firmware Public Key
Firmware RSA Key Pair
HSA
Public Key
Private Key
Integrated Key Controller
Private Key
Flash Encryption Key / Authentication Key
Firmware Management of the Flash Express Adapter Keys Generated in the Smart Card
AES Key-Encrypting Key
AES Flash Encryption Key / Authentication Key Flash Encryption Key / Authentication Key
RSA Public Key
RSA Public Key
Flash
Figure C-7 Key Serving Topology
The firmware management of the Flash Express adapter requests the flash encryption key / authentication key from the Support Element at Initial Microcode Load (IML) time. When this request arrives, the firmware public key is passed to the Support Element to be used as the transport key. The file containing the KEK-encrypted flash encryption key / authentication key and the firmware public key are passed to the IKC which sends the file contents and the public key to the smart card. The applet on the smart card decrypts the file contents and the flash encryption key / authentication key and then re-encrypts the flash encrypting key / authentication key with the firmware public key. This is then passed back to the Support Element which forwards it on to the firmware management of the Flash Express adapter code. This flow is shown at Figure C-7 on page 453.
453
C.3.3 Error recovery scenarios

Possible error scenarios are described below.
Primary Support Element failure

When the primary Support Element fails, a switch is made to the alternate Support Element which then becomes the new primary. When the former primary is brought back up, it will come up as the alternate SE. The KEK and the Flash encryption key / authentication key from the primary Support Element were already sent to the alternate SE for redundancy at initialization time.
Removal of a Smart Card

If a smart card is removed from the card reader, the card reader signals the event to the Integrated Key Controller (IKC) listening code. The IKC listener will then call the Support Element to take the appropriate action. The appropriate action may involve deleting the flash encryption key or authentication key file. In the case where the smart card is removed while the Support Element is powered off, there is no knowledge of the event. However, when the SE is powered on, notification is sent to the system administrator.
Primary Support Element failure during IML serving of the flash key
If the primary Support Element fails during the serving of the key, the alternate SE takes over as the primary and restarts the key serving operation.
Alternate Support Element failure during switch over from the primary
If the alternate Support Element fails during switch over when the primary SE fails, the key serving state is lost. When the primary comes back up, the key serving operation may be restarted.
Primary and Alternate Support Elements failure

If the primary and the alternate Support Elements fail, the key cannot be served. If the devices are still up, the key is still valid. If either or both Support Elements are recovered, the file(s) holding the flash encryption key / authentication key should still be valid (Even in a key roll case, there should be both new and current (old) keys available until the key serving operation is complete.) If both Support Elements are down and the Flash Express goes down and comes back online before the SEs become available, all data on the Flash Express is lost. Reformatting will then be necessary when the device is powered up. As long as both Flash Express devices are still powered up, it would be necessary that the customer get the primary Support Element back online as fast as possible with the flash encryption key / authentication key file and associated smart card still intact. Once that happens, the alternate SE can be brought online with a new smart card and be taken through the initialization procedure.
454
8049bibl.fm
Related publications
The publications listed in this section are considered particularly suitable for a more detailed discussion of the topics covered in this book.
IBM Redbooks
The following IBM Redbooks publications provide additional information about the topic in this document. Note that some publications referenced in this list might be available in softcopy only. ????full title???????, xxxx-xxxx ????full title???????, SG24-xxxx ????full title???????, REDP-xxxx ????full title???????, TIPS-xxxx You can search for, view, download or order these documents and other Redbooks, Redpapers, Web Docs, draft and additional materials, at the following website: ibm.com/redbooks
Other publications
These publications are also relevant as further information sources: ????full title???????, xxxx-xxxx ????full title???????, xxxx-xxxx ????full title???????, xxxx-xxxx
Online resources
These websites are also relevant as further information sources: Description1 http://????????.???.???/ Description2 http://????????.???.???/ Description3 http://????????.???.???/
Help from IBM

IBM Support and downloads
455
8049bibl.fm
ibm.com/support IBM Global Services ibm.com/services
456
8049IX.fm
Index
Numerics
1x InfiniBand 35 240V line 386 64-I/128-D KB 76 central processor (CP) 6, 357 central processor complex (CPC) 9 central storage (CS) 105 CFCC 6 channel subsystem 6, 195, 198 logical partitions 195, 198 channel-to-channel (CTC) 11 Chinese Remainder Theorem (CRT) 202, 292294 CHPID number 208 CHPID type OSM 427 Cipher Block Chaining (CBC) 200 CIU application 315316 CIU facility 312314, 328 given server 312, 329 IFLs processors 328 Permanent upgrade 319320 CoD can provide (CP) 312313 Common Cryptographic Architecture 186, 209 Common Cryptographic Architecture (CCA) 186 concurrent book add (CBA) 318 concurrent book replacement 363, 366367 Concurrent Driver Upgrade (CDU) 357358, 368 concurrent hardware upgrade 318 concurrent upgrade 17, 92, 313, 316 cooling requirements 378 Coordinated Server Time (CST) 16 Coordinated Time Network (CTN) 16 Coordinated Timing Network (CTN) 16 coprocessor 1314, 167, 193, 198, 419 coupling facility (CF) 6, 16, 35, 114115 coupling link 4, 10, 35, 163 CP 76, 93, 183184, 194, 317 conversion 9 CP Assist 8, 13, 167 CP capacity 65, 67, 317, 319 CP Cryptographic Assist Facility (CPACF) 87 CP rule 67 CPACF 194 cryptographic capabilities 13 feature code 194 PU design 87 CPC cage 7 CPCs 19, 426 CPE 313, 316 CPM 313 CPs 196, 199, 313314, 317 capacity identifier 317, 324 concurrent and temporary activation 321322 Crypto enablement 193 Crypto Express coprocessor 186, 205 coprocessor feature 207 feature 185186
A
activated capacity 313 active core 8 Active Energy Manager (AEM) 390, 392, 395 Advanced Encryption Standard (AES) 13, 184, 209 application preservation 103 application program interface (API) 320 application program interface (API) 320
B
billable capacity 313 BladeCenter 385386, 425 BladeCenter chassis 17 book ring topology 39, 82 upgrade 61 Broadband RSF 327 Bulk Power Hub (BPH) 12 Bulk Power Regulator (BPR) 377
C
cache level 79 cage I/O cage 198, 326 capacity 313 Capacity Backup 65, 312, 316 See CBU Capacity for Planned Event See CPE Capacity for Planned Event (CPE) 312313 Capacity On Demand (COD) 67, 312 Capacity on Demand (CoD) 313, 322323, 325326 Capacity Provisioning Manager (CPM) 313 capacity setting 65, 313314, 317 full capacity 67 capacity token 321 CBU 313, 316, 321322 contract 322 conversions 67 feature 322 record 65 test 66 5-year CBU contract 66 CBU for CP 65 CBU for IFL 65 CBU test 322
457
8049IX.fm
tamper-resistant feature 183 Crypto Express2 11, 13, 35, 185, 196198, 200, 202 accelerator 14, 200, 202, 208 coprocessor 14, 199, 201, 207208 Crypto Express3 187, 194 operational keys 186 cryptographic asynchronous functions 184 domain 196, 198, 201202 feature codes 193 Cryptographic Function 8, 13, 167, 183184 cryptographic function security-relevant portion 201 cryptography Secure Hash Algorithm (SHA) 13 CSS definition 4 CSSs 4, 196, 198 Customer Engineer (CE) 363 Customer Initiated Upgrade (CIU) 312 activation 331 ordering 330 customer profile 314
F
fanout 35 FC 0864 167, 194, 197 FC 2000 377 FC 3863 167 feature code FC 28xx 367 flexible memory option 360 feature code (FC) 193, 376, 381383, 424 Federal Information Processing Standard (FIPS) 12, 183, 194 FICON channel 11, 35 FICON Express 35 channel 35 FICON Express2 11, 35 FICON Express4 10km LX 151 FICON Express8 11 feature 11 field replaceable unit (FRU) 358 FIPS 140-2 Level 4 183 five-model structure 6 flexible memory option 54, 315, 318, 358 flexible memory option 360, 362 full capacity CP feature 314
D
Data Encryption Standard (DES) 13, 183185, 204 data integrity 184 decimal floating point (DFP) 8 dial modem 327 Digital Signature Verify (CSFNDFV) 186 direct memory access (DMA) 12 double-key DES 184185 double-key MAC 184 dynamic coupling facility dispatching 95 dynamic SAP sparing and reassignment 103 dynamic storage reconfiguration (DSR) 116, 326
G
Gbps 10 granular capacity 65, 67, 93 granularity of both (GB) 326
H
hardware management 17, 376, 425 console 12, 17, 187, 319, 331, 361 hardware management console (HMC) 121 Hardware Security Module (HSM) 200 hardware system area See HSA hardware system area (HSA) 187 HCA2-C fanout 35 HCA2-O LR 35 high water mark 314 HMC 331, 426 host channel adapter (HCA) 9, 21 HSA 3
E
EBA process 361 Model Conversion panel 362 Elliptic Curve Cryptography 186 Digital Signature Algorithm 186 emergency power-off 377 enhanced book availability (EBA) 21, 323, 325, 363 prepare 363 enhanced driver maintenance (EDM) 21 ESA/390 Architecture mode 113 ESA/390 TPF mode 114 ESCON channel 35 estimated capacity (EC) 7 Europay Mastercard VISA (EMV) 2000 185 expanded storage 106 extended translation facility 90 external time source (ETS) 16
I
I/O cage, I/O slot 326 card 323 I/O cage 4, 1011, 35, 199, 315, 361362, 376, 381 installed I/O domains 362 I/O card 9, 312, 359, 369 I/O connectivity 361 I/O domain 362 I/O drawer 312, 318, 359, 376 I/O feature 5, 10 I/O unit 376
458
8049IX.fm
IBM representative 312 IBM Systems Director Active Energy Manager 390 ICF 76 CBU 65 IEC 60793 236 IEDN 12, 17 IEEE Floating Point 90 IFB cable 76 I/O interface 76 IFL 6, 76, 318 IFLs, SAPs, book, memory (IBM) 311 InfiniBand coupling links LR 164 InfiniBand coupling 35 link 10 links 164 InfiniBand Double Data Rate (IB-DDR) 10 initial configuration 318, 323, 327, 377 line-cord pairs 377 initial machine load (IML) 368 initial order (I/O) 167, 196 initial program load (IPL) 327 installed book 4, 318319, 361 additional memory capacity 323 PU count 319 installed record 314 instruction decoding 90 fetching 90 grouping 90 set extensions 91 Instruction fetch and branch (IFB) 358 Integrated Cluster Bus-4 (ICB-4) 15 Integrated Cryptographic Service Facility (ICSF) 13, 184, 188, 204, 290 Integrated Facility for Linux 6 See IFL Intelligent resource director (IRD) 117 inter-book communication 4 Internal Battery Feature (IBF) 7, 69, 76, 377378 Internal Coupling Channels 15 internal coupling facility See ICF internal coupling facility (ICF) 8, 318 internal QDIO (IQDIO) 12 InterSystem Channel-3 15 Intraensemble data network 12, 17 intranode management network 12, 17 IP address 370, 404, 426 ISC-3 link 35 ISC-3 coupling link 164 ISO 16609 CBC Mode 186
L
Large Systems Performance Reference (LSPR) 3 LICCC 318 I/O 318 memory 318 processors 318 Licensed Internal Code (LIC) 5, 16, 193, 318, 323, 359 See also LICCC line-cords 376 Linux 6, 8, 13, 184, 209210, 327 mode 114 Linux-only mode 111 loading of initial ATM keys 185 Local Area Network (LAN) 12, 193, 204, 426 logical partition 4, 12, 107, 195, 198, 201, 317318, 324, 326, 357 dynamic add and delete 111 mode 110 relative priority 361 reserved storage 326 logical processor 4, 306, 324 Long Reach (LR) 35 LPAR single storage pool 106 LPAR mode 111 LSPR 3
M
M80 model 381 machine type 6 master key entry 201 maximum number 196, 198 MBA fanout card 9 Mbps 193 MCI 314, 317, 323 Capacity on Demand 314 list of identifiers 64 model upgrade 317 updated 323 MCM 3, 6, 8 memory allocation 104 card 318, 323, 325 physical 360, 367 size 76 memory hierarchy 23 performance sensitive area 24 memory upgrade 53, 313, 315 MES order 35 MES upgrade 322 message authentication code (MAC) 200 message authentication code (MAC) 184, 200 Message-Security Assist (MSA) 13, 184, 194 MIDAW facility 11 MIF image ID (MIF ID) 201 miscellaneous equipment specification (MES) 312, 322 model capacity 65, 67, 314315
K
key exchange 186 kW 387
Index
459
8049IX.fm
identifier 6465, 68, 317 model M15 317 model M32 50, 317, 324 model M49 50, 317 model M80 50, 318 Model Permanent Capacity Identifier (MPCI) 314 model S08 76, 324 Model Temporary Capacity Identifier (MTCI) 314 model upgrade 317 Modulus Exponent (ME) 202, 292294 MPCI 314 MSU value 22 MTCI 314 multi-chip module 3, 6 multimode fiber 236 multiple platform 393
N
network security considerations 233 Network Time Protocol (NTP) 16 NTP client 16 NTP server 16
O
On/Off Capacity 67, 312, 316 On/Off CoD 15, 6768, 312, 314, 316, 322 activation 321 CP6 temporary CPs 68 granular capacity 67 offering 320 rules 68 Open Systems Adapter (OSA) 11 operating system 6, 18, 305, 313, 316, 318 optionally assignable SAPs 101 OSA-Express 11 OSA-Express2 10 Gb Ethernet LR 35 1000BASE-T Ethernet 35 OSA-Express3 1000BASE-T Ethernet 35 OSA-Express3 feature 12 data router function present 12 oscillator 39 out-of-order (OOO) 9
P
parallel access volume (PAV) 4 Parallel Sysplex 5, 307 certain coupling link configurations 12 system images 117 payment card industry (PCI) 187 PCHID 195, 198, 209 PCI Cryptographic Accelerator (PCICA) 195, 198 PCI Express adapter 35 cryptographic adapter 167, 209 PCICC 195, 198
PCI-e cryptographic adapter level 202 number 196, 199 PCI-X cryptographic adapter 35, 193198 cryptographic coprocessor 194, 197 Peripheral Component Interconnect Express (PCIE) 183 permanent capacity 314 permanent entitlement record (PER) 314 permanent upgrade 314315 retrieve and apply data 332 personal identification number (PIN) 193, 200, 208 physical memory 9, 360, 367 PKA Encrypt 202 PKA Key Import (CSNDPKI) 185 PKA Key Token Change (CSNDKTC) 186 plan-ahead memory 318, 325 capacity 55 planned event 312314 capacity 312, 316 point of sale (POS) 185 point unit (PU) 3, 316, 318, 360 port J07 427 power consumption 390391 power distribution unit (PDU) 386 power estimation tool 390 POWER7 blade system support 19 power-on reset (POR) 196, 199, 362 PR/SM 107 primary HMC 370, 426 explicit action 424 policy information 426 processing unit (PU) 6, 9, 76, 102, 318, 329 characterization 102 concurrent conversion 318 conversion 318 feature code 6 maximum number 6 pool 92 sparing 92 type 111, 318, 367 Processor Resource/Systems Manager (PR/SM) 4, 201 processor unit (PU) 3 production workload 322 Provide Cryptographic Key Management Operation (PCKMO) 188, 194 Pseudo Random Number Generation (PRNG) 184 pseudorandom number generator (PRNG) 184185, 209 PU chip 8, 42 PU type 316, 367 public key algorithm 185186, 200, 204 decrypt 209 encrypt 185, 209 public key algorithm (PKA) 201 Pulse per Second (PPS) 17 purchased capacity 314 PUs 3, 317318, 360
460
8049IX.fm
Q
queued direct input/output (QDIO) 12
R
Redbooks website 455 Contact us xxii reducing all sources (RAS) 5, 12 redundant array of independent drives (RAID) 359 redundant array of independent memory (RAIM) 3, 9, 359 redundant I/O 361 redundant I/O interconnect (RII) 9, 21, 362 relative nest intensity (RNI) 24 reliability, availability, serviceability (RAS) 1920 remaining book 360 sufficient dormant resources 360 sufficient inactive physical resources 362 Remote Support Facility (RSF) 313, 319320, 330331 replacement capacity 313314 reserved storage 115 Resource Access Control Facility (RACF) 188 Resource Link 313314, 329 CIU application 316 CIU facility 319 machine profile 331 Rivest-Shamir-Adelman (RSA) 185, 200, 202, 210
subcapacity setting 4 subchannel 357 Summary 328 superscalar 86 superscalar processor 86 Support Element (SE) 7, 314, 331, 362 Change LPAR Cryptographic Controls task 196, 199 logical partition 207 support element (SE) 17, 76, 196, 199, 202, 368, 395, 426 SUSE Linux Enterprise Server (SLES) 116 system activity display (SAD) 390 system assist processor variable number 8 system assist processor (SAP) 6, 9, 317 System Management Facilities (SMF) 204 System Storage Interoperation Center (SSIC) 237 system upgrade 312 System z 3, 32, 35, 184, 311, 319 distribution 19 hardware platform management 19 High Performance FICON 11 New Application License Charge 307 UDX toolkit 186 System z BladeCenter Extension (zBX) 1718
T
target configuration 318 TB 1, 9 temporary capacity 6768, 314 CP count 68 temporary entitlement record (TER) 314 temporary upgrade 312 time synchronization 16 TKE 201 additional smart cards 193 Smart Card Reader 193 workstation 1415, 21, 193, 203 workstation feature 203 Top of Rack (TOR) switches 370 total number 6, 65, 319, 362 Transaction Processing Facility (TPF) 327 translation look-aside buffer (TLB) 90 Transport Layer Security (TLS) 14, 183 triple-key DES 185 Trusted Key Entry (TKE) 14, 193, 203
S
SAP 6, 101 concurrent book replacement 367 number of 76, 318, 331 SC chip 8, 45 SE 426 secondary approval 314 Secure Hash Algorithm (SHA) 13 Secure Sockets Layer (SSL) 14, 183, 185, 194, 202, 208 Server Time Protocol (STP) 1516, 165 SHA-1 184 SHA-1 and SHA-256 184 SHA-256 184 single I/O path 365 rule 367 single-key MAC 184 small form factor pluggable (SFP) 232 soft capping 306 specialty engine 4, 67, 315, 361 SSL/TLS 14, 183 staged CoD records 15 staged record 314 stand-alone z196 ensemble node 427 Standard SAP 76 storage area network (SAN) 11 store system information (STSI) instruction 323 STP-only CTN 17 subcapacity 314 subcapacity model 317 subcapacity models 317
U
unassigned IFL 6364 unassigned PUs 323 Unified Resource Manager 2, 12, 370 unplanned upgrades 320 unused PUs 318, 322, 360 upgrade 62 for I/O 326 for memory 325 for processors 323 permanent upgrade 328
Index
461
8049IX.fm
user ID 329 User-Defined Extension (UDX) 183, 186, 202, 208
V
virtual server 12, 18, 427 VPD 315
W
wild branch 89 Workload License Charge (WLC) 305, 329 CIU 325
Z
z/Architecture 6, 194 z/OS 290, 313 Capacity Provisioning Manager 15 z/OS logical partition EBA function 368 z/TPF 19 z/VM V5R4 4, 103 z10 EC 7 z10 server 10 zAAP 61, 76, 95, 361, 367 CBU 65 LPAR definitions 96 zBX 17 zBX Model 003 upgrade 327 zBX Rack-B 228 zEnterprise 196 12, 4, 6, 375 zEnterprise BladeCenter Extension (ZBX) 2, 241, 312, 322, 358, 375, 385, 427 zEnterprise System 2, 370 environmental requirements 375 zIIP 6, 61, 76, 318319
462
To determine the spine width of a book, you divide the paper PPI into the number of pages in the book. An example is a 250 page book using Plainfield opaque 50# smooth which has a PPI of 526. Divided 250 by 526 which equals a spine width of .4752". In this case, you would use the .5 spine. Now select the Spine width for the book and hide the others: Special>Conditional Text>Show/Hide>SpineSize(-->Hide:)>Set . Move the changed Conditional text settings to all files in your book by opening the book file with the spine.fm still open and File>Import>Formats the
Conditional Text Settings (ONLY!) to the book files.

8049spine.fm
463
(1.5 spine) 1.5<-> 1.998 789 <->1051 pages
(1.0 spine) 0.875<->1.498 460 <-> 788 pages
(0.5 spine) 0.475<->0.873 250 <-> 459 pages
(0.2spine) 0.17<->0.473 90<->249 pages
(0.1spine) 0.1<->0.169 53<->89 pages
To determine the spine width of a book, you divide the paper PPI into the number of pages in the book. An example is a 250 page book using Plainfield opaque 50# smooth which has a PPI of 526. Divided 250 by 526 which equals a spine width of .4752". In this case, you would use the .5 spine. Now select the Spine width for the book and hide the others: Special>Conditional Text>Show/Hide>SpineSize(-->Hide:)>Set . Move the changed Conditional text settings to all files in your book by opening the book file with the spine.fm still open and File>Import>Formats the
Conditional Text Settings (ONLY!) to the book files.

8049spine.fm
464
(2.5 spine) 2.5<->nnn.n 1315<-> nnnn pages
(2.0 spine) 2.0 <-> 2.498 1052 <-> 1314 pages
Back cover
Describes the zEnterprise System and related features and functions Discusses hardware and software capabilities Explains virtualizing and managing the infrastructure for complex applications
This IBM Redbooks publication discusses the new IBM zEnterprise System, which consists of the IBM zEnterprise EC12 (zEC12), an updated IBM zEnterprise Unified Resource Manager, and the IBM zEnterprise BladeCenter Extension (zBX) Model 003. The zEC12 is designed with improved scalability, performance, security, resiliency, availability, and virtualization. The superscalar design allows the zEC12 to deliver a record level of capacity over the prior System z servers. It is powered by 120 of the worlds most powerful microprocessors running at 5.5 GHz and is capable of executing more than 75,000 millions of instructions per second (MIPS). The zEC12 Model HA1 is estimated to provide up to 50% more total system capacity than the z196 Model M80. The zBX Model 003 infrastructure works with the zEC12 to enhance System z virtualization and management through an integrated hardware platform that spans mainframe, POWER7, and System x technologies. Through the Unified Resource Manager, the zEnterprise System is managed as a single pool of resources, integrating system and workload management across the environment. This book provides information about the zEnterprise System and its functions, features, and associated software support. Greater detail is offered in areas relevant to technical planning. This book is intended for systems engineers, consultants, planners, and anyone wanting to understand the zEnterprise System functions and plan for their usage. It is not intended as an introduction to mainframes. Readers are expected to be generally familiar with existing IBM System z technology and terminology.
INTERNATIONAL TECHNICAL SUPPORT ORGANIZATION
BUILDING TECHNICAL INFORMATION BASED ON PRACTICAL EXPERIENCE

IBM Redbooks are developed by the IBM International Technical Support Organization. Experts from IBM, Customers and Partners from around the world create timely technical information based on realistic scenarios. Specific recommendations are provided to help you implement IT solutions more effectively in your environment.
For more information: ibm.com/redbooks

SG24-8049-00 ISBN

IBM EC12 sg248049

Uploaded by

Copyright:

Available Formats

IBM EC12 sg248049

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

IBM EC12 sg248049

Uploaded by

Copyright:

Available Formats

Front cover

Draft Document for Review August 29, 2012 10:29 am SG24-8049-00

IBM zEnterprise EC12 Technical Guide

Draft Document for Review August 29, 2012 10:28 am

Draft Document for Review August 29, 2012 10:28 am

Draft Document for Review August 29, 2012 10:28 am

Draft Document for Review August 29, 2012 10:28 am

IBM zEnterprise EC12 Technical Guide

Draft Document for Review August 29, 2012 10:28 am

Draft Document for Review August 29, 2012 10:28 am

IBM zEnterprise EC12 Technical Guide

Draft Document for Review August 29, 2012 10:28 am

Draft Document for Review August 29, 2012 10:28 am

IBM zEnterprise EC12 Technical Guide

Draft Document for Review August 29, 2012 10:28 am

Draft Document for Review August 29, 2012 10:28 am

Draft Document for Review August 29, 2012 10:28 am

Draft Document for Review August 29, 2012 10:28 am

Draft Document for Review August 29, 2012 10:28 am

Draft Document for Review August 29, 2012 10:28 am

IBM zEnterprise EC12 Technical Guide

Draft Document for Review August 29, 2012 10:28 am

Draft Document for Review August 29, 2012 10:28 am

IBM zEnterprise EC12 Technical Guide

Draft Document for Review August 29, 2012 10:28 am

Draft Document for Review August 29, 2012 10:28 am

IBM zEnterprise EC12 Technical Guide

Draft Document for Review August 29, 2012 10:28 am

The team who wrote this book

Copyright IBM Corp. 2010. All rights reserved.

Draft Document for Review August 29, 2012 10:28 am

Draft Document for Review August 29, 2012 10:28 am

Draft Document for Review August 29, 2012 10:28 am

Now you can become a published author, too!

Stay connected to IBM Redbooks

IBM zEnterprise EC12 Technical Guide

Draft Document for Review August 29, 2012 10:28 am

Introducing the IBM zEnterprise EC12

Copyright IBM Corp. 2010. All rights reserved.

Draft Document for Review August 29, 2012 10:28 am

1.1 IBM zEnterprise EC12 elements

zEnterprise BladeCenter Extension

Figure 1-1 Elements of the zEC12 ensemble node

Draft Document for Review August 29, 2012 10:28 am

1.2 zEC12 highlights

Processor and memory

Capacity and performance

CMOS 13S is a 32 nanometer CMOS logic fabrication process.

Chapter 1. Introducing the IBM zEnterprise EC12

Draft Document for Review August 29, 2012 10:28 am

I/O subsystem and I/O features

Increased flexibility with z/VM-mode logical partition

IBM zEnterprise EC12 Technical Guide

Draft Document for Review August 29, 2012 10:28 am

zAware-mode logical partition

Reliability, availability, and serviceability

1.2.1 Flash Express

Chapter 1. Introducing the IBM zEnterprise EC12

Draft Document for Review August 29, 2012 10:28 am