z/OS V2R1 Communications Server

Performance Update

David Herr – dherr@us.ibm.com

IBM Raleigh, NC

Thursday, August 15th, 9:30am

Session: 1363313633
 Trademarks
The following are trademarks of the International Business Machines Corporation in the United States and/or other countries.

AIX* DB2* HiperSockets* MQSeries* PowerHA* RMF System z* zEnterprise* z/VM*

BladeCenter* DFSMS HyperSwap NetView* PR/SM Smarter Planet* System z10* z10 z/VSE*
CICS* EASY Tier IMS OMEGAMON* PureSystems Storwize* Tivoli* z10 EC
Cognos* FICON* InfiniBand* Parallel Sysplex* Rational* System Storage* WebSphere* z/OS*
DataPower* GDPS* Lotus* POWER7* RACF* System x* XIV*

* Registered trademarks of IBM Corporation

The following are trademarks or registered trademarks of other companies.
Adobe, the Adobe logo, PostScript, and the PostScript logo are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States, and/or other countries.
Cell Broadband Engine is a trademark of Sony Computer Entertainment, Inc. in the United States, other countries, or both and is used under license therefrom.
Intel, Intel logo, Intel Inside, Intel Inside logo, Intel Centrino, Intel Centrino logo, Celeron, Intel Xeon, Intel SpeedStep, Itanium, and Pentium are trademarks or registered trademarks of Intel
Corporation or its subsidiaries in the United States and other countries.
IT Infrastructure Library is a registered trademark of the Central Computer and Telecommunications Agency which is now part of the Office of Government Commerce.
ITIL is a registered trademark, and a registered community trademark of the Office of Government Commerce, and is registered in the U.S. Patent and Trademark Office.
Java and all Java based trademarks and logos are trademarks or registered trademarks of Oracle and/or its affiliates.
Linear Tape-Open, LTO, the LTO Logo, Ultrium, and the Ultrium logo are trademarks of HP, IBM Corp. and Quantum in the U.S. and
Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both.
Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both.
OpenStack is a trademark of OpenStack LLC. The OpenStack trademark policy is available on the OpenStack website.
TEALEAF is a registered trademark of Tealeaf, an IBM Company.
Windows Server and the Windows logo are trademarks of the Microsoft group of countries.
Worklight is a trademark or registered trademark of Worklight, an IBM Company.
UNIX is a registered trademark of The Open Group in the United States and other countries.
* Other product and service names might be trademarks of IBM or other companies.
Notes:
Performance is in Internal Throughput Rate (ITR) ratio based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput that any user will experience will vary
depending upon considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that
an individual user will achieve throughput improvements equivalent to the performance ratios stated here.
IBM hardware products are manufactured from new parts, or new and serviceable used parts. Regardless, our warranty terms apply.
All customer examples cited or described in this presentation are presented as illustrations of the manner in which some customers have used IBM products and the results they may have achieved. Actual environmental
costs and performance characteristics will vary depending on individual customer configurations and conditions.
This publication was produced in the United States. IBM may not offer the products, services or features discussed in this document in other countries, and the information may be subject to change without notice. Consult
your local IBM business contact for information on the product or services available in your area.
All statements regarding IBM's future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only.
Information about non-IBM products is obtained from the manufacturers of those products or their published announcements. IBM has not tested those products and cannot confirm the performance, compatibility, or any
other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products.
Prices subject to change without notice. Contact your IBM representative or Business Partner for the most current pricing in your geography.
This information provides only general descriptions of the types and portions of workloads that are eligible for execution on Specialty Engines (e.g, zIIPs, zAAPs, and IFLs) ("SEs"). IBM authorizes customers to use IBM
SE only to execute the processing of Eligible Workloads of specific Programs expressly authorized by IBM as specified in the “Authorized Use Table for IBM Machines” provided at
www.ibm.com/systems/support/machine_warranties/machine_code/aut.html (“AUT”). No other workload processing is authorized for execution on an SE. IBM offers SE at a lower price than General Processors/Central
Processors because customers are authorized to use SEs only to process certain types and/or amounts of workloads as specified by IBM in the AUT.
Page 2
© 2013 SHARE and IBM Corporation
 Agenda

 V2R1 Performance Enhancements

 Optimizing inbound communications using
OSA-Express
 Optimizing outbound communications
using OSA-Express
 OSA-Express4
 z/OS Communications Server Performance
Summaries

Disclaimer: All statements regarding IBM future direction or intent, including current product plans, are subject to
change or withdrawal without notice and represent goals and objectives only. All information is provided for
informational purposes only, on an “as is” basis, without warranty of any kind.
Page 3
© 2013 SHARE and IBM Corporation
 V2R1 Performance Enhancements

Page 4
© 2013 SHARE and IBM Corporation
 V2R1
Shared Memory Communications – Remote (SMC-R)
Both TCP and SMC-R “connections”
SMC-R Background remain active

z/OS System A z/OS System B

Middleware/Application Middleware/Application

Sockets Sockets

TCP TCP
SMC-R SMC-R
RMBe IP IP RMBe
Interface Interface
ROCE OSA OSA ROCE

TCP connection establishment over IP

TCP syn flows (with TCP Options

indicating SMC-R capability) App data
RDMA Network RoCE
App data
IP Network (Ethernet)

Dynamic (in-line) negotiation for SMC-R is initiated by presence of TCP Options

TCP connection transitions to SMC-R allowing application data to be exchanged using RDMA

Page 5
© 2013 SHARE and IBM Corporation
 V2R1
SMC-R - RDMA

 Key attributes of RDMA

 Enables a host to read or write directly from/to a remote host’s memory
without involving the remote host’s CPU
 By registering specific memory for RDMA partner use
 Interrupts still required for notification (i.e. CPU cycles are not
completely eliminated)
 Reduced networking stack overhead by using streamlined, low level, RMDA
interfaces
 Key requirements:
 A reliable “lossless” network fabric (LAN for layer 2 data center network
distance)
 An RDMA capable NIC (RNIC) and RDMA capable switched fabric
(switches)

Page 6
© 2013 SHARE and IBM Corporation
 V2R1
SMC-R - Solution

 Shared Memory Communications over RDMA (SMC-R) is a protocol that

allows TCP sockets applications to transparently exploit RDMA (RoCE)
 SMC-R is a “hybrid” solution that:
 Uses TCP connection (3-way handshake) to establish SMC-R connection
 Each TCP end point exchanges TCP options that indicate whether it
supports the SMC-R protocol
 SMC-R “rendezvous” (RDMA attributes) information is then exchanged
within the TCP data stream (similar to SSL handshake)
 Socket application data is exchanged via RDMA (write operations)
 TCP connection remains active (controls SMC-R connection)
 This model preserves many critical existing operational and network
management features of TCP/IP

Page 7
© 2013 SHARE and IBM Corporation
 V2R1
SMC-R – Role of the RMBe (buffer size)
 The RMBe is a slot in the RMB buffer for a specific TCP connection
Based on TCPRCVBufrsize – NOT equal to
Can be controlled by application using setsockopt() SO_RCVBUF
5 sizes – 32K, 64K, 128K, 256K and 1024K (1MB)
Depending on the workload, a larger RMBe can improve performance
Streaming (bulk) workloads
Less wrapping of the RMBe = less RDMA writes
Less frequent “acknowledgement” interrupts to sending side
Less write() blocks on sending side
RMB – 1MB

Appl

Space available –
Keep writing!
(pipe stays full)
Data waiting to be received
RMBe – TCP connection
TCP/IP Available
SMC link

Wrap point

Page 8
© 2013 SHARE and IBM Corporation
 V2R1
SMC-R – Micro benchmark performance results

 Response time/Throughput and CPU improvements

 Workload:
 Using AWM (Application Workload Modeler) to model “socket to socket”
performance using SMC-R
 AWM very lightweight - contains no application/business logic
 Stresses and measures the networking infrastructure
 Real workload benefits will be smaller than the improvements
seen in AWM benchmarks!
 MTU: RoCE (1K and 2K) OSA (1500 and 8000)
 Large Send enabled for some of the TCP/IP streaming runs
 RR1(1/1): Single interactive session with 1 byte request and 1 byte reply
 RR10: 10 concurrent connections with various message sizes
 STR1(1/20M): Single Streaming session with 1 byte request (Client) and
20,000,000 bytes reply (Server)
 Used large RMBs – 1MB

Page 9
© 2013 SHARE and IBM Corporation
 V2R1
SMC-R – Micro benchmark performance results
SMC-R(RoCE) vs. TCP/IP(OSA)
Performance Summary
177,972 trans/secs (R/R) Request/Response micro-benchmark 15.9Gb/sec
9.1Gb/sec 14.6Gb/sec
875
716.70
675.60 15.7Gb/sec
675
%(Relative toTCP/IP)

539.61

475 Raw Tput

289.93 CPU-Server
281.35 267.80
275 CPU-Client
108.19
Resp Time
8
.9
68

73

75
14
7.

5.

73

75
4

-1 1

39
-2 7
.5
.3

51
0

-3 9

78
8
2.

-5 8

-5 8
8.
2.
-8

2
-9

7.
-125

9.

63
35

77

5
6.
-1

3.
39
37

38
-2

1.
5.

3.
-2

2.
-4
4.

3.

-5
4.
7.

8.

-7
-7

-7

-8
-8

-8
-325

)
)

6k

2k
1)

1k

2k

4k

8k
1/

/1

/3
k/

k/

k/

k/
1(

6k

2k
(1

(2

(4

(8
R

10

10

10

10

(1

(3
R

10

10
R

R
R

R
Latency 28 mics
R

R
June 4, 2013
(full roundtrip) AWM IPv4 R/R Workload Client, Server : 4 CPs 2827-791 (zEC12 GA2)
Interfaces: 10GbE RoCE Express and 10GbE OSA Expess5

Significant Latency reduction across all data sizes (52-88%) Note: vs typical OSA customer configuration
Reduced CPU cost as payload increases (up to 56% CPU savings) MTU (1500), Large Send disabled
Impressive throughput gains across all data sizes (Up to +717%) RoCE MTU: 1K
Page 10
© 2013 SHARE and IBM Corporation
 V2R1
SMC-R – Micro benchmark performance results
z/OS V2R1 SMC-R vs TCP/IP
1MB RMBs
Streaming Data Performance Summary (AWM)
100
8.8Gb/sec

2
5

.8
8.9Gb/sec
.7

68
65

.7
%(Relative to TCP/IP)

58
50 29

8
.5
5
.
21

.1

20
Raw Tput

16
CPU-Server
0 CPU-Client
Resp Time

-1

-1
-1

1.

5.
5.

73

1
56

-50 -3
-3

-4 4.26

-4
6

-4

-41.0
9

-43.9
.9

0. 1
.6

3. 5
9

76
7

88
-6

-6
-63.7
-63.9

-6
-6

-65.2
-65.6

4. 7
4.

5. 3
6. 7

28
84

74
23

-100
STR1(1/20M) STR3(1/20M) STR1(1/20M) STR3(1/20M) STR1(1/20M) STR3(1/20M)
MTU 2K/1500 1K/1500 2K/8000-LS May 29,2013
Client, Server: 2827-791 2CPs LPARs
Interfaces: 10GbE RoCE Express and 10GbE OSA Express 5

Notes:
• Significant throughput benefits and CPU reduction benefits
• Up to 69% throuput improvement
Saturation
• Up to 66% reduction in CPU costs reached
• 2K RoCE MTU does yield throughput advantages
• LS – Large Send enabled (Segmentation offload)
Page 11
© 2013 SHARE and IBM Corporation
 V2R1
SMC-R – Micro benchmark performance results

• Summary –
– Network latency for z/OS TCP/IP based OLTP
(request/response) workloads reduced by up to 80%*
• Networking related CPU consumption reduction for z/OS TCP/IP
based OLTP (request/response) workloads increases as payload
size increases
– Networking related CPU consumption for z/OS TCP/IP based
workloads with streaming data patterns reduced by up to 60%
with a network throughput increase of up to 60%**
– CPU consumption can be further optimized by using larger
RMBe sizes
• Less data consumed processing
• Less data wrapping
• Less data queuing

* Based on benchmarks of modeled z/OS TCP sockets based workloads with request/response traffic patterns using SMC-R vs. TCP/IP. The
actual response times and CPU savings any user will experience will vary.
** Based on benchmarks of modeled z/OS TCP sockets based workloads with streaming data patterns using SMC-R vs. TCP/IP. The benefits
any user will experience will vary
Page 12
© 2013 SHARE and IBM Corporation
 V2R1
SMC-R – FTP performance summary
zEC12 V2R1 SMC vs. OSD Performance Summary
FTP Performance
AWM FTP client
73

06
0.

1.
%(Relative to OSD)

Raw Tput

83
-25

5.
86

-1
CPU-Client
0.
-2

CPU-Server

-50
49
18

7.
8.

-4
-4

-75
FTP1(1200M) FTP3(1200M)

FTP binary PUTs to z/OS FTP server, 1 and 3 sessions, transferring 1200
MB data
OSD – OSA Express4 10Gb interface
Reading from and writing to DASD datasets – Limits throughput
The performance measurements discussed in this document were collected using a dedicated system environment. The results
obtained in other configurations or operating system environments may vary significantly depending upon environments used.

Page 13
© 2013 SHARE and IBM Corporation
SMC-R - WebSphere MQ for z/OS performance V2R1
improvement

 Latency improvements
 Workload
 Measurements using WebSphere MQ V7.1.0
 MQ between 2 LPARs on zEC12 machine (10 processors each)
 On each LPAR, a queue manager was started and configured with 50
outbound sender channels and 50 inbound receiver channels, with
default options for the channel definitions (100 TCP connections)
 Each configuration was run with message sizes of 2KB, 32KB and
64KB where all messages were non-persistent
 Results were consistent across all three message sizes

Page 14
© 2013 SHARE and IBM Corporation
SMC-R - WebSphere MQ for z/OS performance V2R1

improvement
 Latency improvements 2k, 32k and 64k message sizes
50 TCP connections each way

WebSphere MQ for z/OS using SMC-R

z/OS SYSB
z/OS SYSA SMC-R

MQ messages
WebSphere WebSphere MQ
MQ
RoCE

WebSphere MQ for z/OS realizes up to a 3x increase in messages per

second it can deliver across z/OS systems when using SMC-R vs standard
TCP/IP *

Based on internal IBM benchmarks using a modeled WebSphere MQ for z/OS workload driving non-persistent messages across z/OS
systems in a request/response pattern. The benchmarks included various data sizes and number of channel pairs. The actual throughput
and CPU savings users will experience may vary based on the user workload and configuration.

Page 15
© 2013 SHARE and IBM Corporation
 V2R1
SMC-R – CICS performance improvement
• Response time and CPU utilization
improvements
• Workload - Each transaction
– Makes 5 DPL (Distributed Program Link) IPIC - IP Interconnectivity
requests over an IPIC connection
• Introduced in CICS TS
– Sends 32K container on each request 3.2/TG 7.1
•TCP/IP based
– Server program Receives the data and communications
•Alternative to LU6.2/SNA for
Send back 32K Distributed program calls
– Receives back a 32K container for each
request

Note: Results based on internal IBM benchmarks using a modeled CICS workload driving a CICS transaction that performs 5 DPL calls to a
CICS region on a remote z/OS system, using 32K input/output containers. Response times and CPU savings measured on z/OS system initiating
the DPL calls. The actual response times and CPU savings any user will experience will vary.

Page 16
© 2013 SHARE and IBM Corporation
 V2R1
SMC-R – CICS performance improvement
CICS DPL benchmark over IPIC
zOS z/OS V2R1 SYSA z/OS V2R1 SYSB

CICS A CICS B
5 DPL calls per CICS
TPNS (IBM transaction
Teleprocessing CICS transaction CICS mirror transaction
32K request/response
Network Simulator) Program invokes 5 (DPL target program)
Using Containers
DPL calls
per transaction
1) IPIC over

OSA ROCE

OSA ROCE
Multiple
SMC-R
3270
Sessions 2) IPIC
over
TCP/IP

• Benchmarks run on z/OS V2R1 with latest zEC12 and new 10GbE RoCE Express
feature
– Compared use of SMC-R (10GbE RoCE Express) vs standard TCP/IP (10GbE OSA
Express4S) with CICS IPIC communications for DPL (Distributed Program Link)
processing
– Up to 48% improvement in CICS transaction response time as measured on CICS
system issuing the DPL calls (CICS A)
– Up to 10% decrease in overall z/OS CPU consumption on CICS system issuing the
DPL calls (SYSA)
Page 17
© 2013 SHARE and IBM Corporation
SMC-R – Websphere to DB2 communications performance
improvement V2R1

• Response time improvements

WebSphere to DB2 communications using SMC-R

z/OS SYSB
z/OS SYSA SMC-R
Linux on x

TCP/IP
JDBC/DRDA
HTTP/REST 3 per HTTP
40 Concurrent WAS Connection
Workload Client Simulator TCP/IP Connections Liberty
(JIBE) TradeLite DB2
RoCE

40% reduction in overall

Transaction response time! –
As seen from client’s perspective

Small data sizes ~ 100 bytes

Based on projections and measurements completed in a controlled environment. Results may vary by customer based on
individual workload, configuration and software levels.
Page 18
© 2013 SHARE and IBM Corporation
 V2R1
TCP/IP Enhanced Fast Path Sockets
TCP/IP sockets (normal path) TCP/IP fast path sockets (Pre-V2R1)
z/OS
z/OS
Socket Application
Socket Application
Recv(s1,buffer,…);
Recv(s1,buffer,…);
SpaceSwitch
Space Switch (OMVS)
USS Logical File System (LFS) (OMVS) Streamlined path
USS USS Logical File System (LFS)
Suspend/Resume
Space Switch
Space Switch Services TCP/IP Physical File System (PFS) (TCP/IP) TCP/IP
TCP/IP Physical File System (PFS) (TCP/IP)
Suspend/Resume
Wait/Post Transport Layer (TCP, UDP, RAW)
Transport Layer (TCP, UDP, RAW) Services
Suspend/Resume
Space switch
Space Switch
Wait/Post
(OMVS)
IP (OMVS) IP Suspend/Resume

Interface/Device Driver Interface/Device Driver

OSA OSA

Full function support for sockets, including Streamlined path through USS LFS for
support for Unix signals, POSIX compliance selected socket APIs
When TCP/IP needs to suspend a thread TCP/IP performs the wait/post or
waiting for network flows, USS suspend/resume suspend/resume inline using its own services
services are invoked Significant reduction in path length

Page 19
© 2013 SHARE and IBM Corporation
 V2R1
TCP/IP Enhanced Fast Path Sockets

Pre-V2R1 fast path provided CPU savings but not widely

adopted:
 No support for Unix signals (other than SIGTERM)
Only useful to applications that have no requirement
for signal support
 No DBX support (debugger)
 Must be explicitly enabled!
BPXK_INET_FASTPATH environment variable
Iocc#FastPath IOCTL
 Only supported for UNIX System Services socket API or
the z/OS XL C/C++ Run-time Library functions

Page 20
© 2013 SHARE and IBM Corporation
 V2R1
TCP/IP Enhanced Fast Path Sockets

z/OS
Fast path sockets performance without all
Socket Application the conditions!:
Recv Send
Recvfrom Sendto • Enabled by default
Recvmsg Sendmsg

Space Switch
OMVS
• Full POSIX compliance, signals support
and DBX support
Streamlined
Streamlined path
path
USS Logical File System (LFS)
ThroughLFS
Through LFS
• Valid for ALL socket APIs (with the
Space Switch
TCP/IP Physical File System (PFS) TCP/IP
USS
exception of the Pascal API
Suspend/Resume
Transport Layer (TCP, UDP, RAW) Services
Pause/Release
No Services
IP Space
Switch!

Interface/Device Driver

Page 21
© 2013 SHARE and IBM Corporation
 V2R1
TCP/IP Enhanced Fast Path Sockets
 No new externals

 Still supports “activating Fast path explicitly” to avoid migration

issues
Provides performance benefits of enhanced Fast Path
sockets
Keeps the following restrictions:
Does not support POSIX signals (blocked by z/OS
UNIX)
Cannot use dbx debugger

Page 22
© 2013 SHARE and IBM Corporation
 V2R1
TCP/IP Enhanced Fast Path Sockets

V2R1 IPv4 AWM Primitives

V2R1 with Fastpath vs. V2R1 without Fastpath
%(Relative to without Fastpath)

30
12.48
20
2.23
10
Raw TPUT
0 -0.35 CPU-Client
-3.04 CPU-Server
-4.97 -4.97
-10 -9.12

-20
-22.32 -23.77
-30
-40
RR40 (1h/8h) CRR20 (64/8k) STR3 (1/20M)

IPv4 AWM Primitive Workloads

May 2, 2013
Client and server LPARs: zEC12 with 6 CPs per LPAR
Interface: OSA-E4 10 GbE

Note: The performance measurements discussed in this presentation are z/OS V2R1 Communications Server
numbers and were collected using a dedicated system environment. The results obtained in other
configurations or operating system environments may vary.

Page 23
© 2013 SHARE and IBM Corporation
 Optimizing inbound communications
using
OSA-Express

Page 24
© 2013 SHARE and IBM Corporation
Inbound Workload Queuing
V1R12
With OSA-Express3/4S IWQ and z/OS
V1R12, OSA now directs streaming traffic
onto its own input queue – transparently V1R13 z/OS
separating the streaming traffic away from
the more latency-sensitive interactive CPU 0 CPU 1 CPU 2 CPU 3

flows…

And each input queue has its own LAN-Idle

timer, so the Dynamic LAN Idle function can EE Sysplex Streaming Default
now tune the streaming (bulk) queue to Distributor (interactive)
conserve CPU (high LAN-idle timer setting),
while generally allowing the primary queue
to operate with very low latency (minimizing
its LAN-idle timer setting). So interactive
traffic (on the primary input queue) may see
significantly improved response time.
OSA
The separation of streaming traffic away Custom Lan Idle timer and
Interrupt processing for
from interactive also enables new streaming
each traffic pattern
traffic efficiencies in Communications
Server. This results in improved in-order
delivery (better throughput and CPU LAN
consumption). LAN
Page 25
© 2013 SHARE and IBM Corporation
 Improved Streaming Traffic Efficiency With IWQ
Before we had IWQ, Multiprocessor races would degrade streaming IWQ does away
performance!
SRB 1 on CP 0
with MP-race-
B at the time CP1 (SRB2) starts the TCP-layer induced ordering
B processing for Connection A's 1st packet, CP0 (SRB1) problems!
progress through the batch of inbound packets

has progressed only into Connection C's packets...

B
C SRB 2 on CP 1 With streaming
C A So, the Connection A traffic sorted onto
C packets being carried by
A SRB 2 will be seen its own queue, it is
D A before those carried by now convenient to
D A SRB 1...
D
service streaming
D
D This is out-of-order traffic from a
D packet delivery,
D B
single CP (i.e.,
brought on by
A B multiprocessor races using a single
A B through TCP/IP SRB).
inbound code.
B
C
Out-of-order delivery So with IWQ, we
C will consume no longer have
C excessive CPU and inbound SRB
memory, and usually
leads to throughput races for
problems. streaming data.
t1 - qdio rd interrupt, SRB disp CP 0 t2 - qdio rd interrupt, SRB disp CP 1
x x interrupt time.......
Page 26
© 2013 SHARE and IBM Corporation
 QDIO Inbound Workload Queuing – Configuration

 INBPERF DYNAMIC WORKLOADQ enables QDIO Inbound Workload

Queuing (IWQ)

>>-INTERFace--intf_name----------------------------------------->
.
.-INBPERF BALANCED--------------------.
>--+-------------------------------------+-->
| .-NOWORKLOADQ-. |
‘-INBPERF-+-DYNAMIC-+-------------+-+-’
| ‘-WORKLOADQ---’ |
+-MINCPU------------------+
‘-MINLATENCY--------------’

– INTERFACE statements only - no support for DEVICE/LINK definitions

– QDIO Inbound Workload Queuing requires VMAC

Page 27
© 2013 SHARE and IBM Corporation
QDIO Inbound Workload Queuing

 Display OSAINFO command (V1R12) shows you what’s registered in OSA

D TCPIP,,OSAINFO,INTFN=V6O3ETHG0
.
Ancillary Input Queue Routing Variables:
Queue Type: BULKDATA Queue ID: 2 Protocol: TCP
5-Tuples Src: 2000:197:11:201:0:1:0:1..221
Dst: 100::101..257
Src: 2000:197:11:201:0:2:0:1..290
Dst: 200::202..514
Total number of IPv6 connections: 2
Queue Type: SYSDIST Queue ID: 3 Protocol: TCP
Addr: 2000:197:11:201:0:1:0:1
DVIPAs
Addr: 2000:197:11:201:0:2:0:1
Total number of IPv6 addresses: 2
36 of 36 Lines Displayed
End of report

 BULKDATA queue registers 5-tuples with OSA (streaming connections)

 SYSDIST queue registers Distributable DVIPAs with OSA

Page 28
© 2013 SHARE and IBM Corporation
QDIO Inbound Workload Queuing: Netstat DEvlinks/-d

 Display TCPIP,,Netstat,DEvlinks to see whether QDIO inbound workload

queueing is enabled for a QDIO interface

D TCPIP,,NETSTAT,DEVLINKS,INTFNAME=QDIO4101L
EZD0101I NETSTAT CS V1R12 TCPCS1
INTFNAME: QDIO4101L INTFTYPE: IPAQENET INTFSTATUS: READY
PORTNAME: QDIO4101 DATAPATH: 0E2A DATAPATHSTATUS: READY
CHPIDTYPE: OSD
SPEED: 0000001000
...
READSTORAGE: GLOBAL (4096K)
INBPERF: DYNAMIC
WORKLOADQUEUEING: YES
CHECKSUMOFFLOAD: YES
SECCLASS: 255 MONSYSPLEX: NO
ISOLATE: NO OPTLATENCYMODE: NO
...
1 OF 1 RECORDS DISPLAYED
END OF THE REPORT

Page 29
© 2013 SHARE and IBM Corporation
QDIO Inbound Workload Queuing: Display TRLE

 Display NET,TRL,TRLE=trlename to see whether QDIO inbound workload

queueing is in use for a QDIO interface
D NET,TRL,TRLE=QDIO101
IST097I DISPLAY ACCEPTED
...
IST2263I PORTNAME = QDIO4101 PORTNUM = 0 OSA CODE LEVEL = ABCD
...
IST1221I DATA DEV = 0E2A STATUS = ACTIVE STATE = N/A
IST1724I I/O TRACE = OFF TRACE LENGTH = *NA*
IST1717I ULPID = TCPCS1
IST2310I ACCELERATED ROUTING DISABLED
IST2331I QUEUE QUEUE READ
IST2332I ID TYPE STORAGE
IST2205I ------ -------- ---------------
IST2333I RD/1 PRIMARY 4.0M(64 SBALS)
IST2333I RD/2 BULKDATA 4.0M(64 SBALS)
IST2333I RD/3 SYSDIST 4.0M(64 SBALS)
...
IST924I -------------------------------------------------------------
IST314I END

Page 30
© 2013 SHARE and IBM Corporation
QDIO Inbound Workload Queuing: Netstat ALL/-A

 Display TCPIP,,Netstat,ALL to see whether QDIO inbound workload

BULKDATA queueing is in use for a given connection

D TCPIP,,NETSTAT,ALL,CLIENT=USER1
EZD0101I NETSTAT CS V1R12 TCPCS1
CLIENT NAME: USER1 CLIENT ID: 00000046
LOCAL SOCKET: ::FFFF:172.16.1.1..20
FOREIGN SOCKET: ::FFFF:172.16.1.5..1030
BYTESIN: 00000000000023316386
BYTESOUT: 00000000000000000000
SEGMENTSIN: 00000000000000016246
SEGMENTSOUT: 00000000000000000922
LAST TOUCHED: 21:38:53 STATE: ESTABLSH
...
Ancillary Input Queue: Yes
BulkDataIntfName: QDIO4101L
...
APPLICATION DATA: EZAFTP0S D USER1 C PSSS
----
1 OF 1 RECORDS DISPLAYED
END OF THE REPORT

Page 31
© 2013 SHARE and IBM Corporation
QDIO Inbound Workload Queuing: Netstat STATS/-S

 Display TCPIP,,Netstat,STATS to see the total number of TCP segments

received on BULKDATA queues

D TCPIP,,NETSTAT,STATS,PROTOCOL=TCP
EZD0101I NETSTAT CS V1R12 TCPCS1
TCP STATISTICS
CURRENT ESTABLISHED CONNECTIONS = 6
ACTIVE CONNECTIONS OPENED = 1
PASSIVE CONNECTIONS OPENED = 5
CONNECTIONS CLOSED = 5
ESTABLISHED CONNECTIONS DROPPED = 0
CONNECTION ATTEMPTS DROPPED = 0
CONNECTION ATTEMPTS DISCARDED = 0
TIMEWAIT CONNECTIONS REUSED = 0
SEGMENTS RECEIVED = 38611
...
SEGMENTS RECEIVED ON OSA BULK QUEUES= 2169
SEGMENTS SENT = 2254
...
END OF THE REPORT

Page 32
© 2013 SHARE and IBM Corporation
 Quick INBPERF Review Before We Push On….

 The original static INBPERF settings (MINCPU, MINLATENCY, BALANCED)

provide sub-optimal performance for workloads that tend to shift between
request/response and streaming modes.

 We therefore recommend customers specify INBPERF DYNAMIC, since it

self-tunes, to provide excellent performance even when inbound traffic patterns
shift.

 Inbound Workload Queueing (IWQ) mode is an extension to the Dynamic LAN

Idle function. IWQ improves upon the DYNAMIC setting, in part because it
provides finer interrupt-timing control for mixed (interactive + streaming)
workloads.

Page 33
© 2013 SHARE and IBM Corporation
 Dynamic LAN Idle Timer: Performance Data
Dynamic LAN Idle improved RR1 TPS 50% and RR10 TPS by 33%. Response Time for
these workloads is improved 33% and 47%, respectively.

RR1 and RR10 Dynamic LAN Idle

100 87.7
D y n a m ic L A N Id le v s .

80
60 50.1
B a la n c e d

40 Trans/Sec
20
0
Resp Time
-20
-40 -33.4
-47.4
-60
RR1(1h/8h) RR10(1h/8h) z10 (4 CP LPARs),
z/OS V1R13, OSA-E3
1Gbe
1h/8h indicates 100 bytes in and 800 bytes out
Note: The performance measurements discussed in this presentation are z/OS V1R13 Communications Server
numbers and were collected using a dedicated system environment. The results obtained in other
configurations or operating system environments may vary.
Page 34
© 2013 SHARE and IBM Corporation
 Inbound Workload Queuing: Performance Data

IWQ: Mixed Workload Results vs DYNAMIC:

z/OS V1R12 z/OS V1R12 –z/OS<->AIX R/R Throughput improved 55% (Response
z10 Time improved 36%)
(3 CP
LPARs)
Aix 5.3 –Streaming Throughput also improved in this test: +5%
p570

Mixed Workload (IWQ vs Dynamic)

80000
70000

R R tr a n s /s e c
OSA-Express3’s

S T R K B /s e c
1GBe 60000
In Dynamic 50000
or 10GBe
or IWQ mode 40000

or
network 30000 DYNAMIC
20000
10000
IWQ
0
For z/OS outbound streaming to another
platform, the degree of performance boost RR30 STR1
(due to IWQ) is relative to receiving platform’s
sensitivity to out-of-order packet delivery. For RR (z/OS to AIX)
streaming INTO z/OS, IWQ will be especially STR (z/OS to z/OS)
beneficial for multi-CP configurations.

Page 35
© 2013 SHARE and IBM Corporation
 Inbound Workload Queuing: Performance Data
z/OS V1R12 z/OS V1R12
z10 IWQ: Pure Streaming Results vs DYNAMIC:
(3 CP
LPARs)
Aix 5.3 –z/OS<->AIX Streaming Throughput improved 40%
p570 –z/OS<->z/OS Streaming Throughput improved 24%

Pure Streaming (IWQ vs Dynamic)

OSA-Express3’s
in Dynamic 1GBe 600
or IWQ mode or 10GBe
network 500

400 DYNAMIC
M B /s e c
300 IWQ
200
For z/OS outbound streaming to another
platform, the degree of performance boost 100
(due to IWQ) is relative to receiving platform’s
sensitivity to out-of-order packet delivery. For 0
streaming INTO z/OS, IWQ will be especially z/OS to AIX z/OS to z/OS
beneficial for multi-CP configurations.

Page 36
© 2013 SHARE and IBM Corporation
IWQ Usage Considerations:

 Minor ECSA Usage increase: IWQ will grow ECSA usage by 72KBytes (per
OSA interface) if Sysplex Distributor (SD) is in use; 36KBytes if SD is not in use

 IWQ requires OSA-Express3 in QDIO mode running on IBM System z10 or OSA-
Express3/OSA-Express4 in QDIO mode running on zEnterprise 196/ zEC12.
 IWQ must be configured using the INTERFACE statement (not DEVICE/LINK)
 IWQ is not supported when z/OS is running as a z/VM guest with simulated
devices (VSWITCH or guest LAN)
 Make sure to apply z/OS V1R12 PTF UK61028 (APAR PM20056) for added
streaming throughput boost with IWQ

Page 37
© 2013 SHARE and IBM Corporation
 Optimizing outbound
communications using OSA-
Express

Page 38
© 2013 SHARE and IBM Corporation
TCP Segmentation Offload V1R13

 Segmentation consumes (high cost) host CPU cycles in the TCP stack
 Segmentation Offload (also referred to as “Large Send”)
– Offload most IPv4 and/or IPv6 TCP segmentation processing to OSA
– Decrease host CPU utilization
– Increase data transfer efficiency
– Checksum offload also added for IPv6

Single Large Segment Individual Segments

1-4 1 2 3 4
Host OSA
LAN
LAN

TCP Segmentation
Performed In the OSA

Page 39
© 2013 SHARE and IBM Corporation
z/OS Segmentation Offload performance measurements

OSA-Express3 10Gb OSA-Express4 10Gb

8.6
10 1.2 10

0 CPU/MB 0 CPU/MB

R e la t iv e t o n o
R e la t iv e t o n o

-10 Throughput -10 Throughput

o f f lo a d
o f f lo a d

-20 -20

-30 -30

-40 -40
-35.8
-41.5
-50 -50
STR-3 STR-3

Segmentation offload may significantly reduce CPU

Send buffer size: 180K for streaming workloads cycles when sending bulk data from z/OS!

Note: The performance measurements discussed in this presentation are z/OS V1R13 Communications Server
numbers and were collected using a dedicated system environment. The results obtained in other
configurations or operating system environments may vary.
Page 40
© 2013 SHARE and IBM Corporation
 V1R13
TCP Segmentation Offload: Configuration
 Enabled with IPCONFIG/IPCONFIG6 SEGMENTATIONOFFLOAD

>>-IPCONFIG------------------------------------------------->
.
.
>----+-----------------------------------------------+-+-------><
| .-NOSEGMENTATIONOFFLoad-. |
+-+-----------------------+--------------------+
| '-SEGMENTATIONOFFLoad---' |

 Disabled by default
 Previously enabled via GLOBALCONFIG Reminder!
 Segmentation cannot be offloaded for Checksum Offload
enabled by default
– Packets to another stack sharing OSA port
– IPSec encapsulated packets
– When multipath is in effect (unless all interfaces in the multipath group support
segmentation offload)

Page 41
© 2013 SHARE and IBM Corporation
z/OS Checksum Offload performance measurements
V1R13

zEC12 2CPs V2R1 - Effect of ChecksumOffload - IPv6

Performance Relative to NoChecksumOffload
OSA Exp4 10Gb interface
%(Relative to NoChecksumOffload)

10

0
-1.59
-2.66
CPU-Client
-5 -5.23 CPU-Server
-8.06
-10

-15 -13.65
-14.83

-20
RR30(1h/8h) CRR20(64/8k) STR3(1/20M)
AWM IPv6 Primitives Workloads

Note: The performance measurements discussed in this presentation are z/OS V2R1 Communications Server
numbers and were collected using a dedicated system environment. The results obtained in other
configurations or operating system environments may vary.

Page 42
© 2013 SHARE and IBM Corporation
 OSA-Express4

Page 43
© 2013 SHARE and IBM Corporation
OSA-Express4 Enhancements – 10GB improvements
 Improved on-card processor speed and memory bus provides better utilization of
10GB network

OSA 10GBe - Inbound Bulk traffic

1000 874
800
Th rou g hp u t

489
( M B /s e c )

600

400

200

0 z196 (4 CP LPARs),
OSA-E3 OSA-E4 z/OS V1R13, OSA-
E3/OSA-E4 10Gbe

Note: The performance measurements discussed in this presentation are z/OS V1R13 Communications Server
numbers and were collected using a dedicated system environment. The results obtained in other
configurations or operating system environments may vary.
Page 44
© 2013 SHARE and IBM Corporation
OSA-Express4 Enhancements – EE Inbound Queue
 Enterprise Extender queue provides internal optimizations
 EE traffic processed quicker
 Avoids memory copy of data

OSA 1GBe - mixed TCP and EE workloads

40 32.9
M IQ v s . D y n a m ic

30

20 Trans/Sec
10 CPU/trans
2.6
0 -0.4 -2.9
-10
TCP STR1(1/20MB) EE RR10(1h/8h) z196 (4 CP LPARs),
z/OS V1R13, OSA-
E3/OSA-E4 1Gbe

Note: The performance measurements discussed in this presentation are z/OS V1R13 Communications Server
numbers and were collected using a dedicated system environment. The results obtained in other
configurations or operating system environments may vary.
Page 45
© 2013 SHARE and IBM Corporation
OSA-Express4 Enhancements – Other improvements
 Checksum Offload support for IPv6 traffic
 Segmentation Offload support for IPv6 traffic

Page 46
© 2013 SHARE and IBM Corporation
 z/OS Communications Server
Performance Summaries

Page 47
© 2013 SHARE and IBM Corporation
z/OS Communications Server Performance Summaries

 Performance of each z/OS Communications Server release is studied by an

internal performance team
 Summaries are created and published on line
– http://www-01.ibm.com/support/docview.wss?rs=852&uid=swg27005524
 Ex: The z/OS V1R13 Communications Server Performance Summary includes:
– The z/OS V1R13 Communications Server performance summary includes:
• Performance of z/OS V1R13 Communications Server line items
• Release to release performance comparisons (z/OS V1R13
Communications Server versus z/OS V1R12 Communications Server)
• Capacity planning performance for:
– TN3270 (Clear Text, AT-TLS, and IPSec )
– FTP (Clear Text, AT-TLS, and IPSec)
– CICS Sockets performance
• CSM usage
• VTAM buffer usage

Page 48
© 2013 SHARE and IBM Corporation
z/OS Communications Server Performance Website

http://www-01.ibm.com/support/docview.wss?uid=swg27005524

Page 49
© 2013 SHARE and IBM Corporation
Please fill out your session evaluation

 z/OS CS Performance Improvements

 Session # 13633
 QR Code:

Page 50
© 2013 SHARE and IBM Corporation