We built the world’s first production petaflops system with the Cray XT5™

supercomputer. Then we reinvented high performance networking with

the Gemini™ interconnect. Now we bring it all together with many-core
technology. Introducing the first production hybrid supercomputing system
– the Cray XK7™ supercomputer.
 Redefining Supercomputing
The Cray XK7 supercomputer is a trifecta of scalar, network and many-core innovation. It combines the
proven high performance Gemini interconnect, AMD’s leading multi-core scalar processors and NVIDIA’s
powerful many-core GPU processors to create a true, productive hybrid supercomputer.

Capable of scaling to 500,000 scalar

processors and 50 petaflops of hybrid peak
performance, every aspect of the Cray XK7
system – from its resiliency features to its
scalability-boosting technologies – has been
engineered to meet science’s real-world
demands. And with the expertise of the
industry’s only company singularly focused
on high performance computing behind it,
the Cray XK7 supercomputer brings Cray’s
trademark reliability, flexibility and scalability
to the many-core HPC environment.

Scalable Many-Core Performance

Adaptive Hybrid Computing
The Cray XK7 compute node combines AMD’s 16-core Opteron™
6200 Series processor and NVIDIA® Tesla® K20 GPU Accelerator to
create a hybrid unit with the intra-node scalability, power-efficiency
of acceleration and flexibility to run applications with either scalar
or accelerator components. This compute unit, combined with the
Gemini interconnect’s excellent inter-node scalability, creates a
system geared for any computing challenge.

Gemini Scalable Interconnect

Capable of tens of millions of MPI messages per second, the
Gemini ASIC complements current and future massively multi-
core and many-core processors. Each hybrid compute node is
interfaced to the Gemini interconnect through HyperTransport™
3.0 technology. This direct connect architecture bypasses the PCI
bottlenecks inherent in commodity networks and provides a peak
of over 20 GB/s of injection bandwidth per node. The Gemini
router’s connectionless protocol scales from hundreds to hundreds
of thousands of cores without the increase in buffer memory
required in the point-to-point connection method of commodity
interconnects. The Cray XK7 network provides industry-leading
sub-microsecond latency for remote puts and 1-2 microsecond
latency for most other point-to-point messages. An internal block
transfer engine is available to provide high bandwidth and good
overlap of computation and communication for long messages.
Advanced features include support for one-sided communication
primitives and support for atomic memory operations. The proven
3-D torus topology provides powerful bisection and global
bandwidth characteristics as well as support for dynamic routing
of messages.
Scalable Programming Capabilities
The Cray XK7 supercomputer supports a full range of powerful
scalar tools, libraries, compilers, operating system and third- In addition to supporting MPI over the standard programming
party software. languages of C, C++ and Fortran, the Gemini interconnect has
direct hardware support for partitioned global address space
Programming Environment (PGAS) programming models including Unified Parallel C (UPC),
Each Cray XK7 system includes a fully integrated Cray programming Co-array Fortran and Chapel. Gemini allows remote references
environment with tools designed to maximize programmer to be pipelined in these programming models which can result
productivity and application scalability and performance. This in orders-of-magnitude performance improvement over library-
feature-rich, flexible programming environment facilitates the based message passing models. This feature brings highly scalable
development of scalable applications. For example, the Cray XK7 performance to communication-intensive, irregular algorithms.
supercomputer can use a variety of high performance Fortran, C
and C++ compilers and libraries for both x86 and GPU workloads. Scalable Software
Available compilers include GPU-enabled PGI, CAPS, PathScale and The Cray XK7 system ships with the latest Cray Linux Environment™
the Cray Compiler Environment for x86 with support for optimized (CLE), a suite of high performance software including a SUSE™
C, C++ and Fortran 90, UPC and Co-Array Fortran. Available high Linux-based operating system designed to scale efficiently and
performance-optimized math libraries include BLAS, FFTs, LAPACK, run large, complex applications. With highly scalable applications,
ScaLAPACK, SuperLU and Cray Scientific and Math Libraries. Compute Node Linux (CNL) runs in Extreme Scalability Mode
(ESM), ensuring operating system services do not interfere with
Programming Models application scalability. Real-world applications have proven this
Supported parallel programming models include MPI, CUDA, optimized design scales to 250,000 cores.
Cray SHMEM™, UPC, Co-Array Fortran, OpenMP and OpenACC
(for directives-based accelerator programming). The MPI Additionally, the Cray XK7 system provides for tightly integrated,
implementation is compliant with the MPI 2.0 standard and is industry-standard batch schedulers, compilers and debuggers.
optimized to take advantage of the Gemini interconnect. Cray’s Supported workload managers include Altair PBS Professional®,
performance and analysis toolkit CrayPat™ allows users to analyze Moab Adaptive Computing Suite™ and Platform LSF®. Supported
resource utilization throughout their code at scale and eliminate compilers include PGI, PathScale, CAPS and Cray. Supported
bottleneck and load imbalance issues. debuggers include TotalView Technologies and Allinea as well as

Scalable Compute Nodes Scalable I/O

Each Cray XK7 blade includes four compute nodes for high scalability The Cray XK7 I/O subsystem scales to meet the bandwidth needs
in a small footprint with up to 64 AMD processor cores per blade. Each of even the most data-intensive applications. Each Cray XIO service
compute node has an AMD Opteron 6200 Series processor with four blade provides four multi-purpose I/O nodes, each with a six-core
channels of DDR3 memory and an NVIDIA® Tesla® K20 GPU Computing AMD Opteron Series 2000 processor coupled to 16 GB of DDR2
Accelerator with 6GB of GDDR5 memory. The CUDA GPU architecture memory and a PCI-express GEN2 interface. Additionally, the Cray XIO
of the NVIDIA Tesla processor incorporates error correcting code for service blade provides 32 GB/s of peak I/O bandwidth and supports
memories and double precision floating point units. connectivity to networks and storage devices using Ethernet, Fibre
Channel or InfiniBand interfaces.
Each Cray XK7 node can be configured with 16 GB or 32 GB DDR3
memory. Memory on compute nodes is registered and memory The Cray user data storage architecture consists of RAID6 arrays
controllers provide x4 device correction, ensuring reliable memory connected directly to Cray XIO nodes or via external SANs with
performance while retaining the upgradeability, serviceability and complete multi-path failover. Cray systems may be ordered with
flexibility of a socketed component. a Lustre file system that manages the striping of file operations
across these arrays. This highly scalable I/O architecture allows for
configuring bandwidth and data capacity by selecting the appropriate
number of arrays and service nodes.
The AMD Opteron 6200 Series processor’s The NVIDIA Tesla is designed from the
highly associative on-chip data cache supports ground up for high performance computing.
aggressive out-of-order execution. The integrat- Based on the next generation CUDA GPU
ed memory controller eliminates the need for a architecture codenamed “Kepler,” it supports
separate Northbridge memory chip and provides many must-have features for technical and
a high-bandwidth path to local memory with enterprise computing. These include ECC
four channels of DDR3 memory per compute protection for uncompromised accuracy and
node. This design brings a significant perfor- data reliability, support for C++, and double
mance advantage to algorithms that stress local precision floating-point performance.
memory bandwidth and plenty of headroom for
processor upgrades.

Adaptive Supercomputing

CRAY LINUX ENVIRONMENT

Extreme Scale and Cluster Compatibility in One System
The Cray XK7 system provides complete workload flexibility. For
ESM – CCM – the first time, you can buy a single machine to run both a highly
Extreme Scalability Mode Cluster Compatibility Mode scalable custom workload and industry-standard ISV workload.
• No compromise compatibility
CLE accomplishes this through the Cluster Compatibility Mode
• Uncompromised scalability
• Low-Noise Kernel for scalability • Fully standard x86/Linux (CCM). CCM allows out-of-the-box compatibility with Linux/x86
• Native Comm. & Optimized MPI • Standardized Communication versions of ISV software – without recompilation or relinking – and
• Application-specific Layer allows for the use of various versions of MPI (e.g., MPICH, Platform
performance tuning and scaling • Out-of-the-box ISV Installation
MPI™). At job submission, you can request the CNL compute
nodes be configured with CCM, complete with the necessary
services to ensure Linux/x86 compatibility. The service is dynamic
and available on an individual job basis.
Production Reliability
Integrated Hardware Supervisory System Support for Other File System and Data Management Services
Cray’s Hardware Supervisory System (HSS) integrates hardware You can select the Lustre parallel file system or another option
and software components to provide system monitoring, fault including connecting to an existing parallel file system. The Cray
identification and recovery. An independent system with its own Data Virtualization Service allows for the projection of various other
control processors and supervisory network, the HSS monitors and file systems (including NFS, GPFS™, Panasas® and StorNext®) to
manages all major hardware and software components in the Cray the compute and login nodes on the Cray XK7 system. The Cray
XK7 supercomputer. In addition to providing recovery services in Data Management group can also provide solutions for backup,
the event of a hardware or software failure, HSS controls power- archiving and data lifecycle management.
up, power-down and boot sequences, manages the interconnect,
reroutes around failed interconnect links, and displays the Emphasis on Power Efficiency
machine state to the system administrator. The Cray XK7 system Many-core processing is the key to ultimate energy efficiency.
also supports a warm swap capability allowing a system operator Applications using the Cray XK7 many-core GPU processors will
to remove and repair system blades without disrupting an active experience industry-leading energy efficiency when measured
workload. for real application workloads. Combined with our standard air-
or liquid-cooled High Efficiency cabinet and optional ECOphlex™
Cray XK7 System Resiliency technology, the Cray XK7 system can reduce cooling costs and
The Gemini interconnect is designed for large systems in which increase flexibility in datacenter design. Each High Efficiency
failures are to be expected and applications must run to successful cabinet can be configured with inline phase-change evaporator
completion in the presence of errors. coils which extract virtually all the heat imparted to the airstream
as it passes through the cabinet. Coolant is recondensed in a heat
Gemini uses error correcting code (ECC) to protect major exchange unit connected to the building chilled water supply.
memories and data paths within the device. The ECC combined
with the Gemini adaptive routing hardware (which spreads data ECOphlex technology accommodates a range of building water
packets over the four available lanes which comprise each of the temperatures, so a modern datacenter can operate chillers and
torus links) provide improved system and applications resiliency. air handlers less often, reducing electrical costs. In fact, a system
In the event of a lane failure, the adaptive routing hardware will fitted with ECOphlex operating at full capacity needs only cooling
automatically mask it out. In the event of losing all connectivity towers during much of the year in many climates.
between two interconnects, the HSS automatically reconfigures to
route around the bad link. Investment Protection and Blended Systems
The Cray XK7 supercomputer is engineered for easy, flexible up-
Additionally, the Cray Linux Environment features NodeKARE™ grades and expansion – prolonging its productive lifetime and
(Node Knowledge and Reconfiguration). If a program terminates your investment. As new technologies become available, you can
abnormally, NodeKARE automatically runs diagnostics on all take advantage of these next-generation compute processors, I/O
involved compute nodes and removes any unhealthy ones from technologies and interconnect without replacing the entire Cray
the compute pool. Subsequent jobs are allocated only to healthy XK7 system. In addition, Cray XK7 and Cray XE6 systems, support
nodes and run reliably to completion. blended configurations on the same Gemini interconnect and
share the same power, cooling, I/O and service infrastructure,
The Lustre file system can be configured with object storage target making it easy for current Cray XE6 users to add Cray XK7
failover and metadata server failover. Software failover is provided technology.
for all critical system software functions.
Cray XK7 Specifications
16-core 64-bit AMD Opteron 6200 Series processors, up to 96 per cabinet; NVIDIA® Tesla® K20
Processor
GPU Computing Accelerators, up to 96 per cabinet
16-64 GB per node
Memory
Memory Bandwidth: Up to 52 GB/s per node
AMD processing cores: 1,536 processor cores per system cabinet
Compute Cabinet
Peak performance: 100+ Tflops per system cabinet
1 Gemini routing and communications ASIC per two compute nodes
Interconnect 48 switch ports per Gemini chip, (160 GB/s internal switching capacity per chip)
3-D torus interconnect
Cray System Management workstation
Graphical and command line system administration
System Administration
Single-system view for system administration
System software rollback capability
Cray Hardware Supervisory System (HSS) with independent 100 Mb/s management fabric
between all system blades and cabinet-level controllers
Reliability Features (Hardware) Full ECC protection of all packet traffic in the Gemini network
Redundant power supplies; redundant voltage regulator modules
Redundant paths to all system RAID
HSS system monitors operation of all operating system kernels
Lustre file system object storage target failover; Lustre metadata server failover
Reliability Features (Software) Software failover for critical system services including system database, system logger
and batch subsystems
NodeKARE (Node Knowledge and Reconfiguration)
Cray Linux Environment (components include SUSE Linux SLES11, HSS and SMW software)
Operating System
Extreme Scalabiliity Mode (ESM) and Cluster Compatibility Mode (CCM)
PGI compilers, Cray Compiler Environment, PathScale, CUDA, CAPS, support for Fortran 77, 90, 95,
Compilers, Libraries & Tools
C/C++, UPC, Co-Array Fortran MPI 2.0, Cray SHMEM, other standard MPI libraries using CCM
Job Management PBS Professional, Moab Adaptive Computing Suite, Platform LSF
External I/O Interface InfiniBand, 10 Gigabit Ethernet, Fibre Channel (FC) and Ethernet
Disk Storage Full line of FC-attached disk arrays with support for FC and SATA disk drives
Parallel File System Lustre, Data Virtualization Service allows support for NFS, external Lustre and other file systems
45-54.1 kW (45.9 – 55.2 kVA) per cabinet, depending on configuration
Power Circuit requirements: three-phase wye, 100 AMP at 480/277 and 125 AMP at 400/230 (three-
phase, neutral and ground)
Air-cooled, air flow: 3,000 cfm (1.41 m3/s); intake: bottom; exhaust: top
Cooling
Optional ECOphlex liquid cooling
Dimensions (Cabinet) H 93 in. (2,362 mm) x W 22.50 in. (572 mm) x D 56.75 in. (1,441 mm)
Weight (Maximum) 1,600 lbs. per cabinet (725 kg) air cooled; 2,000 lbs. per cabinet (907 kg) liquid cooled
Regulatory Compliance UL 60950-1, CAN/CSA – C 22.2 No. 60950-1, CE-mark, RoHS, WEEE
FCC Class A, VCCI Class A, ICES-003, EN 50022:2006 Class A, AS/NZS CISPR 22:2006, EN 55024:
Safety
1998 +A1:2002 +A2:2003

Cray Inc. • 901 Fifth Avenue, Suite 1000 • Seattle, WA 98164 • Tel: 206.701.2000 • Fax: 206.701.2500
 ©2011 Cray Inc. All rights reserved. Specifications subject to change without notice. Cray is a registered trademark, Cray XK7, Cray Linux Environment, Cray XT5, XE6, ECOPhlex,
Cray SHMEM, Cray Apprentice2, and NodeKARE are trademarks of Cray Inc. All other trademarks mentioned herein are the properties of their respective owners. 20120926_6200JRC