Virtualization Guide - SUSE Linux Enterprise Server 12 SP2

Virtualization Guide | SUSE Linux Enterprise Server 12 SP2 https://www.suse.com/documentation/sles-12/singlehtml/book_virt/boo...
Virtualization Guide
SUSE Linux Enterprise Server 12 SP2
Describes virtualization technology in general, and introduces libvirt—the

unified interface to virtualization—and detailed information on specific
hypervisors.
Publication Date: March 31, 2017
About This Manual

Available Documentation
Feedback
Documentation Conventions
I Introduction
1 Virtualization Technology
1.1 Overview
1.2 Virtualization Capabilities
1.3 Virtualization Benefits
1.4 Understanding Virtualization Modes
1.5 I/O Virtualization
2 Introduction to Xen Virtualization

2.1 Basic Components
2.2 Xen Virtualization Architecture
3 Introduction to KVM Virtualization

3.2 KVM Virtualization Architecture
1 z 366 2017-05-27 13:21

4 Introduction to Linux Containers

5 Virtualization Tools
5.1 Virtualization Console Tools
5.2 Virtualization GUI Tools
6 Installation of Virtualization Components

6.1 Installing KVM
6.2 Installing Xen
6.3 Installing Containers
6.4 Patterns
7 Supported Guests, Hosts and Features

7.1 Supported VM Guests
7.2 Supported VM Host Servers for SUSE Linux Enterprise Server 12 SP2 VM
Guests
7.3 KVM Hardware Requirements
7.4 Feature Support
II Managing Virtual Machines with libvirt

8 Starting and Stopping libvirtd
9 Guest Installation
9.1 GUI-Based Guest Installation
9.2 Installing from the Command Line with virt-install
9.3 Advanced Guest Installation Scenarios
10 Basic VM Guest Management

10.1 Listing VM Guests
10.2 Accessing the VM Guest via Console
10.3 Changing a VM Guest's State: Start, Stop, Pause
10.4 Saving and Restoring the State of a VM Guest
10.5 Creating and Managing Snapshots
10.6 Deleting a VM Guest
10.7 Migrating VM Guests
10.8 Monitoring
11 Connecting and Authorizing

11.1 Authentication
11.2 Connecting to a VM Host Server
2 z 366 2017-05-27 13:21

11.3 Configuring Remote Connections
12 Managing Storage
12.1 Managing Storage with Virtual Machine Manager
12.2 Managing Storage with virsh
12.3 Locking Disk Files and Block Devices with virtlockd
12.4 Online Resizing of Guest Block Devices
12.5 Sharing Directories between Host and Guests (File System
Pass-Through)
13 Managing Networks
13.1 Virtual Networks
13.2 Bridged Networking
14 Configuring Virtual Machines

14.1 Machine Setup
14.2 Storage
14.3 Controllers
14.4 Networking
14.5 Enabling Seamless and Synchronized Mouse Pointer Movement
14.6 Adding a CD/DVD-ROM Device with Virtual Machine Manager
14.7 Adding a Floppy Device with Virtual Machine Manager
14.8 Ejecting and Changing Floppy or CD/DVD-ROM Media with Virtual
Machine Manager
14.9 Changing the Machine Type with virsh
14.10 Adding a PCI Device to a VM Guest
14.11 Adding SR-IOV Devices
III Hypervisor-Independent Features

15 Disk Cache Modes
15.1 Disk Interface Cache Modes
15.2 Description of Cache Modes
15.3 Data Integrity Implications of Cache Modes
15.4 Performance Implications of Cache Modes
15.5 Effect of Cache Modes on Live Migration
16 VM Guest Clock Settings

16.1 KVM: Using kvm_clock
3 z 366 2017-05-27 13:21

16.2 Xen Virtual Machine Clock Settings
17 libguestfs
17.1 VM Guest Manipulation Overview
17.2 Package Installation
17.3 Guestfs Tools
17.4 Troubleshooting
17.5 External References
IV Managing Virtual Machines with Xen

18 Setting Up a Virtual Machine Host
18.1 Best Practices and Suggestions
18.2 Managing Dom0 Memory
18.3 Network Card in Fully Virtualized Guests
18.4 Starting the Virtual Machine Host
18.5 PCI Pass-Through
18.6 USB Pass-Through with PVUSB
19 Virtual Networking
19.1 Network Devices for Guest Systems
19.2 Host-Based Routing in Xen
19.3 Creating a Masqueraded Network Setup
19.4 Special Configurations
20 Managing a Virtualization Environment

20.1 XL—Xen Management Tool
20.2 Automatic Start of Guest Domains
20.3 Event Actions
20.4 Time Stamp Counter
20.5 Saving Virtual Machines
20.6 Restoring Virtual Machines
20.7 Virtual Machine States
21 Block Devices in Xen

21.1 Mapping Physical Storage to Virtual Disks
21.2 Mapping Network Storage to Virtual Disk
21.3 File-Backed Virtual Disks and Loopback Devices
21.4 Resizing Block Devices
4 z 366 2017-05-27 13:21

21.5 Scripts for Managing Advanced Storage Scenarios
22 Virtualization: Configuration Options and Settings

22.1 Virtual CD Readers
22.2 Remote Access Methods
22.3 VNC Viewer
22.4 Virtual Keyboards
22.5 Dedicating CPU Resources
22.6 HVM Features
23 Administrative Tasks
23.1 The Boot Loader Program
23.2 Sparse Image Files and Disk Space
23.3 Migrating Xen VM Guest Systems
23.4 Monitoring Xen
23.5 Providing Host Information for VM Guest Systems
24 XenStore: Configuration Database Shared between Domains

24.1 Introduction
24.2 File System Interface
25 Xen as a High-Availability Virtualization Host

25.1 Xen HA with Remote Storage
25.2 Xen HA with Local Storage
25.3 Xen HA and Private Bridges
V Managing Virtual Machines with QEMU

26 QEMU Overview
27 Setting Up a KVM VM Host Server
27.1 CPU Support for Virtualization
27.2 Required Software
27.3 KVM Host-Specific Features
28.1 Basic Installation with qemu-system-ARCH
28.2 Managing Disk Images with qemu-img
29 Running Virtual Machines with qemu-system-ARCH

29.1 Basic qemu-system-ARCH Invocation
5 z 366 2017-05-27 13:21

29.2 General qemu-system-ARCH Options

29.3 Using Devices in QEMU
29.4 Networking in QEMU
29.5 Viewing a VM Guest with VNC
30 Virtual Machine Administration Using QEMU Monitor

30.1 Accessing Monitor Console
30.2 Getting Information about the Guest System
30.3 Changing VNC Password
30.4 Managing Devices
30.5 Controlling Keyboard and Mouse
30.6 Changing Available Memory
30.7 Dumping Virtual Machine Memory
30.8 Managing Virtual Machine Snapshots
30.9 Suspending and Resuming Virtual Machine Execution
30.10 Live Migration
30.11 QMP - QEMU Machine Protocol
VI Managing Virtual Machines with LXC

31 Linux Containers
31.1 Setting Up LXC Distribution Containers
31.2 Setting Up LXC Application Containers
31.3 Securing a Container Using AppArmor
31.4 Differences Between the libvirt LXC Driver and LXC
31.5 For More Information
32 Migration from LXC to libvirt-lxc

32.1 Host Migration
32.2 Container Migration
32.3 Starting the Container
Glossary
A Virtual Machine Drivers
B Appendix
B.1 Installing Paravirtualized Drivers
B.2 Generating x509 Client/Server Certificates
C XM, XL Toolstacks and Libvirt framework
6 z 366 2017-05-27 13:21

C.1 Xen Toolstacks

C.2 Import Xen Domain Configuration into libvirt
C.3 Differences Between the xm and xl Applications
C.4 External links
C.5 Saving a Xen Guest Configuration in an xm Compatible Format
D Documentation Updates
D.1 November 2016 (Release of SUSE Linux Enterprise Server 12 SP2)
D.2 March 2016 (Maintenance Release of SUSE Linux Enterprise Server 12 SP1)
D.3 December 2015 (Initial Release of SUSE Linux Enterprise Server 12 SP1)
D.4 February 2015 (Documentation Maintenance Update)
D.5 October 2014 (Initial Release of SUSE Linux Enterprise Server 12)
E GNU Licenses
E.1 GNU Free Documentation License
List of Figures
13.1 Connection Details
13.2 Create virtual network
14.1 Details View of a VM Guest
14.2 Overview details
14.3 VM Guest Title and Description
14.4 Performance
14.5 Statistics Charts
14.6 Processor View
14.7 Memory View
14.8 Boot Options
14.9 Add a New Storage
14.10 Add a New Controller
14.11 Add a New Controller
14.12 Adding a PCI Device
23.1 Boot Loader Settings
27.1 Installing the KVM Hypervisor and Tools
27.2 Network Bridge
28.1 New 2 GB Partition in Guest YaST Partitioner
29.1 QEMU Window with SLES 11 SP3 as VM Guest
7 z 366 2017-05-27 13:21

29.2 Configuring Network Bridge with YaST

29.3 QEMU VNC Session
29.4 Authentication Dialog in Vinagre
List of Tables
7.1 Paravirtualized OS Support
7.2 Feature Support—Host ( Dom0 )
7.3 Feature Support—Paravirtualized Guest
7.4 Feature Support—Fully Virtualized Guest
25.1 Xen Remote Storage
C.1 Notation Conventions
C.2 New Global Options
C.3 Common Options
C.4 Domain Management Removed Options
C.5 USB Devices Management Removed Options
C.6 CPU Management Removed options
C.7 Other Options
C.8 xl create Changed Options
C.9 xm create Removed Options
C.10 xl create Added Options
C.11 xl console Added Options
C.12 xm info Removed Options
C.13 xm dump-core Removed Options
C.14 xm list Removed Options
C.15 xl list Added Options
C.16 xl mem-* Changed Options
C.17 xm migrate Removed Options
C.18 xl migrate Added Options
C.19 xm reboot Removed Options
C.20 xl reboot Added Options
C.21 xl save Added Options
C.22 xl restore Added Options
C.23 xm shutdown Removed Options
C.24 xl shutdown Added Options
C.25 xl trigger Changed Options
C.26 xm sched-credit Removed Options
C.27 xl sched-credit Added Options
C.28 xm sched-credit2 Removed Options
C.29 xl sched-credit2 Added Options
C.30 xm sched-sedf Removed Options
C.31 xl sched-sedf Added Options
C.32 xm cpupool-list Removed Options
8 z 366 2017-05-27 13:21

C.33 xm cpupool-create Removed Options

C.34 xl pci-detach Added Options
C.35 xm block-list Removed Options
C.36 Other Options
C.37 Network Options
C.38 xl network-attach Removed Options
C.39 New Options
List of Examples
9.1 Example of a virt-install command line
10.1 Typical Output of kvm_stat
13.1 NAT Based Network
13.2 Routed Network
13.3 Isolated Network
13.4 Using an Existing Bridge on VM Host Server
20.1 Guest Domain Configuration File: /etc/xen/sled12.cfg
27.1 Exporting Host's File System with VirtFS
29.1 Restricted User-mode Networking
29.2 User-mode Networking with Custom IP Range
29.3 User-mode Networking with Network-boot and TFTP
29.4 User-mode Networking with Host Port Forwarding
29.5 Password Authentication
29.6 x509 Certificate Authentication
29.7 x509 Certificate and Password Authentication
29.8 SASL Authentication
C.1 Converting Xen Domain Configuration to libvirt
Copyright © 2006– 2017 SUSE LLC and contributors. All rights reserved.
Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Docu-
mentation License, Version 1.2 or (at your option) version 1.3; with the Invariant Section being this copyright
notice and license. A copy of the license version 1.2 is included in the section entitled “GNU Free Documenta-
tion License”.
For SUSE trademarks, see http://www.suse.com/company/legal/↗ . All other third-party trademarks are the
property of their respective owners. Trademark symbols (®, ™ etc.) denote trademarks of SUSE and its affili-
ates. Asterisks (*) denote third-party trademarks.
All information found in this book has been compiled with utmost attention to detail. However, this does not
guarantee complete accuracy. Neither SUSE LLC, its affiliates, the authors nor the translators shall be held li-
able for possible errors or the consequences thereof.
9 z 366 2017-05-27 13:21

About This Manual

Available Documentation
Feedback
Documentation Conventions
This manual offers an introduction to setting up and managing virtualization with

KVM (Kernel-based Virtual Machine), Xen, and Linux Containers (LXC) on SUSE Linux
Enterprise Server. The first part introduces the different virtualization solutions by
describing their requirements, their installations and SUSE's support status. The sec-
ond part deals with managing VM Guests and VM Host Servers with libvirt . The
following parts describe various administration tasks and practices and the last
three parts deal with hypervisor-specific topics.
Many chapters in this manual contain links to additional documentation resources.

This includes additional documentation that is available on the system and docu-
mentation available on the Internet.
For an overview of the documentation available for your product and the latest doc-
umentation updates, refer to http://www.suse.com/doc↗.
1 Available Documentation
We provide HTML and PDF versions of our books in different languages. The follow-
ing manuals for users and administrators are available for this product:
Article “Installation Quick Start”

Lists the system requirements and guides you step-by-step through the installa-
tion of SUSE Linux Enterprise Server from DVD, or from an ISO image.
Book “Deployment Guide”
10 z 366 2017-05-27 13:21

Shows how to install single or multiple systems and how to exploit the product
inherent capabilities for a deployment infrastructure. Choose from various ap-
proaches, ranging from a local installation or a network installation server to a
mass deployment using a remote-controlled, highly-customized, and automated
installation technique.
Book “Administration Guide”

Covers system administration tasks like maintaining, monitoring and customiz-
ing an initially installed system.
Virtualization Guide
Describes virtualization technology in general, and introduces libvirt—the uni-
fied interface to virtualization—and detailed information on specific hypervi-
sors.
Book “Storage Administration Guide”

Provides information about how to manage storage devices on a SUSE Linux En-
terprise Server.
Book “AutoYaST”
AutoYaST is a system for installing one or more SUSE Linux Enterprise Server
systems automatically and without user intervention, using an AutoYaST profile
that contains installation and configuration data. The manual guides you
through the basic steps of auto-installation: preparation, installation, and config-
uration.
Book “Security Guide”

Introduces basic concepts of system security, covering both local and network
security aspects. Shows how to use the product inherent security software like
AppArmor or the auditing system that reliably collects information about any se-
curity-relevant events.
Book “Security and Hardening Guide”

Deals with the particulars of installing and setting up a secure SUSE Linux Enter-
prise Server, and additional post-installation processes required to further se-
cure and harden that installation. Supports the administrator with security-re-
lated choices and decisions.
Book “System Analysis and Tuning Guide”

An administrator's guide for problem detection, resolution and optimization.
Find how to inspect and optimize your system by means of monitoring tools
and how to efficiently manage resources. Also contains an overview of common
problems and solutions and of additional help and documentation resources.
11 z 366 2017-05-27 13:21

Book “Subscription Management Tool for SLES 12 SP2”

An administrator's guide to Subscription Management Tool—a proxy system for
SUSE Customer Center with repository and registration targets. Learn how to in-
stall and configure a local SMT server, how to mirror and manage repositories,
how to manage client machines, and how to configure clients to use SMT.
Book “GNOME User Guide”

Introduces the GNOME desktop of SUSE Linux Enterprise Server. It guides you
through using and configuring the desktop and helps you perform key tasks. It
is intended mainly for end users who want to make efficient use of GNOME as
their default desktop.
Find HTML versions of most product manuals in your installed system under
/usr/share/doc/manual . The latest documentation updates are available at
http://www.suse.com/documentation/↗ where you can download the documenta-
tion for your product in various formats.
2 Feedback
Several feedback channels are available:
Bugs and Enhancement Requests

For services and support options available for your product, refer to
http://www.suse.com/support/↗.
To report bugs for a product component, go to https://scc.suse.com/support
/requests↗, log in, and click Create New.
User Comments
We want to hear your comments about and suggestions for this manual and the
other documentation included with this product. Use the User Comments fea-
ture at the bottom of each page in the online documentation or go to
http://www.suse.com/documentation/feedback.html↗ and enter your com-
ments there.
Mail
For feedback on the documentation of this product, you can also send a mail to
doc-team@suse.com . Make sure to include the document title, the product
version and the publication date of the documentation. To report errors or sug-
gest enhancements, provide a concise description of the problem and refer to
the respective section number and page (or URL).
12 z 366 2017-05-27 13:21

3 Documentation Conventions
The following notices and typographical conventions are used in this documenta-
tion:
/etc/passwd : directory names and file names
PLACEHOLDER : replace PLACEHOLDER with the actual value

PATH : the environment variable PATH
ls , --help : commands, options, and parameters
user : users or groups
package name : name of a package

Alt, Alt – F1 : a key to press or a key combination; keys are shown in up-
percase as on a keyboard
File, File › Save As: menu items, buttons
x86_64 This paragraph is only relevant for the AMD64/Intel 64 architecture.

The arrows mark the beginning and the end of the text block.
System z, POWERThis paragraph is only relevant for the architectures

z Systems and POWER . The arrows mark the beginning and the end of the
text block.
Dancing Penguins (Chapter Penguins, ↑Another Manual): This is a reference to

a chapter in another manual.
Commands that must be run with root privileges. Often you can also prefix
these commands with the sudo command to run them as non-privileged
user.
root # command
tux > sudo command
Commands that can be run by non-privileged users.
tux > command
Notices
13 z 366 2017-05-27 13:21

Warning: Warning Notice
Vital information you must be aware of before proceeding. Warns you

about security issues, potential loss of data, damage to hardware, or
physical hazards.
Important: Important Notice
Important information you should be aware of before proceeding.
Note: Note Notice
Additional information, for example about differences in software ver-

sions.
Tip: Tip Notice
Helpful information, like a guideline or a piece of practical advice.
Part I Introduction
Virtualization is a technology that provides a way for a machine (Host) to run an-
other operating system (guest virtual machines) on top of the host operating
system.

This chapter introduces and explains the components and technologies you
need to understand to set up and manage a Xen-based virtualization environ-
ment.
14 z 366 2017-05-27 13:21


Linux containers are a lightweight virtualization method to run multiple virtual
units (“containers”) simultaneously on a single host. This is similar to the chroot
environment. Containers are isolated with kernel Control Groups (cgroups) and
kernel Namespaces.
libvirt is a library that provides a common API for managing popular virtual-
ization solutions, among them KVM, LXC, and Xen. The library provides a nor-
malized management API for these virtualization solutions, allowing a stable,
cross-hypervisor interface for higher-level management tools. The library also
provides APIs for management of virtual networks and storage on the VM Host
Server. The configuration of each VM Guest is stored in an XML file.
With libvirt you can also manage your VM Guests remotely. It supports TLS
encryption, x509 certificates and authentication with SASL. This enables manag-
ing VM Host Servers centrally from a single workstation, alleviating the need to
access each VM Host Server individually.
Using the libvirt -based tools is the recommended way of managing VM
Guests. Interoperability between libvirt and libvirt -based applications
has been tested and is an essential part of SUSE's support stance.
6 Installation of Virtualization Components

None of the virtualization tools is installed by default.
7 Supported Guests, Hosts and Features

Supported virtualization limits for Xen and KVM are outlined in the Release
Notes↗.
Abstract
15 z 366 2017-05-27 13:21

Virtualization is a technology that provides a way for a machine (Host) to

run another operating system (guest virtual machines) on top of the host
operating system.
1.1 Overview
1.1 Overview
SUSE Linux Enterprise includes the latest open source virtualization technologies,
Xen and KVM. With these Hypervisors, SUSE Linux Enterprise can be used to provi-
sion, de-provision, install, monitor and manage multiple virtual machines (VM
Guests) on a single physical system (for more information see Hypervisor).
Out of the box, SUSE Linux Enterprise can create virtual machines running both
modified, highly tuned, paravirtualized operating systems and fully virtualized un-
modified operating systems. Full virtualization allows the guest OS to run unmodi-
fied and requires the presence of AMD64/Intel 64 processors which supports either
Intel* Virtualization Technology (Intel VT) or AMD* Virtualization (AMD-V)).
The primary component of the operating system that enables virtualization is a Hy-
pervisor (or virtual machine manager), which is a layer of software that runs directly
on server hardware. It controls platform resources, sharing them among multiple
VM Guests and their operating systems by presenting virtualized hardware inter-
faces to each VM Guest.
SUSE Linux Enterprise is an enterprise-class Linux server operating system that of-
fers two types of Hypervisors: Xen and KVM. Both Hypervisors support virtualization
on 64-bit x86-based hardware architectures. Both Xen and KVM support full virtual-
ization mode. In addition, Xen supports paravirtualized mode. SUSE Linux Enter-
prise with Xen or KVM acts as a virtualization host server (VHS) that supports VM
Guests with its own guest operating systems. The SUSE VM Guest architecture con-
sists of a Hypervisor and management components that constitute the VHS, which
runs many application-hosting VM Guests.
16 z 366 2017-05-27 13:21

In Xen, the management components run in a privileged VM Guest often called

Dom0. In KVM, where the Linux kernel acts as the hypervisor, the management
components run directly on the VHS.

Virtualization design provides many capabilities to your organization. Virtualization
of operating systems is used in many computing areas:
Server consolidation: Many servers can be replaced by one big physical

server, so hardware is consolidated, and Guest Operating Systems are con-
verted to virtual machine. It provides the ability to run legacy software on
new hardware.
Isolation: guest operating system can be fully isolated from the Host running
it. So if the virtual machine is corrupted, the Host system is not harmed.
Migration: A process to move a running virtual machine to another physical

machine. Live migration is an extended feature that allows this move without
disconnection of the client or the application.
Disaster recovery: Virtualized guests are less dependent on the hardware,

and the Host server provides snapshot features to be able to restore a
known running system without any corruption.
Dynamic load balancing: A migration feature that brings a simple way to

load-balance your service across your infrastructure.

Virtualization brings a lot of advantages while providing the same service as a hard-
ware server.
First, it reduces the cost of your infrastructure. Servers are mainly used to provide a
service to a customer, and a virtualized operating system can provide the same ser-
vice, with:
Less hardware: You can run several operating system on one host, so all
hardware maintenance will be reduced.
Less power/cooling: Less hardware means you do not need to invest more in
electric power, backup power, and cooling if you need more service.
17 z 366 2017-05-27 13:21

Save space: Your data center space will be saved because you do not need
more hardware servers (less servers than service running).
Less management: Using a VM Guest simplifies the administration of your in-

frastructure.
Agility and productivity: Virtualization provides migration capabilities, live mi-

gration and snapshots. These features reduce downtime, and bring an easy
way to move your service from one place to another without any service in-
terruption.

Guest operating systems are hosted on virtual machines in either full virtualization
(FV) mode or paravirtual (PV) mode. Each virtualization mode has advantages and
disadvantages.
Full virtualization mode lets virtual machines run unmodified operating sys-
tems, such as Windows* Server 2003. However, it also requires the VM Host
Server to support hardware-assisted virtualization technology, such as AMD*
Virtualization or Intel* Virtualization Technology.
Some guest operating systems hosted in full virtualization mode can be con-
figured to run the Novell* Virtual Machine Drivers instead of drivers originat-
ing from the operating system. Running virtual machine drivers improves
performance dramatically on guest operating systems, such as Windows
Server 2003. For more information, see Appendix A, Virtual Machine Drivers.
Paravirtual mode does not require the host computer to support hardware-
assisted virtualization technology, but does require the guest operating sys-
tem to be modified for the virtualization environment. Typically, operating
systems running in paravirtual mode enjoy better performance than those
requiring full virtualization mode.
Operating systems currently modified to run in paravirtual mode are called

paravirtualized operating systems and include SUSE Linux Enterprise Server
and NetWare® 6.5 SP8.
18 z 366 2017-05-27 13:21

VM Guests not only share CPU and memory resources of the host system, but also
the I/O subsystem. Because software I/O virtualization techniques deliver less per-
formance than bare metal, hardware solutions that deliver almost “native” perfor-
mance have been developed recently. SUSE Linux Enterprise Server supports the fol-
lowing I/O virtualization techniques:
Full Virtualization
Fully Virtualized (FV) drivers emulate widely supported real devices, which can
be used with an existing driver in the VM Guest. The guest is also called Hard-
ware Virtual Machine (HVM). Since the physical device on the VM Host Server
may differ from the emulated one, the hypervisor needs to process all I/O oper-
ations before handing them over to the physical device. Therefore all I/O opera-
tions need to traverse two software layers, a process that not only significantly
impacts I/O performance, but also consumes CPU time.
Paravirtualization
Paravirtualization (PV) allows direct communication between the hypervisor and
the VM Guest. With less overhead involved, performance is much better than
with full virtualization. However, paravirtualization requires either the guest op-
erating system to be modified to support the paravirtualization API or paravirtu-
alized drivers. See Section 7.1.1, “Availability of Paravirtualized Drivers” for a list
of guest operating systems supporting paravirtualization.
PVHVM
This type of virtualization enhances HVM (see Full Virtualization) with paravirtu-
alized (PV) drivers, and PV interrupt and timer handling.
VFIO
VFIO stands for Virtual Function I/O and is a new user-level driver framework for
Linux. It replaces the traditional KVM PCI Pass-Through device assignment. The
VFIO driver exposes direct device access to user space in a secure memory
(IOMMU) protected environment. With VFIO, a VM Guest can directly access
hardware devices on the VM Host Server (pass-through), avoiding performance
issues caused by emulation in performance critical paths. This method does not
allow to share devices—each device can only be assigned to a single VM Guest.
VFIO needs to be supported by the VM Host Server CPU, chipset and the
BIOS/EFI.
Compared to the legacy KVM PCI device assignment, VFIO has the following ad-
vantages:
Resource access is compatible with secure boot.
Device is isolated and its memory access protected.
19 z 366 2017-05-27 13:21

Offers a user space device driver with more flexible device ownership
model.
Is independent of KVM technology, and not bound to x86 architecture

only.
As of SUSE Linux Enterprise Server 12 SP2, the USB and PCI Pass-through meth-
ods of device assignment are considered deprecated and were superseded by
the VFIO model.
SR-IOV
The latest I/O virtualization technique, Single Root I/O Virtualization SR-IOV com-
bines the benefits of the aforementioned techniques—performance and the
ability to share a device with several VM Guests. SR-IOV requires special I/O de-
vices, that are capable of replicating resources so they appear as multiple sepa-
rate devices. Each such “pseudo” device can be directly used by a single guest.
However, for network cards for example the number of concurrent queues that
can be used is limited, potentially reducing performance for the VM Guest com-
pared to paravirtualized drivers. On the VM Host Server, SR-IOV must be sup-
ported by the I/O device, the CPU and chipset, the BIOS/EFI and the hypervi-
sor—for setup instructions see Section 14.10, “Adding a PCI Device to a VM
Guest”.
Important: Requirements for VFIO and SR-IOV
To be able to use the VFIO and SR-IOV features, the VM Host Server needs to fulfill
the following requirements:
IOMMU needs to be enabled in the BIOS/EFI.
For Intel CPUs, the Kernel parameter intel_iommu=on needs to be pro-

vided on the Kernel command line. For more information, see Book “Ad-
ministration Guide”, Chapter 11 “The Boot Loader GRUB 2”, Section 11.3.3.2
“Kernel Parameters Tab”.
The VFIO infrastructure needs to be available. This can be achieved by

loading the Kernel module vfio_pci . For more information, see Book
“Administration Guide”, Chapter 13 “The systemd Daemon”, Section 13.6.4
“Loading Kernel Modules”.
20 z 366 2017-05-27 13:21


This chapter introduces and explains the components and technologies you need to
understand to set up and manage a Xen-based virtualization environment.

The basic components of a Xen-based virtualization environment are the Xen hyper-
visor, the Dom0, any number of other VM Guests, and the tools, commands, and con-
figuration files that let you manage virtualization. Collectively, the physical computer
running all these components is called a VM Host Server because together these
components form a platform for hosting virtual machines.
The Xen Hypervisor

The Xen hypervisor, sometimes simply called a virtual machine monitor, is an
open source software program that coordinates the low-level interaction be-
tween virtual machines and physical hardware.
The Dom0
The virtual machine host environment, also called Dom0 or controlling domain,
is composed of several components, such as:
The SUSE Linux Enterprise operating system gives the administrator a

graphical and command line environment to manage the virtual ma-
chine host components and its virtual machines.
21 z 366 2017-05-27 13:21

Note
The term “Dom0” refers to a special domain that provides the man-
agement environment. This may be run either in graphical or in com-
mand line mode.
The xl tool stack based on the xenlight library (libxl). Use it to manage
Xen guest domains.
QEMU—an open source software that emulates a full computer system,

including a processor and various peripherals. It provides the ability to
host operating systems in both full virtualization or paravirtualization
mode.
Xen-Based Virtual Machines

A Xen-based virtual machine, also called a VM Guest or DomU , consists of the
following components:
At least one virtual disk that contains a bootable operating system. The
virtual disk can be based on a file, partition, volume, or other type of
block device.
A configuration file for each guest domain. It is a text file following the
syntax described in the manual page man 5 xl.conf .
Several network devices, connected to the virtual network provided by

the controlling domain.
Management Tools, Commands, and Configuration Files

There is a combination of GUI tools, commands, and configuration files to help
you manage and customize your virtualization environment.

The following graphic depicts a virtual machine host with four virtual machines. The
Xen hypervisor is shown as running directly on the physical hardware platform.
Note that the controlling domain is also a virtual machine, although it has several
additional management tasks compared to all the other virtual machines.
22 z 366 2017-05-27 13:21

FIGURE 2.1: XEN VIRTUALIZATION ARCHITECTURE
On the left, the virtual machine host’s Dom0 is shown running the SUSE Linux Enter-
prise operating system. The two virtual machines shown in the middle are running
paravirtualized operating systems. The virtual machine on the right shows a fully vir-
tual machine running an unmodified operating system, such as the latest version of
Microsoft Windows/Server.

23 z 366 2017-05-27 13:21


KVM is a full virtualization solution for the AMD64/Intel 64 and the z Systems archi-
tectures supporting hardware virtualization.
VM Guests (virtual machines), virtual storage, and virtual networks can be managed
with QEMU tools directly, or with the libvirt -based stack. The QEMU tools in-
clude qemu-system-ARCH , the QEMU monitor, qemu-img , and qemu-ndb . A
libvirt -based stack includes libvirt itself, along with libvirt -based applica-
tions such as virsh , virt-manager , virt-install , and virt-viewer .

This full virtualization solution consists of two main components:
A set of Kernel modules ( kvm.ko , kvm-intel.ko , and kvm-amd.ko ) that

provides the core virtualization infrastructure and processor-specific drivers.
A user space program ( qemu-system-ARCH ) that provides emulation for vir-

tual devices and control mechanisms to manage VM Guests (virtual ma-
chines).
The term KVM more properly refers to the Kernel level virtualization functionality,
but is in practice more commonly used to refer to the user space component.
24 z 366 2017-05-27 13:21

FIGURE 3.1: KVM VIRTUALIZATION ARCHITECTURE
Note: Hyper-V Emulation Support
QEMU can provide certain Hyper-V hypercalls for Windows* guests to partly em-
ulate a Hyper-V environment. This can be used to achieve better behavior for
Windows* guests that are Hyper-V enabled.
25 z 366 2017-05-27 13:21

Linux containers are a lightweight virtualization method to run multiple virtual units
(“containers”) simultaneously on a single host. This is similar to the chroot environ-
ment. Containers are isolated with kernel Control Groups (cgroups) and kernel
Namespaces.
Containers provide virtualization at the operating system level where the kernel con-
trols the isolated containers. This is unlike full virtualization solutions like Xen or
KVM where the processor simulates a complete hardware environment and controls
virtual machines.
Conceptually, containers can be seen as an improved chroot technique. The differ-

ence is that a chroot environment separates only the file system, whereas containers
go further and provide resource management and control via cgroups.
BENEFITS OF CONTAINERS
Isolating applications and operating systems through containers.
Providing nearly native performance as container manages allocation of re-

sources in real-time.
Controlling network interfaces and applying resources inside containers

through cgroups.
LIMITATIONS OF CONTAINERS
All containers run inside the host system's kernel and not with a different
kernel.
Only allows Linux “guest” operating systems.
Security depends on the host system. Container is not secure. If you need a
secure system, you can confine it using an AppArmor or SELinux profile.
Abstract
26 z 366 2017-05-27 13:21

libvirt is a library that provides a common API for managing popular

virtualization solutions, among them KVM, LXC, and Xen. The library pro-
vides a normalized management API for these virtualization solutions, al-
lowing a stable, cross-hypervisor interface for higher-level management
tools. The library also provides APIs for management of virtual networks
and storage on the VM Host Server. The configuration of each VM Guest is
stored in an XML file.
With libvirt you can also manage your VM Guests remotely. It sup-
ports TLS encryption, x509 certificates and authentication with SASL. This
enables managing VM Host Servers centrally from a single workstation, al-
leviating the need to access each VM Host Server individually.
Using the libvirt -based tools is the recommended way of managing
VM Guests. Interoperability between libvirt and libvirt -based ap-
plications has been tested and is an essential part of SUSE's support
stance.


The following libvirt-based tools for the command line are available on SUSE Linux
Enterprise Server. All tools are provided by packages carrying the tool's name.
virsh
A command line tool to manage VM Guests with similar functionality as the Vir-
tual Machine Manager. Allows you to change a VM Guest's status (start, stop,
pause, etc.), to set up new guests and devices, or to edit existing configurations.
virsh is also useful to script VM Guest management operations.
virsh takes the first argument as a command and further arguments as op-
tions to this command:
virsh [-c URI] command domain-id [OPTIONS]
27 z 366 2017-05-27 13:21

Like zypper , virsh can also be called without a command. In this case it
starts a shell waiting for your commands. This mode is useful when having to
run subsequent commands:
~> virsh -c qemu+ssh://wilber@mercury.example.com/system

Enter passphrase for key '/home/wilber/.ssh/id_rsa':
Welcome to virsh, the virtualization interactive terminal.
Type: 'help' for help with commands

'quit' to quit
virsh # hostname
mercury.example.com
virt-install
A command line tool for creating new VM Guests using the libvirt library. It
supports graphical installations via VNC or SPICE protocols. Given suitable com-
mand line arguments, virt-install can run completely unattended. This al-
lows for easy automation of guest installs. virt-install is the default instal-
lation tool used by the Virtual Machine Manager.

The following libvirt-based graphical tools are available on SUSE Linux Enterprise
Server. All tools are provided by packages carrying the tool's name.
Virtual Machine Manager ( virt-manager )

The Virtual Machine Manager is a desktop tool for managing VM Guests. It pro-
vides the ability to control the life cycle of existing machines (start/shutdown,
pause/resume, save/restore) and create new VM Guests. It allows managing vari-
ous types of storage and virtual networks. It provides access to the graphical
console of VM Guests with a built-in VNC viewer and can be used to view perfor-
mance statistics. virt-manager supports connecting to a local libvirtd ,
managing a local VM Host Server, or a remote libvirtd managing a remote
VM Host Server.
28 z 366 2017-05-27 13:21

To start the Virtual Machine Manager, enter virt-manager at the command

prompt.
Note
To disable automatic USB device redirection for VM Guest using spice, either
launch virt-manager with the --spice-disable-auto-usbredir pa-
rameter or run the following command to persistently change the default be-
havior:
tux > dconf write /org/virt-manager/virt-manager/console/auto-redirect false
virt-viewer
29 z 366 2017-05-27 13:21

A viewer for the graphical console of a VM Guest. It uses SPICE (configured by

default on the VM Guest) or VNC protocols and supports TLS and x509 certifi-
cates. VM Guests can be accessed by name, ID, or UUID. If the guest is not al-
ready running, the viewer can be told to wait until the guest starts, before at-
tempting to connect to the console. virt-viewer is not installed by default
and is available after installing the package virt-viewer .
Note
To disable automatic USB device redirection for VM Guest using spice, add an
empty filter using the --spice-usbredir-auto-redirect-filter='' pa-
rameter.
yast2 vm
A YaST module that simplifies the installation of virtualization tools and can set
up a network bridge:
6 Installation of Virtualization
30 z 366 2017-05-27 13:21

Components
6.1 Installing KVM
6.2 Installing Xen
6.4 Patterns
None of the virtualization tools is installed by default.
6.1 Installing KVM

To install KVM and KVM tools, proceed as follows:
1. Start YaST and choose Virtualization › Installing Hypervisor and Tools.
2. Select KVM server for a minimal installation of QEMU tools. Select KVM tools if
a libvirt -based management stack is also desired. Confirm with Accept.
3. To enable normal networking for the VM Guest, using a network bridge is

recommended. YaST offers to automatically configure a bridge on the VM
Host Server. Agree to do so by choosing Yes, otherwise choose No.
4. After the setup has been finished, you can start setting up VM Guests. Re-
booting the VM Host Server is not required.
6.2 Installing Xen

To install Xen and Xen tools, proceed as follows:
2. Select Xen server for a minimal installation of Xen tools. Select Xen tools if a
libvirt -based management stack is also desired. Confirm with Accept.
3. To enable normal networking for the VM Guest, using a network bridge is

recommended. YaST offers to automatically configure a bridge on the VM
Host Server. Agree to do so by choosing Yes, otherwise choose No.
4.
31 z 366 2017-05-27 13:21

After the setup has been finished, you need to reboot the machine with the
Xen kernel.
Tip: Default Boot Kernel
If everything works as expected, change the default boot kernel with YaST
and make the Xen-enabled kernel the default. For more information
about changing the default kernel, see Book “Administration Guide”, Chap-
ter 11 “The Boot Loader GRUB 2”, Section 11.3 “Configuring the Boot Loader
with YaST”.

To install containers, proceed as follows:
2. Select libvirt lxc daemon and confirm with Accept.
6.4 Patterns
It is possible using Zypper and patterns to install virtualization packages. Run the
command zypper in -t pattern PATTERN . Available patterns are:
KVM
kvm_server : sets up the KVM VM Host Server with QEMU tools for
management
kvm_tools : installs the libvirt tools for managing and monitoring

VM Guests
Xen
xen_server : sets up the Xen VM Host Server with Xen tools for man-
agement
32 z 366 2017-05-27 13:21

xen_tools : installs the libvirt tools for managing and monitoring VM

Guests
Containers
There is no pattern for containers; install the libvirt-daemon-lxc package.
7 Supported Guests, Hosts and

Features
7.2 Supported VM Host Servers for SUSE Linux Enterprise Server 12 SP2 VM
Guests
7.4 Feature Support
Supported virtualization limits for Xen and KVM are outlined in the Release Notes↗.

This section lists the support status for various guest operating systems virtualized
on top of SUSE Linux Enterprise Server 12 SP2. All guest operating systems are sup-
ported both fully-virtualized (“FV” in the following table) and paravirtualized (“PV” in
the following table) with two exceptions: Windows, which is only supported fully-vir-
tualized, and OES and NetWare operating systems, which are only supported on Xen
paravirtualized. All guest operating systems are supported both in 32-bit and 64-bit
flavors, unless stated otherwise (see NetWare).
TABLE 7.1: PARAVIRTUALIZED OS SUPPORT
33 z 366 2017-05-27 13:21

Operating System FV Support (Xen/KVM) PV Support (Xen)

SLES 10 SP4 Full Full
SLES 12 Full Full
SLED 12 SP2 Technology preview1 Technology preview1
SLED 12 SP3 Technology preview1 Technology preview1
OES 11 SP3 None Full2, 3
OES 2015 None Full2, 3
OES 2015 SP1 None Full2, 3
Netware 6.5 SP8 None Full (32-bit only)2
RHEL 5.11+ Full/best effort4 Full/best effort4
Windows Server 2008 SP2+ Full None
Windows Server 2008 R2 SP1+ Full None
Windows Server 2012+ Full None
Windows Server 2012 R2+ Full None
Windows Server 2016 Full None
Windows Vista SP2+ Best effort None
Windows 7 SP1+ Best effort None
Windows 8+ Best effort None
Windows 8.1+ Best effort None
Windows 10+ Best effort None
34 z 366 2017-05-27 13:21

1 Technology preview: The operating system has been tested to install and run suc-
cessfully. Bugs can be reported and will be tracked by SUSE Technical Services, but
no support commitments or service level agreements apply. Potential fixes and
patches will be evaluated for future inclusion.
2 You need a static IP address for each virtual machine running NetWare or OES.
3 OES can only be installed from a network installation source.
4 RedHat* guest operating systems are fully supported with Expanded Support. Oth-
erwise, they will be supported on a best-effort basis (fixes if reasonable).
7.1.1 Availability of Paravirtualized Drivers

To improve the performance of the guest operating system, paravirtualized drivers
are provided when available. Although they are not required, it is strongly recom-
mended to use them. The paravirtualized drivers are available as follows:
SUSE Linux Enterprise Server 12 / 12 SP1 / 12 SP2

Included in kernel
SUSE Linux Enterprise Server 11 / 11 SP1 / 11 SP2 / 11 SP3 / 11 SP4

Included in kernel
SUSE Linux Enterprise Server 10 SP4

Included in kernel
RedHat
Available in RedHat Enterprise Linux 5.4 and newer
Windows
SUSE has developed virtio-based drivers for Windows, which are available in the
Virtual Machine Driver Pack (VMDP). For more information, see
http://www.suse.com/products/vmdriverpack/↗.
7.2 Supported VM Host Servers for SUSE Linux

Enterprise Server 12 SP2 VM Guests
This section lists the support status of SUSE Linux Enterprise Server 12 SP2 running
as a guest on top of various virtualization hosts (Hypervisor). Both 32-bit and 64-bit
versions are supported for the host if available. The support status is defined as fol-
lows:
35 z 366 2017-05-27 13:21

Full support for all SUSE host systems and SUSE Linux Enterprise Server 12
SP2 VM Guests
Full support for SUSE Linux Enterprise Server 12 SP2 VM Guests on

third-party host systems
The following SUSE host operating systems are supported:
SUSE Linux Enterprise Server 11 SP4 (KVM/Xen)
The following third party host operating systems are supported:
KVM for IBM z Systems 1.1.0
PowerKVM
VMware ESX 5.5
VMware ESXi 6.0
Windows 2008 SP2+
Windows 2008 R2 SP1+
Windows 2012+
Windows 2012 R2+
Microsoft Windows 2016
Citrix XenServer 6.5
Oracle VM 3.3
The following host operating systems will be supported when released:
VMware ESXi 2016
Citrix XenServer 6.5
36 z 366 2017-05-27 13:21

Currently, SUSE only supports KVM full virtualization on AMD64/Intel 64 hosts and
on z Systems (only as Technology Preview). On the AMD64/Intel 64 architecture,
KVM is designed around hardware virtualization features included in AMD* (AMD-V)
and Intel* (VT-x) CPUs. It supports virtualization features of chipsets, and PCI de-
vices, such as an I/O Memory Mapping Unit (IOMMU) and Single Root I/O Virtualiza-
tion (SR-IOV).
On the AMD64/Intel 64 architecture, you can test whether your CPU supports hard-
ware virtualization with the following command:
egrep '(vmx|svm)' /proc/cpuinfo
If this command returns no output, your processor either does not support hard-
ware virtualization, or this feature has been disabled in the BIOS or Firmware.
The following Web sites identify AMD64/Intel 64 processors that support hardware
virtualization: http://ark.intel.com/Products/VirtualizationTechnology↗ (for Intel
CPUs), and http://products.amd.com/↗ (for AMD CPUs).
Note: KVM Kernel Modules Not Loading
The KVM kernel modules only load if the CPU hardware virtualization features are
available.
The general minimum hardware requirements for the VM Host Server are the same
as outlined in Book “Deployment Guide”, Chapter 2 “Installation on AMD64 and
Intel 64”, Section 2.1 “System Requirements for Operating Linux”. However, additional
RAM for each virtualized guest is needed. It should at least be the same amount that
is needed for a physical installation. It is also strongly recommended to have at least
one processor core or hyper-thread for each running guest.
7.4 Feature Support
7.4.1 Host (Dom0)

TABLE 7.2: FEATURE SUPPORT—HOST ( Dom0 )
37 z 366 2017-05-27 13:21

Features Xen
Network and block device hotplugging Yes
Physical CPU hotplugging No
Virtual CPU hotplugging Yes
Virtual CPU pinning Yes
Virtual CPU capping Yes
Intel* VT-x2: FlexPriority, FlexMigrate (migration constraints apply to dis-
Yes
similar CPU architectures)
Intel* VT-d2 (DMA remapping with interrupt filtering and queued invali-
Yes
dation)
AMD* IOMMU (I/O page table with guest-to-host physical address
Yes
translation)
Note: Adding or Removing Physical CPUs at Runtime Is Not Supported
The addition or removal of physical CPUs at runtime is not supported. However,

virtual CPUs can be added or removed for each VM Guest.
7.4.2 Paravirtualized Guest

TABLE 7.3: FEATURE SUPPORT—PARAVIRTUALIZED GUEST
38 z 366 2017-05-27 13:21

Features Xen
Virtual network and virtual block de-
Yes
vice hotplugging
Virtual CPU hotplugging Yes
Virtual CPU over-commitment Yes
Dynamic virtual memory resize Yes
VM save and restore Yes
Yes, between like virtual host systems with
VM live migration
similar resources
Advanced debugging with GDBC Yes
Dom0 metrics visible to VM Yes
Memory ballooning Yes
PCI pass-through Yes (Netware guests are excluded)
For live migration, both source and target system architectures need to match; that
is, the processors (AMD* or Intel*) must be the same. Unless CPU ID masking is
used, such as with Intel FlexMigration, the target should feature the same processor
revision or a more recent processor revision than the source. If VMs are moved
among different systems, the same rules apply for each move. To avoid failing opti-
mized code at runtime or application start-up, source and target CPUs need to ex-
pose the same processor extensions. Xen exposes the physical CPU extensions to
the VMs transparently. To summarize, guests can be 32-bit or 64-bit, but the VHS
must be identical.
Note: Intel FlexMigration
For machines that support Intel FlexMigration, CPU-ID masking and faulting allow
more flexibility in cross-CPU migration.
7.4.3 Fully Virtualized Guest

TABLE 7.4: FEATURE SUPPORT—FULLY VIRTUALIZED GUEST
39 z 366 2017-05-27 13:21

Features Xen KVM

Virtual network and virtual
Yes Yes
block device hotplugging
Virtual CPU hotplugging No No
Virtual CPU over-commit-
Yes Yes
ment
Dynamic virtual memory re-
Yes Yes
size
VM save and restore Yes Yes
Yes between like virtual host systems with
VM Live Migration similar resources (that is, from 32-bit to 32-bit, Yes
64-bit to 64-bit)
VM snapshot Yes Yes
Advanced debugging with
Yes Yes
GDBC
Dom0 metrics visible to VM Yes Yes
PCI pass-through Yes Yes
Note: Windows Guest
Hotplugging of virtual network and virtual block devices, and resizing, shrinking,
and restoring dynamic virtual memory are supported in Xen and KVM only if PV
drivers are being used (VMDP).
For KVM, a detailed description of supported limits, features, recommended settings

and scenarios, and other useful information is maintained in kvm-supported.txt .
This file is part of the KVM package and can be found in
/usr/share/doc/packages/kvm .
Part II Managing Virtual Machines

with libvirt
40 z 366 2017-05-27 13:21


The communication between the virtualization solutions (KVM, Xen, LXC) and
the libvirt API is managed by the daemon libvirtd, which needs to run on the VM
Host Server. libvirt client applications such as virt-manager, possibly running on
a remote machine, communicate with libvirtd running on the VM …
A VM Guest consists of an image containing an operating system and data files
and a configuration file describing the VM Guest's virtual hardware resources.
VM Guests are hosted on and controlled by the VM Host Server. This section
provides generalized instructions for installing a VM Guest.

Most management tasks, such as starting or stopping a VM Guest, can either be
done using the graphical application Virtual Machine Manager or on the com-
mand line using virsh . Connecting to the graphical console via VNC is only
possible from a graphical user interface.

Managing several VM Host Servers, each hosting multiple VM Guests, quickly be-
comes difficult. One benefit of libvirt is the ability to connect to several VM
Host Servers at once, providing a single interface to manage all VM Guests and
to connect to their graphical console.
12 Managing Storage
When managing a VM Guest on the VM Host Server itself, you can access the
complete file system of the VM Host Server to attach or create virtual hard disks
or to attach existing images to the VM Guest. However, this is not possible when
managing VM Guests from a remote host. For this reason, libvirt…
This chapter introduces common networking configurations supported by
libvirt . It does not depend on the hypervisor used. It is valid for all hypervi-
sors supported by libvirt , such as KVM or Xen. These setups can be achieved
using both the graphical interface of Virtual Machine Manager and the com-
mand line tool virsh .
41 z 366 2017-05-27 13:21

Virtual Machine Manager's Details view offers in-depth information about the
VM Guest's complete configuration and hardware equipment. Using this view,
you can also change the guest configuration or add and modify virtual hard-
ware. To access this view, open the guest's console in Virtual Machine Manager
and either choose View › Details from the menu, or click Show virtual hardware
details in the toolbar.
The communication between the virtualization solutions (KVM, Xen, LXC) and the lib-
virt API is managed by the daemon libvirtd , which needs to run on the VM Host
Server. libvirt client applications such as virt-manager, possibly running on a remote
machine, communicate with libvirtd running on the VM Host Server, which ser-
vices the request using native hypervisor APIs. Use the following commands to start
and stop libvirtd or check its status:
tux > sudo systemctl start libvirtd
tux > sudo systemctl status libvirtd

libvirtd.service - Virtualization daemon
Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled)
Active: active (running) since Mon 2014-05-12 08:49:40 EDT; 2s ago
[...]
tux > sudo systemctl stop libvirtd
tux > sudo systemctl status libvirtd

[...]
Active: inactive (dead) since Mon 2014-05-12 08:51:11 EDT; 4s ago
[...]
To automatically start libvirtd at boot time, either activate it using the YaST Ser-
vices Manager module or by entering the following command:
tux > sudo systemctl enable libvirtd
42 z 366 2017-05-27 13:21

Important: Conflicting Services: libvirtd and xendomains
In case libvirtd fails to start, check if the service xendomains is loaded:
tux > systemctl is-active xendomains

active
In case the command returns active , you need to stop xendomains before
you will be able to start the libvirtd daemon. In case you want libvirtd to
also start after rebooting, you should additionally prevent xendomains from
starting automatically by disabling the service:
tux > sudo systemctl stop xendomains

tux > sudo systemctl disable xendomains
tux > sudo systemctl start libvirtd
xendomains and libvirtd provide the same service and when used in parallel
may interfere with one another. As an example, xendomains may attempt to
start a domU already started by libvirtd .
9.2 Installing from the Command Line with virt-install
A VM Guest consists of an image containing an operating system and data files and a
configuration file describing the VM Guest's virtual hardware resources. VM Guests
are hosted on and controlled by the VM Host Server. This section provides general-
ized instructions for installing a VM Guest.
43 z 366 2017-05-27 13:21

Virtual machines have few if any requirements above those required to run the op-
erating system. If the operating system has not been optimized for the virtual ma-
chine host environment, it can only run on hardware-assisted virtualization com-
puter hardware, in full virtualization mode, and requires specific device drivers to be
loaded. The hardware that is presented to the VM Guest depends on the configura-
tion of the host.
Note: Virtual Machine Architectures
The virtual machine host runs only on AMD64 and Intel 64. Additionally, KVM for
z Systems is included on SUSE Linux Enterprise Server as a technology preview. It
does not run on other system architectures such as POWER. A 64-bit virtual ma-
chine host can, however, run both 32-bit and 64-bit VM Guests.
You should be aware of any licensing issues related to running a single licensed copy
of an operating system on multiple virtual machines. Consult the operating system
license agreement for more information.

The New VM wizard helps you through the steps required to create a virtual machine
and install its operating system. There are two ways to start it: Within Virtual Ma-
chine Manager, either click Create New Virtual Machine or choose File › New Virtual
Machine. Alternatively, start YaST and choose Virtualization › Create Virtual Machines
for Xen and KVM.
1. Start the New VM wizard either from YaST or Virtual Machine Manager.
2. Choose an installation source—either a locally available media or a network

installation source. If you want to set up your VM Guest from an existing im-
age, choose import existing disk image.
On a VM Host Server running the Xen hypervisor, you can choose whether
to install a paravirtualized or a fully virtualized guest. The respective option
is available under Architecture Options. Depending on this choice, not all in-
stallation options may be available.
3. Depending on your choice in the previous step, you need to provide the fol-
lowing data:
Local Installation Media (ISO image or CDROM)
44 z 366 2017-05-27 13:21

Specify the path on the VM Host Server to an iso image containing the
installation data. If it is available as a volume in a libvirt storage pool,
you can also select it using Browse. For more information, see Chap-
ter 12, Managing Storage.
Alternatively, choose a physical CD-ROM or DVD inserted in the optical
drive of the VM Host Server.
Network Installation (HTTP, FTP, or NFS)

Provide the URL pointing to the installation source. Valid URL prefixes
are, for example, ftp:// , http:// , https:// , and nfs:// .
Under URL Options, provide a path to an auto-installation file (AutoYaST
or Kickstart, for example) and Kernel parameters. Having provided a
URL, the operating system should be automatically detected correctly. If
this is not the case, deselect Automatically Detect Operating System Based
on Install-Media and manually select the OS Type and Version.
Network Boot (PXE)

When booting via PXE, you only need to provide the OS Type and the
Version.
Import Existing Disk Image

To set up the VM Guest from an existing image, you need to specify the
path on the VM Host Server to the image. If it is available as a volume in
a libvirt storage pool, you can also select it using Browse. For more infor-
mation, see Chapter 12, Managing Storage.
4. Choose the memory size and number of CPUs for the new virtual machine.
5. This step is omitted when Import an Existing Image is chosen in the first step.
Set up a virtual hard disk for the VM Guest. Either create a new disk image or
choose an existing one from a storage pool (for more information, see
Chapter 12, Managing Storage). If you choose to create a disk, a qcow2 im-
age will be created. By default, it is stored under
/var/lib/libvirt/images .
Setting up a disk is optional. If you are running a live system directly from
CD or DVD, for example, you can omit this step by deactivating Enable Stor-
age for this Virtual Machine.
6. On the last screen of the wizard, specify the name for the virtual machine. To
be offered the possibility to review and make changes to the virtualized
hardware selection, activate Customize configuration before install. Find op-
tions to specify the network device under Network Selection.
Click Finish.
45 z 366 2017-05-27 13:21

7. (Optional) If you kept the defaults in the previous step, the installation will
now start. If you selected Customize configuration before install, a VM Guest
configuration dialog opens. For more information about configuring VM
Guests, see Chapter 14, Configuring Virtual Machines.
When you are done configuring, click Begin Installation.
Tip: Passing Key Combinations to Virtual Machines
The installation starts in a Virtual Machine Manager console window. Some key
combinations, such as Ctrl – Alt – F1 , are recognized by the VM Host Server but
are not passed to the virtual machine. To bypass the VM Host Server, Virtual Ma-
chine Manager provides the “sticky key ” functionality. Pressing Ctrl , Alt , or
Shift three times makes the key sticky, then you can press the remaining keys to
pass the combination to the virtual machine.
For example, to pass Ctrl – Alt – F2 to a Linux virtual machine, press Ctrl
three times, then press Alt – F2 . You can also press Alt three times, then press
Ctrl – F2 .
The sticky key functionality is available in the Virtual Machine Manager during
and after installing a VM Guest.
9.2 Installing from the Command Line with

virt-install
virt-install is a command line tool that helps you create new virtual machines
using the libvirt library. It is useful if you cannot use the graphical user interface,
or need to automatize the process of creating virtual machines.
virt-install is a complex script with a lot of command line switches. The follow-
ing are required. For more information, see the man page of virt-install (1).
General Options
--name vm_guest_name : Specify the name of the new virtual ma-

chine. The name must be unique across all guests known to the hypervi-
sor on the same connection. It is used to create and name the guest’s
configuration file and you can access the guest with this name from
virsh . Alphanumeric and _-.:+ characters are allowed.
46 z 366 2017-05-27 13:21

--memory required_memory : Specify the amount of memory to allo-

cate for the new virtual machine in megabytes.
--vcpus number_of_cpus : Specify the number of virtual CPUs. For

best performance, the number of virtual processors should be less than
or equal to the number of physical processors.
Virtualization Type
--paravirt : Set up a paravirtualized guest. This is the default if the VM

Host Server supports paravirtualization and full virtualization.
--hvm : Set up a fully virtualized guest.
--virt-type hypervisor : Specify the hypervisor. Supported values

are kvm , xen , or lxc .
Guest Storage
Specify one of --disk , --filesystem or --nodisks the type of the storage
for the new virtual machine. For example, --disk size=10 creates 10 GB disk
in the default image location for the hypervisor and uses it for the VM Guest.
--filesystem /export/path/on/vmhost specifies the directory on the VM
Host Server to be exported to the guest. And --nodisks sets up a VM Guest
without a local storage (good for Live CDs).
Installation Method
Specify the installation method using one of --location , --cdrom , --pxe ,
--import , or --boot .
Accessing the Installation

Use the --graphics value option to specify how to access the installation.
SUSE Linux Enterprise Server supports the values vnc or none .
If using vnc virt-install tries to launch virt-viewer . If it is not installed
or cannot be run, connect to the VM Guest manually with you preferred viewer.
To explicitly prevent virt-install from launching the viewer use
--noautoconsole . To define a password for accessing the VNC session, use
the following syntax: --graphics vnc,password= PASSWORD .
In case you are using --graphics none , you can access the VM Guest through
operating system supported services, such as SSH or VNC. Refer to the operat-
ing system installation manual on how to set up these services in the installation
system.
EXAMPLE 9.1: EXAMPLE OF A virt-install COMMAND LINE
47 z 366 2017-05-27 13:21

The following command line example creates a new SUSE Linux Enterprise Desk-
top 12 virtual machine with a virtio accelerated disk and network card. It creates
a new 10 GB qcow2 disk image as a storage, the source installation media being
the host CD-ROM drive. It will use VNC graphics, and it will auto-launch the
graphical client.
KVM
virt-install --connect qemu:///system --virt-type kvm --name sled12 \

--memory 1024 --disk size=10 --cdrom /dev/cdrom --graphics vnc \
--os-variant sled12
Xen
virt-install --connect xen:// --virt-type xen --name sled12 \

--memory 1024 --disk size=10 --cdrom /dev/cdrom --graphics vnc \
--os-variant sled12

This section provides instructions for operations exceeding the scope of a normal in-
stallation, such as including modules and extensions packages.
9.3.1 Memory Ballooning with Windows Guests

Memory ballooning is a method to change the amount of memory used by VM
Guest at runtime. Both the KVM and Xen hypervisors provide this method, but it
needs to be supported by the guest as well.
While openSUSE and SLE-based guests support memory ballooning, Windows

guests need the Virtual Machine Driver Pack (VMDP)↗ to provide ballooning. To set
the maximum memory greater than the initial memory configured for Windows
guests, follow these steps:
1. Install the Windows guest with the maximum memory equal or less than the
initial value.
2. Install the Virtual Machine Driver Pack in the Windows guest to provide re-
quired drivers.
3. Shut down the Windows guest.
48 z 366 2017-05-27 13:21

4. Reset the maximum memory of the Windows guest to the required value.
5. Start the Windows guest again.
9.3.2 Including Add-On Products in the Installation

Some operating systems such as SUSE Linux Enterprise Server offer to include
add-on products in the installation process. In case the add-on product installation
source is provided via network, no special VM Guest configuration is needed. If it is
provided via CD/DVD or ISO image, it is necessary to provide the VM Guest installa-
tion system with both, the standard installation medium and an image for the
add-on product.
In case you are using the GUI-based installation, select Customize Configuration Be-
fore Install in the last step of the wizard and add the add-on product ISO image via
Add Hardware › Storage. Specify the path to the image and set the Device Type to
CD-ROM.
If installing from the command line, you need to set up the virtual CD/DVD drives
with the --disk parameter rather than with --cdrom . The device that is specified
first is used for booting. The following example will install SUSE Linux Enterprise
Server 12 plus SDK:
virt-install --name sles12+sdk --memory 1024 --disk size=10 \

--disk /virt/iso/SLES12.iso,device=cdrom \
--disk /virt/iso/SLES12_SDK.iso,device=cdrom \
--graphics vnc --os-variant sles12

10.3 Changing a VM Guest's State: Start, Stop, Pause
10.4 Saving and Restoring the State of a VM Guest
49 z 366 2017-05-27 13:21


10.8 Monitoring
Most management tasks, such as starting or stopping a VM Guest, can either be

done using the graphical application Virtual Machine Manager or on the command
line using virsh . Connecting to the graphical console via VNC is only possible from
a graphical user interface.
Note: Managing VM Guests on a Remote VM Host Server
If started on a VM Host Server, the libvirt tools Virtual Machine Manager,

virsh , and virt-viewer can be used to manage VM Guests on the host. How-
ever, it is also possible to manage VM Guests on a remote VM Host Server. This
requires configuring remote access for libvirt on the host. For instructions,
see Chapter 11, Connecting and Authorizing.
To connect to such a remote host with Virtual Machine Manager, you need to set
up a connection as explained in Section 11.2.2, “Managing Connections with Vir-
tual Machine Manager”. If connecting to a remote host using virsh or
virt-viewer , you need to specify a connection URI with the parameter -c (for
example, virsh -c qemu+tls://saturn.example.com/system or
virsh -c xen+ssh:// ). The form of connection URI depends on the connec-
tion type and the hypervisor—see Section 11.2, “Connecting to a VM Host Server”
for details.
Examples in this chapter are all listed without a connection URI.

The VM Guest listing shows all VM Guests managed by libvirt on a VM Host
Server.
10.1.1 Listing VM Guests with Virtual Machine Manager

The main window of the Virtual Machine Manager lists all VM Guests for each VM
Host Server it is connected to. Each VM Guest entry contains the machine's name, its
status (Running, Paused, or Shutoff) displayed as an icon and literally, and a CPU us-
age bar.
50 z 366 2017-05-27 13:21

10.1.2 Listing VM Guests with virsh

Use the command virsh list to get a list of VM Guests:
List all running guests
virsh list
List all running and inactive guests
virsh list --all
For more information and further options, see virsh help list or
man 1 virsh .

VM Guests can be accessed via a VNC connection (graphical console) or, if supported
by the guest operating system, via a serial console.
10.2.1 Opening a Graphical Console

Opening a graphical console to a VM Guest lets you interact with the machine like a
physical host via a VNC connection. If accessing the VNC server requires authentica-
tion, you are prompted to enter a user name (if applicable) and a password.
When you click into the VNC console, the cursor is “grabbed” and cannot be used
outside the console anymore. To release it, press Alt – Ctrl .
Tip: Seamless (Absolute) Cursor Movement
To prevent the console from grabbing the cursor and to enable seamless cursor
movement, add a tablet input device to the VM Guest. See Section 14.5, “Enabling
Seamless and Synchronized Mouse Pointer Movement” for more information.
51 z 366 2017-05-27 13:21

Certain key combinations such as Ctrl – Alt – Del are interpreted by the host sys-
tem and are not passed to the VM Guest. To pass such key combinations to a VM
Guest, open the Send Key menu from the VNC window and choose the desired key
combination entry. The Send Key menu is only available when using Virtual Machine
Manager and virt-viewer . With Virtual Machine Manager, you can alternatively
use the “sticky key ” feature as explained in Tip: Passing Key Combinations to Virtual
Machines.
Note: Supported VNC Viewers
Principally all VNC viewers can connect to the console of a VM Guest. However, if
you are using SASL authentication and/or TLS/SSL connection to access the guest,
the options are limited. Common VNC viewers such as tightvnc or tigervnc
support neither SASL authentication nor TLS/SSL. The only supported alternative
to Virtual Machine Manager and virt-viewer is vinagre .
10.2.1.1 Opening a Graphical Console with Virtual Machine

Manager
1. In the Virtual Machine Manager, right-click a VM Guest entry.
2. Choose Open from the pop-up menu.
10.2.1.2 Opening a Graphical Console with virt-viewer

virt-viewer is a simple VNC viewer with added functionality for displaying VM
Guest consoles. For example, it can be started in “wait” mode, where it waits for a
VM Guest to start before it connects. It also supports automatically reconnecting to a
VM Guest that is rebooted.
virt-viewer addresses VM Guests by name, by ID or by UUID. Use virsh

list --all to get this data.
To connect to a guest that is running or paused, use either the ID, UUID, or name.
VM Guests that are shut off do not have an ID—you can only connect to them by
UUID or name.
Connect to guest with the ID 8
virt-viewer 8
52 z 366 2017-05-27 13:21

Connect to the inactive guest named sles12 ; the connection window will open
once the guest starts
virt-viewer --wait sles12
With the --wait option, the connection will be upheld even if the VM Guest is
not running at the moment. When the guest starts, the viewer will be launched.
For more information, see virt-viewer --help or man 1 virt-viewer .
Note: Password Input on Remote connections with SSH
When using virt-viewer to open a connection to a remote host via SSH, the
SSH password needs to be entered twice. The first time for authenticating with
libvirt , the second time for authenticating with the VNC server. The second
password needs to be provided on the command line where virt-viewer was
started.
10.2.2 Opening a Serial Console

Accessing the graphical console of a virtual machine requires a graphical environ-
ment on the client accessing the VM Guest. As an alternative, virtual machines man-
aged with libvirt can also be accessed from the shell via the serial console and
virsh . To open a serial console to a VM Guest named “sles12”, run the following
command:
virsh console sles12
virsh console takes two optional flags: --safe ensures exclusive access to the
console, --force disconnects any existing sessions before connecting. Both fea-
tures need to be supported by the guest operating system.
Being able to connect to a VM Guest via serial console requires that the guest oper-
ating system supports serial console access and is properly supported. Refer to the
guest operating system manual for more information.
53 z 366 2017-05-27 13:21

Tip: Enabling Serial Console Access for SUSE Linux Enterprise and
openSUSE Guests
Serial console access in SUSE Linux Enterprise and openSUSE is disabled by de-
fault. To enable it, proceed as follows:
SLES 12 / openSUSE
Launch the YaST Boot Loader module and switch to the Kernel Parameters
tab. Add console=ttyS0 to the field Optional Kernel Command Line Parame-
ter.
SLES 11
Launch the YaST Boot Loader module and select the boot entry for which to
activate serial console access. Choose Edit and add console=ttyS0 to the
field Optional Kernel Command Line Parameter. Additionally, edit
/etc/inittab and uncomment the line with the following content:
#S0:12345:respawn:/sbin/agetty -L 9600 ttyS0 vt102
10.3 Changing a VM Guest's State: Start, Stop,

Pause
Starting, stopping or pausing a VM Guest can be done with either Virtual Machine
Manager or virsh . You can also configure a VM Guest to be automatically started
when booting the VM Host Server.
When shutting down a VM Guest, you may either shut it down gracefully, or force
the shutdown. The latter is equivalent to pulling the power plug on a physical host
and is only recommended if there are no alternatives. Forcing a shutdown may
cause file system corruption and loss of data on the VM Guest.
54 z 366 2017-05-27 13:21

Tip: Graceful Shutdown
To be able to perform a graceful shutdown, the VM Guest must be configured to

support ACPI. If you have created the guest with the Virtual Machine Manager,
ACPI should be available in the VM Guest.
Depending on the guest operating system, availability of ACPI may not be suffi-
cient to perform a graceful shutdown. It is strongly recommended to test shutting
down and rebooting a guest before using it in production. openSUSE or SUSE
Linux Enterprise Desktop, for example, can require PolKit authorization for shut-
down and reboot. Make sure this policy is turned off on all VM Guests.
If ACPI was enabled during a Windows XP/Windows Server 2003 guest installa-
tion, turning it on in the VM Guest configuration only is not sufficient. For more
information, see:
https://support.microsoft.com/en-us/kb/314088↗
https://support.microsoft.com/en-us/kb/309283↗
Regardless of the VM Guest's configuration, a graceful shutdown is always possi-

ble from within the guest operating system.
10.3.1 Changing a VM Guest's State with Virtual Machine

Manager
Changing a VM Guest's state can be done either from Virtual Machine Manager's
main window, or from a VNC window.
PROCEDURE 10.1: STATE CHANGE FROM THE VIRTUAL MACHINE MANAGER WINDOW
1. Right-click a VM Guest entry.
2. Choose Run, Pause, or one of the Shutdown options from the pop-up menu.
PROCEDURE 10.2: STATE CHANGE FROM THE VNC WINDOW
1. Open a VNC Window as described in Section 10.2.1.1, “Opening a Graphical

Console with Virtual Machine Manager”.
2. Choose Run, Pause, or one of the Shut Down options either from the toolbar
or from the Virtual Machine menu.
55 z 366 2017-05-27 13:21

10.3.1.1 Automatically Starting a VM Guest

You can automatically start a guest when the VM Host Server boots. This feature is
not enabled by default and needs to be enabled for each VM Guest individually.
There is no way to activate it globally.
1. Double-click the VM Guest entry in Virtual Machine Manager to open its con-
sole.
2. Choose View › Details to open the VM Guest configuration window.
3. Choose Boot Options and check Start virtual machine on host boot up.
4. Save the new configuration with Apply.
10.3.2 Changing a VM Guest's State with virsh

In the following examples, the state of a VM Guest named “sles12” is changed.
Start
virsh start sles12
Pause
virsh suspend sles12
Reboot
virsh reboot sles12
Graceful shutdown
virsh shutdown sles12
Force shutdown
virsh destroy sles12
Turn on automatic start
virsh autostart sles12
56 z 366 2017-05-27 13:21

Turn off automatic start
virsh autostart --disable sles12
10.4 Saving and Restoring the State of a VM

Guest
Saving a VM Guest preserves the exact state of the guest’s memory. The operation is
similar to hibernating a computer. A saved VM Guest can be quickly restored to its
previously saved running condition.
When saved, the VM Guest is paused, its current memory state is saved to disk, and
then the guest is stopped. The operation does not make a copy of any portion of the
VM Guest’s virtual disk. The amount of time taken to save the virtual machine de-
pends on the amount of memory allocated. When saved, a VM Guest’s memory is re-
turned to the pool of memory available on the VM Host Server.
The restore operation loads a VM Guest’s previously saved memory state file and
starts it. The guest is not booted but instead resumed at the point where it was pre-
viously saved. The operation is similar to coming out of hibernation.
The VM Guest is saved to a state file. Make sure there is enough space on the parti-
tion you are going to save to. For an estimation of the file size in megabytes to be ex-
pected, issue the following command on the guest:
free -mh | awk '/^Mem:/ {print $3}'
Warning: Always Restore Saved Guests
After using the save operation, do not boot or start the saved VM Guest. Doing so
would cause the machine's virtual disk and the saved memory state to get out of
synchronization. This can result in critical errors when restoring the guest.
To be able to work with a saved VM Guest again, use the restore operation. If you
used virsh to save a VM Guest, you cannot restore it using Virtual Machine
Manager. In this case, make sure to restore using virsh .
57 z 366 2017-05-27 13:21

Important: Only for VM Guests with Disk Types raw , qcow2 , qed
Saving and restoring VM Guests is only possible if the VM Guest is using a virtual
disk of the type raw ( .img ), qcow2, or qed .
10.4.1 Saving/Restoring with Virtual Machine Manager

PROCEDURE 10.3: SAVING A VM GUEST
1. Open a VNC connection window to a VM Guest. Make sure the guest is run-
ning.
2. Choose Virtual Machine › Shutdown › Save.
PROCEDURE 10.4: RESTORING A VM GUEST
1. Open a VNC connection window to a VM Guest. Make sure the guest is not
running.
2. Choose Virtual Machine › Restore.
If the VM Guest was previously saved using Virtual Machine Manager, you
will not be offered an option to Run the guest. However, note the caveats on
machines saved with virsh outlined in Warning: Always Restore Saved
Guests.
10.4.2 Saving and Restoring with virsh

Save a running VM Guest with the command virsh save and specify the file
which it is saved to.
Save the guest named opensuse13
virsh save opensuse13 /virtual/saves/opensuse13.vmsav
Save the guest with the ID 37
virsh save 37 /virtual/saves/opensuse13.vmsave
To restore a VM Guest, use virsh restore :
58 z 366 2017-05-27 13:21

virsh restore /virtual/saves/opensuse13.vmsave

VM Guest snapshots are snapshots of the complete virtual machine including the
state of CPU, RAM, and the content of all writable disks. To use virtual machine snap-
shots, you must have at least one non-removable and writable block device using
the qcow2 disk image format.
Note
Snapshots are supported on KVM VM Host Servers only.
Snapshots let you restore the state of the machine at a particular point in time. This
is for example useful to undo a faulty configuration or the installation of a lot of
packages. It is also helpful for testing purposes, as it allows you to go back to a de-
fined state at any time.
Snapshots can be taken either from running guests or from a guest currently not
running. Taking a screenshot from a guest that is shut down ensures data integrity.
In case you want to create a snapshot from a running system, be aware that the
snapshot only captures the state of the disk(s), not the state of the memory. There-
fore you need to ensure that:
All running programs have written their data to the disk. If you are unsure,
terminate the application and/or stop the respective service.
Buffers have been written to disk. This can be achieved by running the com-
mand sync on the VM Guest.
Starting a snapshot reverts the machine back to the state it was in when the snap-
shot was taken. Any changes written to the disk after that point in time will be lost
when starting the snapshot.
Starting a snapshot will restore the machine to the state (shut off or running) it was
in when the snapshot was taken. After starting a snapshot that was created while
the VM Guest was shut off, you will need to boot it.
10.5.1 Creating and Managing Snapshots with Virtual

Machine Manager
59 z 366 2017-05-27 13:21

To open the snapshot management view in Virtual Machine Manager, open the VNC
window as described in Section 10.2.1.1, “Opening a Graphical Console with Virtual
Machine Manager”. Now either choose View › Snapshots or click Manage VM Snap-
shots in the toolbar.
The list of existing snapshots for the chosen VM Guest is displayed in the left-hand
part of the window. The snapshot that was last started is marked with a green tick.
The right-hand part of the window shows details of the snapshot currently marked
in the list. These details include the snapshot's title and time stamp, the state of the
VM Guest at the time the snapshot was taken and a description. Snapshots of run-
ning guests also include a screenshot. The Description can be changed directly from
this view. Other snapshot data cannot be changed.
10.5.1.1 Creating a Snapshot

To take a new snapshot of a VM Guest, proceed as follows:
1. Shut down the VM Guest in case you want to create a snapshot from a guest
that is not running.
2. Click Add in the bottom left corner of the VNC window.
60 z 366 2017-05-27 13:21

The window Create Snapshot opens.
3. Provide a Name and, optionally, a description. The name cannot be changed

after the snapshot has been taken. To be able to identify the snapshot later
easily, use a “speaking name”.
4. Confirm with Finish.
10.5.1.2 Deleting a Snapshot

To delete a snapshot of a VM Guest, proceed as follows:
1. Click Delete in the bottom left corner of the VNC window.
2. Confirm the deletion with Yes.
10.5.1.3 Starting a Snapshot

To start a snapshot, proceed as follows:
1. Click Run in the bottom left corner of the VNC window.
2. Confirm the start with Yes.
10.5.2 Creating and Managing Snapshots with virsh

To list all existing snapshots for a domain ( admin_server in the following), run the
snapshot-list command:
tux > virsh snapshot-list

Name Creation Time State
------------------------------------------------------------
Basic installation incl. SMT finished 2013-09-18 09:45:29 +0200 shutoff
Basic installation incl. SMT for CLOUD3 2013-12-11 15:11:05 +0100 shutoff
Basic installation incl. SMT for CLOUD3-HA 2014-03-24 13:44:03 +0100 shutoff
Basic installation incl. SMT for CLOUD4 2014-07-07 11:27:47 +0200 shutoff
Beta1 Running 2013-07-12 12:27:28 +0200 shutoff
Beta2 prepared 2013-07-12 17:00:44 +0200 shutoff
Beta2 running 2013-07-29 12:14:11 +0200 shutoff
Beta3 admin node deployed 2013-07-30 16:50:40 +0200 shutoff
Beta3 prepared 2013-07-30 17:07:35 +0200 shutoff
Beta3 running 2013-09-02 16:13:25 +0200 shutoff
Cloud2 GM running 2013-12-10 15:44:58 +0100 shutoff
CLOUD3 RC prepared 2013-12-20 15:30:19 +0100 shutoff
CLOUD3-HA Build 680 prepared 2014-03-24 14:20:37 +0100 shutoff
CLOUD3-HA Build 796 installed (zypper up) 2014-04-14 16:45:18 +0200 shutoff
61 z 366 2017-05-27 13:21

GMC2 post Cloud install 2013-09-18 10:53:03 +0200 shutoff

GMC2 pre Cloud install 2013-09-18 10:31:17 +0200 shutoff
GMC2 prepared (incl. Add-On Installation) 2013-09-17 16:22:37 +0200 shutoff
GMC_pre prepared 2013-09-03 13:30:38 +0200 shutoff
OS + SMT + eth[01] 2013-06-14 16:17:24 +0200 shutoff
OS + SMT + Mirror + eth[01] 2013-07-30 15:50:16 +0200 shutoff
The snapshot that was last started is shown with the

snapshot-current command:
tux > virsh snapshot-current --name admin_server

Basic installation incl. SMT for CLOUD4
Details about a particular snapshot can be obtained by running the

snapshot-info command:
tux > virsh snapshot-info sles "Basic installation incl. SMT for CLOUD4"
Name: Basic installation incl. SMT for CLOUD4
Domain: admin_server
Current: yes
State: shutoff
Location: internal
Parent: Basic installation incl. SMT for CLOUD3-HA
Children: 0
Descendants: 0
Metadata: yes
10.5.2.1 Creating a Snapshot

To take a new snapshot of a VM Guest currently not running, use the
snapshot-create-as command as follows:
virsh snapshot-create-as --domain admin_server 1 --name "Snapshot 1" 2 \

--description "First snapshot" 3
1 Domain name. Mandatory.

2 Name of the snapshot. It is recommended to use a “speaking name”, since that
makes it easier to identify the snapshot. Mandatory.
3 Description for the snapshot. Optional.
To take a snapshot of a running VM Guest, you need to specify the --live parame-
ter:
virsh snapshot-create-as --domain admin_server --name "Snapshot 2" \

--description "First live snapshot" --live
Refer to the SNAPSHOT COMMANDS section in man 1 virsh for more details.
10.5.2.2 Deleting a Snapshot
62 z 366 2017-05-27 13:21

To delete a snapshot of a VM Guest, use the snapshot-delete command:
virsh snapshot-delete --domain admin_server --snapshotname "Snapshot 2"
10.5.2.3 Starting a Snapshot

To start a snapshot, use the snapshot-revert command:
virsh snapshot-revert --domain admin_server --snapshotname "Snapshot 1"
To start the current snapshot (the one the VM Guest was started off), it is sufficient to
use --current rather than specifying the snapshot name:
virsh snapshot-revert --domain admin_server --current

By default, deleting a VM Guest using virsh removes only its XML configuration.
Since attached storage is not deleted by default, you can reuse it with another VM
Guest. With Virtual Machine Manager, you can also delete a guest's storage files as
well—this will completely erase the guest.
10.6.1 Deleting a VM Guest with Virtual Machine Manager

1. In the Virtual Machine Manager, right-click a VM Guest entry.
2. From the context menu, choose Delete.
3. A confirmation window opens. Clicking Delete will permanently erase the VM

Guest. The deletion is not recoverable.
You can also permanently delete the guest's virtual disk by activating Delete
Associated Storage Files. The deletion is not recoverable either.
10.6.2 Deleting a VM Guest with virsh

To delete a VM Guest, it needs to be shut down first. It is not possible to delete a run-
ning guest. For information on shutting down, see Section 10.3, “Changing a VM
Guest's State: Start, Stop, Pause”.
To delete a VM Guest with virsh , run virsh undefine VM_NAME .
63 z 366 2017-05-27 13:21

virsh undefine sles12
There is no option to automatically delete the attached storage files. If they are man-
aged by libvirt, delete them as described in Section 12.2.4, “Deleting Volumes from a
Storage Pool”.

One of the major advantages of virtualization is that VM Guests are portable. When
a VM Host Server needs to go down for maintenance, or when the host gets over-
loaded, the guests can easily be moved to another VM Host Server. KVM and Xen
even support “live” migrations during which the VM Guest is constantly available.
10.7.1 Migration Requirements

To successfully migrate a VM Guest to another VM Host Server, the following re-
quirements need to be met:
It is recommended that the source and destination systems have the same
architecture. However, it is possible to migrate between hosts with AMD*
and Intel* architectures.
Storage devices must be accessible from both machines (for example, via
NFS or iSCSI) and must be configured as a storage pool on both machines.
For more information, see Chapter 12, Managing Storage.
This is also true for CD-ROM or floppy images that are connected during the
move. However, you can disconnect them prior to the move as described in
Section 14.8, “Ejecting and Changing Floppy or CD/DVD-ROM Media with Vir-
tual Machine Manager”.
libvirtd needs to run on both VM Host Servers and you must be able to
open a remote libvirt connection between the target and the source host
(or vice versa). Refer to Section 11.3, “Configuring Remote Connections” for
details.
If a firewall is running on the target host, ports need to be opened to allow

the migration. If you do not specify a port during the migration process,
libvirt chooses one from the range 49152:49215. Make sure that either
this range (recommended) or a dedicated port of your choice is opened in
the firewall on the target host.
Host and target machine should be in the same subnet on the network, oth-
erwise networking will not work after the migration.
64 z 366 2017-05-27 13:21

No running or paused VM Guest with the same name must exist on the tar-
get host. If a shut-down machine with the same name exists, its configura-
tion will be overwritten.
All CPU models except host cpu model are supported when migrating VM
Guests.
SATA disk device type is not migratable.
File system pass-through feature is incompatible with migration.
The VM Host Server and VM Guest need to have proper timekeeping in-
stalled. See Chapter 16, VM Guest Clock Settings.
Section 29.3.1.2, “virtio-blk-data-plane” is not supported for migration.
No physical devices can be passed from host to guest. Live migration is cur-
rently not supported when using devices with PCI pass-through or SR-IOV. In
case live migration needs to be supported, you need to use software virtual-
ization (paravirtualization or full virtualization).
Cache mode setting is an important setting for migration. See: Section 15.5,
“Effect of Cache Modes on Live Migration”.
Live migration of VM Guests from a host running one operating system to a

host running a different operating system is fully supported for the following
scenarios:
SLES 12 SP1 to SLES 12 SP2;
SLES 12 SP2 to SLES 12 SP2
SLES 12 SP2 to SLES 12 SP3 (when released)
Backward migration (from SLES 12 SP1 or SP2 to 12 or from SLES SP2 to SP1)
is not supported.
Live migration between SLE 11 and SLE 12 is not supported because of a dif-
ferent tool stack.
The image directory should be located in the same path on both hosts.
10.7.2 Migrating with Virtual Machine Manager

When using the Virtual Machine Manager to migrate VM Guests, it does not matter
on which machine it is started. You can start Virtual Machine Manager on the source
or the target host or even on a third host. In the latter case you need to be able to
open remote connections to both the target and the source host.
65 z 366 2017-05-27 13:21

1. Start Virtual Machine Manager and establish a connection to the target or

the source host. If the Virtual Machine Manager was started neither on the
target nor the source host, connections to both hosts need to be opened.
2. Right-click the VM Guest that you want to migrate and choose Migrate. Make
sure the guest is running or paused—it is not possible to migrate guests that
are shut down.
Tip: Increasing the Speed of the Migration
To increase the speed of the migration somewhat, pause the VM Guest.

This is the equivalent of the former so-called “offline migration” option of
Virtual Machine Manager.
3. Choose a New Host for the VM Guest. If the desired target host does not
show up, make sure that you are connected to the host.
To change the default options for connecting to the remote host, under Con-
nection, set the Mode, and the target host's Address (IP address or host
name) and Port. If you specify a Port, you must also specify an Address.
Under Advanced options, choose whether the move should be permanent

(default) or temporary, using Temporary move.
Additionally, there is the option Allow unsafe, which allows migrating without
disabling the cache of the VM Host Server. This can speed up the migration
but only works when the current configuration allows for a consistent view
of the VM Guest storage without using cache="none" / 0_DIRECT .
Note: Bandwidth Option
In recent versions of Virtual Machine Manager, the option of setting a

bandwidth for the migration has been removed. To set a specific band-
width, use virsh instead.
4. To perform the migration, click Migrate.
When the migration is complete, the Migrate window closes and the VM
Guest is now listed on the new host in the Virtual Machine Manager window.
The original VM Guest will still be available on the target host (in shut down
state).
66 z 366 2017-05-27 13:21

10.7.3 Migrating with virsh

To migrate a VM Guest with virsh migrate , you need to have direct or remote
shell access to the VM Host Server, because the command needs to be run on the
host. The migration command looks like this:
virsh migrate [OPTIONS] VM_ID_or_NAME CONNECTION_URI [--migrateuri tcp://REMOTE_HOST:POR
The most important options are listed below. See virsh help migrate for a full
list.
--live
Does a live migration. If not specified, the guest will be paused during the migra-
tion (“offline migration”).
--suspend
Does an offline migration and does not restart the VM Guest on the target host.
--persistent
By default a migrated VM Guest will be migrated temporarily, so its configura-
tion is automatically deleted on the target host if it is shut down. Use this switch
to make the migration persistent.
--undefinesource
When specified, the VM Guest definition on the source host will be deleted after
a successful migration (however, virtual disks attached to this guest will not be
deleted).
The following examples use mercury.example.com as the source system and

jupiter.example.com as the target system; the VM Guest's name is opensuse131
with Id 37 .
Offline migration with default parameters
virsh migrate 37 qemu+ssh://tux@jupiter.example.com/system
Transient live migration with default parameters
virsh migrate --live opensuse131 qemu+ssh://tux@jupiter.example.com/system
Persistent live migration; delete VM definition on source
67 z 366 2017-05-27 13:21

virsh migrate --live --persistent --undefinesource 37 \

qemu+tls://tux@jupiter.example.com/system
Offline migration using port 49152
virsh migrate opensuse131 qemu+ssh://tux@jupiter.example.com/system \

--migrateuri tcp://@jupiter.example.com:49152
Note: Transient Compared to Persistent Migrations
By default virsh migrate creates a temporary (transient) copy of the VM Guest

on the target host. A shut down version of the original guest description remains
on the source host. A transient copy will be deleted from the server after it is shut
down.
To create a permanent copy of a guest on the target host, use the switch
--persistent . A shut down version of the original guest description remains
on the source host, too. Use the option --undefinesource together with
--persistent for a “real” move where a permanent copy is created on the tar-
get host and the version on the source host is deleted.
It is not recommended to use --undefinesource without the --persistent

option, since this will result in the loss of both VM Guest definitions when the
guest is shut down on the target host.
10.7.4 Step-by-Step Example
10.7.4.1 Exporting the Storage

First you need to export the storage, to share the Guest image between host. This
can be done by an NFS server. In the following example we want to share the
/volume1/VM directory for all machines that are on the network 10.0.1.0/24. We
will use a SUSE Linux Enterprise NFS server. As root user, edit the /etc/exports
file and add:
/volume1/VM 10.0.1.0/24 (rw,sync,no_root_squash)
You need to restart the NFS server:
root # systemctl restart nfsserver

root # exportfs
/volume1/VM 10.0.1.0/24
68 z 366 2017-05-27 13:21

10.7.4.2 Defining the Pool on the Target Hosts

On each host where you want to migrate the VM Guest, the pool must be defined to
be able to access the volume (that contains the Guest image). Our NFS server IP ad-
dress is 10.0.1.99, its share is the /volume1/VM directory, and we want to get it
mounted in the /var/lib/libvirt/images/VM directory. The pool name will be
VM. To define this pool, create a VM.xml file with the following content:
<pool type='netfs'>
<name>VM</name>
<source>
<host name='10.0.1.99'/>
<dir path='/volume1/VM'/>
<format type='auto'/>
</source>
<target>
<path>/var/lib/libvirt/images/VM</path>
<permissions>
<mode>0755</mode>
<owner>-1</owner>
<group>-1</group>
</permissions>
</target>
</pool>
Then load it into libvirt using the pool-define command:
root # virsh pool-define VM.xml
An alternative way to define this pool is to use the virsh command:
root # virsh pool-define-as VM --type netfs --source-host 10.0.1.99 \

--source-path /volume1/VM --target /var/lib/libvirt/images/VM
Pool VM created
The following commands assume that you are in the interactive shell of virsh
which can also be reached by using the command virsh without any arguments.
Then the pool can be set to start automatically at host boot (autostart option):
virsh # pool-autostart VM
Pool VM marked as autostarted
If you want to disable the autostart:
virsh # pool-autostart VM --disable

Pool VM unmarked as autostarted
Check if the pool is present:
virsh # pool-list --all

Name State Autostart
69 z 366 2017-05-27 13:21

-------------------------------------------
default active yes
VM active yes
virsh # pool-info VM
Name: VM
UUID: 42efe1b3-7eaa-4e24-a06a-ba7c9ee29741
State: running
Persistent: yes
Autostart: yes
Capacity: 2,68 TiB
Allocation: 2,38 TiB
Available: 306,05 GiB
Warning: Pool Needs to Exist on All Target Hosts
Remember: this pool must be defined on each host where you want to be able to
migrate your VM Guest.
10.7.4.3 Creating the Volume

The pool has been defined—now we need a volume which will contain the disk im-
age:
virsh # vol-create-as VM sled12.qcow12 8G --format qcow2

Vol sled12.qcow12 created
The volume names shown will be used later to install the guest with virt-install.
10.7.4.4 Creating the VM Guest

Let's create a SUSE Linux Enterprise Server VM Guest with the virt-install com-
mand. The VM pool will be specified with the --disk option, cache=none is recom-
mended if you do not want to use the --unsafe option while doing the migration.
root # virt-install --connect qemu:///system --virt-type kvm --name \

sled12 --memory 1024 --disk vol=VM/sled12.qcow2,cache=none --cdrom \
/mnt/install/ISO/SLE-12-Desktop-DVD-x86_64-Build0327-Media1.iso --graphics \
vnc --os-variant sled12
Starting install...
Creating domain...
10.7.4.5 Migrate the VM Guest

Everything is ready to do the migration now. Run the migrate command on the
VM Host Server that is currently hosting the VM Guest, and choose the destination.
70 z 366 2017-05-27 13:21

virsh # migrate --live sled12 --verbose qemu+ssh://IP/Hostname/system

Password:
Migration: [ 12 %]
10.8 Monitoring
10.8.1 Monitoring with Virtual Machine Manager

After starting Virtual Machine Manager and connecting to the VM Host Server, a CPU
usage graph of all the running guests is displayed.
It is also possible to get information about disk and network usage with this tool,
however, you must first activate this in Preferences:
1. Run virt-manager .
2. Select Edit › Preferences.
3. Change the tab from General to Polling.
4. Activate the check boxes for the kind of activity you want to see: Poll Disk I/O,
Poll Network I/O, and Poll Memory stats.
5. If desired, also change the update interval using Update status every n sec-
onds.
6. Close the Preferences dialog.
7. Activate the graphs that should be displayed under View › Graph.
Afterward, the disk and network statistics are also displayed in the main window of
the Virtual Machine Manager.
More precise data is available from the VNC window. Open a VNC window as de-
scribed in Section 10.2.1, “Opening a Graphical Console”. Choose Details from the
toolbar or the View menu. The statistics are displayed from the Performance entry of
the left-hand tree menu.
10.8.2 Monitoring with virt-top

virt-top is a command line tool similar to the well-known process monitoring
tool top . virt-top uses libvirt and therefore is capable of showing statistics for
VM Guests running on different hypervisors. It is recommended to use virt-top
instead of hypervisor-specific tools like xentop .
71 z 366 2017-05-27 13:21

By default virt-top shows statistics for all running VM Guests. Among the data
that is displayed is the percentage of memory used ( %MEM ) and CPU ( %CPU ) and the
uptime of the guest ( TIME ). The data is updated regularly (every three seconds by
default). The following shows the output on a VM Host Server with seven VM Guests,
four of them inactive:
virt-top 13:40:19 - x86_64 8/8CPU 1283MHz 16067MB 7.6% 0.5%

7 domains, 3 active, 3 running, 0 sleeping, 0 paused, 4 inactive D:0 O:0 X:0
CPU: 6.1% Mem: 3072 MB (3072 MB by guests)
ID S RDRQ WRRQ RXBY TXBY %CPU %MEM TIME NAME

7 R 123 1 18K 196 5.8 6.0 0:24.35 sled12_sp1
6 R 1 0 18K 0 0.2 6.0 0:42.51 sles12_sp1
5 R 0 0 18K 0 0.1 6.0 85:45.67 opensuse_leap
- (Ubuntu_1410)
- (debian_780)
- (fedora_21)
- (sles11sp3)
By default the output is sorted by ID. Use the following key combinations to change
the sort field:
Shift – P : CPU usage
Shift – M : Total memory allocated by the guest
Shift – T : Time
Shift – I : ID
To use any other field for sorting, press Shift – F and select a field from the list. To
toggle the sort order, use Shift – R .
virt-top also supports different views on the VM Guests data, which can be
changed on-the-fly by pressing the following keys:
0 : default view
1 : show physical CPUs
2 : show network interfaces
3 : show virtual disks
virt-top supports more hot keys to change the view of the data and many com-
mand line switches that affect the behavior of the program. For more information,
see man 1 virt-top .
10.8.3 Monitoring with kvm_stat

kvm_stat can be used to trace KVM performance events. It monitors
/sys/kernel/debug/kvm , so it needs the debugfs to be mounted. On SUSE Linux
Enterprise Server it should be mounted by default. In case it is not mounted, use the
following command:
72 z 366 2017-05-27 13:21

mount -t debugfs none /sys/kernel/debug
kvm_stat can be used in three different modes:
kvm_stat # update in 1 second intervals

kvm_stat -1 # 1 second snapshot
kvm_stat -l > kvmstats.log # update in 1 second intervals in log format
# can be imported to a spreadsheet
EXAMPLE 10.1: TYPICAL OUTPUT OF kvm_stat
kvm statistics
efer_reload 0 0
exits 11378946 218130
fpu_reload 62144 152
halt_exits 414866 100
halt_wakeup 260358 50
host_state_reload 539650 249
hypercalls 0 0
insn_emulation 6227331 173067
insn_emulation_fail 0 0
invlpg 227281 47
io_exits 113148 18
irq_exits 168474 127
irq_injections 482804 123
irq_window 51270 18
largepages 0 0
mmio_exits 6925 0
mmu_cache_miss 71820 19
mmu_flooded 35420 9
mmu_pde_zapped 64763 20
mmu_pte_updated 0 0
mmu_pte_write 213782 29
mmu_recycled 0 0
mmu_shadow_zapped 128690 17
mmu_unsync 46 -1
nmi_injections 0 0
nmi_window 0 0
pf_fixed 1553821 857
pf_guest 1018832 562
remote_tlb_flush 174007 37
request_irq 0 0
signal_exits 0 0
tlb_flush 394182 148
See http://clalance.blogspot.com/2009/01/kvm-performance-tools.html↗ for fur-

ther information on how to interpret these values.
73 z 366 2017-05-27 13:21


11.1 Authentication
Managing several VM Host Servers, each hosting multiple VM Guests, quickly be-
comes difficult. One benefit of libvirt is the ability to connect to several VM Host
Servers at once, providing a single interface to manage all VM Guests and to connect
to their graphical console.
To ensure only authorized users can connect, libvirt offers several connection
types (via TLS, SSH, Unix sockets, and TCP) that can be combined with different au-
thorization mechanisms (socket, PolKit, SASL and Kerberos).
11.1 Authentication
The power to manage VM Guests and to access their graphical console is something
that should be restricted to a well defined circle of persons. To achieve this goal, you
can use the following authentication techniques on the VM Host Server:
Access control for Unix sockets with permissions and group ownership. This
method is available for libvirtd connections only.
Access control for Unix sockets with PolKit. This method is available for local
libvirtd connections only.
User name and password authentication with SASL (Simple Authentication

and Security Layer). This method is available for both, libvirtd and VNC
connections. Using SASL does not require real user accounts on the server,
since it uses its own database to store user names and passwords. Connec-
tions authenticated with SASL are encrypted.
74 z 366 2017-05-27 13:21

Kerberos authentication. This method, available for libvirtd connections

only, is not covered in this manual. Refer to http://libvirt.org
/auth.html#ACL_server_kerberos↗ for details.
Single password authentication. This method is available for VNC connec-

tions only.
Important: Authentication for libvirtd and VNC need to be

configured separately
Access to the VM Guest's management functions (via libvirtd ) on the one

hand, and to its graphical console on the other hand, always needs to be config-
ured separately. When restricting access to the management tools, these restric-
tions do not automatically apply to VNC connections!
When accessing VM Guests from remote via TLS/SSL connections, access can be indi-
rectly controlled on each client by restricting read permissions to the certificate's key
file to a certain group. See Section 11.3.2.5, “Restricting Access (Security Considera-
tions)” for details.
11.1.1 libvirtd Authentication

libvirtd authentication is configured in /etc/libvirt/libvirtd.conf . The
configuration made here applies to all libvirt tools such as the Virtual Machine
Manager or virsh .
libvirt offers two sockets: a read-only socket for monitoring purposes and a
read-write socket to be used for management operations. Access to both sockets
can be configured independently. By default, both sockets are owned by
root.root . Default access permissions on the read-write socket are restricted to
the user root ( 0700 ) and fully open on the read-only socket ( 0777 ).
In the following instructions, you will learn how to configure access permissions for
the read-write socket. The same instructions also apply to the read-only socket. All
configuration steps need to be carried out on the VM Host Server.
75 z 366 2017-05-27 13:21

Note: Default Authentication Settings on SUSE Linux Enterprise Server
The default authentication method on SUSE Linux Enterprise Server is access con-
trol for Unix sockets. Only the user root may authenticate. When accessing the
libvirt tools as a non-root user directly on the VM Host Server, you need to
provide the root password through PolKit once. You are then granted access for
the current and for future sessions.
Alternatively, you can configure libvirt to allow “system” access to non-privi-

leged users. See Section 11.2.1, ““system” Access for Non-Privileged Users” for de-
tails.
RECOMMENDED AUTHORIZATION METHODS

Local Connections
Section 11.1.1.2, “Local Access Control for Unix Sockets with PolKit”
Section 11.1.1.1, “Access Control for Unix Sockets with Permissions and Group
Ownership”
Remote Tunnel over SSH

Section 11.1.1.1, “Access Control for Unix Sockets with Permissions and Group
Ownership”
Remote TLS/SSL Connection

Section 11.1.1.3, “User name and Password Authentication with SASL”
none (access controlled on the client side by restricting access to the certificates)
11.1.1.1 Access Control for Unix Sockets with Permissions and

Group Ownership
To grant access for non- root accounts, configure the sockets to be owned and ac-
cessible by a certain group ( libvirt in the following example). This authentication
method can be used for local and remote SSH connections.
1. In case it does not exist, create the group that should own the socket:
groupadd libvirt
Important: Group Needs to Exist
The group must exist prior to restarting libvirtd . If not, the restart will
fail.
76 z 366 2017-05-27 13:21

2. Add the desired users to the group:
usermod --append --groups libvirt tux
3. Change the configuration in /etc/libvirt/libvirtd.conf as follows:
unix_sock_group = "libvirt" 1
unix_sock_rw_perms = "0770" 2
auth_unix_rw = "none" 3
1 Group ownership will be set to group libvirt .

2 Sets the access permissions for the socket ( srwxrwx--- ).
3 Disables other authentication methods (PolKit or SASL). Access is
solely controlled by the socket permissions.
4. Restart libvirtd :
systemctl start libvirtd
11.1.1.2 Local Access Control for Unix Sockets with PolKit

Access control for Unix sockets with PolKit is the default authentication method on
SUSE Linux Enterprise Server for non-remote connections. Therefore, no libvirt
configuration changes are needed. With PolKit authorization enabled, permissions
on both sockets default to 0777 and each application trying to access a socket
needs to authenticate via PolKit.
Important: PolKit Authentication for Local Connections Only
Authentication with PolKit can only be used for local connections on the VM Host
Server itself, since PolKit does not handle remote authentication.
Two policies for accessing libvirt 's sockets exist:
org.libvirt.unix.monitor: accessing the read-only socket
org.libvirt.unix.manage: accessing the read-write socket
By default, the policy for accessing the read-write socket is to authenticate with the
root password once and grant the privilege for the current and for future sessions.
77 z 366 2017-05-27 13:21

To grant users access to a socket without having to provide the root password, you
need to create a rule in /etc/polkit-1/rules.d . Create the file
/etc/polkit-1/rules.d/10-grant-libvirt with the following content to grant
access to the read-write socket to all members of the group libvirt :
polkit.addRule(function(action, subject) {
if (action.id == "org.libvirt.unix.manage" && subject.isInGroup("libvirt")) {
return polkit.Result.YES;
}
});
11.1.1.3 User name and Password Authentication with SASL

SASL provides user name and password authentication and data encryption (di-
gest-md5, by default). Since SASL maintains its own user database, the users do not
need to exist on the VM Host Server. SASL is required by TCP connections and on
top of TLS/SSL connections.
Important: Plain TCP and SASL with digest-md5 Encryption
Using digest-md5 encryption on an otherwise not encrypted TCP connection

does not provide enough security for production environments. It is recom-
mended to only use it in testing environments.
Tip: SASL Authentication on Top of TLS/SSL
Access from remote TLS/SSL connections can be indirectly controlled on the client
side by restricting access to the certificate's key file. However, this might prove er-
ror-prone when dealing with many clients. Using SASL with TLS adds security by
additionally controlling access on the server side.
To configure SASL authentication, proceed as follows:
1. Change the configuration in /etc/libvirt/libvirtd.conf as follows:
a. To enable SASL for TCP connections:
auth_tcp = "sasl"
b. To enable SASL for TLS/SSL connections:
auth_tls = "sasl"
78 z 366 2017-05-27 13:21

systemctl restart libvirtd
3. The libvirt SASL configuration file is located at

/etc/sasl2/libvirtd.conf . Normally, there is no need to change the
defaults. However, if using SASL on top of TLS, you may turn off session en-
cryption to avoid additional overhead (TLS connections are already en-
crypted) by commenting the line setting the mech_list parameter. Only
do this for TSL/SASL, for TCP connections this parameter must be set to di-
gest-md5.
#mech_list: digest-md5
4. By default, no SASL users are configured, so no logins are possible. Use the
following commands to manage users:
Add the user tux
saslpasswd2 -a libvirt tux
Delete the user tux
saslpasswd2 -a libvirt -d tux
List existing users
sasldblistusers2 -f /etc/libvirt/passwd.db
Tip: virsh and SASL Authentication
When using SASL authentication, you will be prompted for a user name and pass-
word every time you issue a virsh command. Avoid this by using virsh in
shell mode.
11.1.2 VNC Authentication
79 z 366 2017-05-27 13:21

Since access to the graphical console of a VM Guest is not controlled by libvirt ,

but rather by the specific hypervisor, it is always necessary to additionally configure
VNC authentication. The main configuration file is
/etc/libvirt/<hypervisor>.conf . This section describes the QEMU/KVM hy-
pervisor, so the target configuration file is /etc/libvirt/qemu.conf .
Note: VNC Authentication for Xen
In contrast to KVM and LXC, Xen does not yet offer more sophisticated VNC au-
thentication than setting a password on a per VM basis. See the
<graphics type='vnc'... libvirt configuration option below.
Two authentication types are available: SASL and single password authentication. If
you are using SASL for libvirt authentication, it is strongly recommended to use
it for VNC authentication as well—it is possible to share the same database.
A third method to restrict access to the VM Guest is to enable the use of TLS encryp-
tion on the VNC server. This requires the VNC clients to have access to x509 client
certificates. By restricting access to these certificates, access can indirectly be con-
trolled on the client side. Refer to Section 11.3.2.4.2, “VNC over TLS/SSL: Client Con-
figuration” for details.
11.1.2.1 User name and Password Authentication with SASL

SASL provides user name and password authentication and data encryption. Since
SASL maintains its own user database, the users do not need to exist on the VM
Host Server. As with SASL authentication for libvirt , you may use SASL on top of
TLS/SSL connections. Refer to Section 11.3.2.4.2, “VNC over TLS/SSL: Client Configu-
ration” for details on configuring these connections.
To configure SASL authentication for VNC, proceed as follows:
1. Create a SASL configuration file. It is recommended to use the existing

libvirt file. If you have already configured SASL for libvirt and are
planning to use the same settings including the same user name and pass-
word database, a simple link is suitable:
ln -s /etc/sasl2/libvirt.conf /etc/sasl2/qemu.conf
If are setting up SASL for VNC only or you are planning to use a different
configuration than for libvirt , copy the existing file to use as a template:
cp /etc/sasl2/libvirt.conf /etc/sasl2/qemu.conf
80 z 366 2017-05-27 13:21

Then edit it according to your needs.
2. Change the configuration in /etc/libvirt/qemu.conf as follows:
vnc_listen = "0.0.0.0"
vnc_sasl = 1
sasldb_path: /etc/libvirt/qemu_passwd.db
The first parameter enables VNC to listen on all public interfaces (rather than
to the local host only), and the second parameter enables SASL authentica-
tion.
3. By default, no SASL users are configured, so no logins are possible. Use the
following commands to manage users:
Add the user tux
saslpasswd2 -f /etc/libvirt/qemu_passwd.db -a qemu tux
Delete the user tux
saslpasswd2 -f /etc/libvirt/qemu_passwd.db -a qemu -d tux
List existing users
sasldblistusers2 -f /etc/libvirt/qemu_passwd.db
5. Restart all VM Guests that have been running prior to changing the configu-
ration. VM Guests that have not been restarted will not use SASL authentica-
tion for VNC connects.
Note: Supported VNC Viewers
SASL authentication is currently supported by Virtual Machine Manager and

virt-viewer . Both of these viewers also support TLS/SSL connections.
11.1.2.2 Single Password Authentication
81 z 366 2017-05-27 13:21

Access to the VNC server may also be controlled by setting a VNC password. You can
either set a global password for all VM Guests or set individual passwords for each
guest. The latter requires to edit the VM Guest's configuration files.
Note: Always Set a Global Password
If you are using single password authentication, it is good practice to set a global
password even if setting passwords for each VM Guest. This will always leave your
virtual machines protected with a “fallback” password if you forget to set a
per-machine password. The global password will only be used if no other pass-
word is set for the machine.
PROCEDURE 11.1: SETTING A GLOBAL VNC PASSWORD
1. Change the configuration in /etc/libvirt/qemu.conf as follows:
vnc_password = "PASSWORD"
The first parameter enables VNC to listen on all public interfaces (rather than
to the local host only), and the second parameter sets the password. The
maximum length of the password is eight characters.
root # systemctl restart libvirtd
ration. VM Guests that have not been restarted will not use password au-
thentication for VNC connects.
PROCEDURE 11.2: SETTING A VM GUEST SPECIFIC VNC PASSWORD
1. Change the configuration in /etc/libvirt/qemu.conf as follows to en-

able VNC to listen on all public interfaces (rather than to the local host only).
2. Open the VM Guest's XML configuration file in an editor. Replace VM NAME

in the following example with the name of the VM Guest. The editor that is
used defaults to $EDITOR . If that variable is not set, vi is used.
virsh edit VM NAME
3. Search for the element <graphics> with the attribute type='vnc' , for
example:
82 z 366 2017-05-27 13:21

<graphics type='vnc' port='-1' autoport='yes'/>
4. Add the passwd=PASSWORD attribute, save the file and exit the editor. The
maximum length of the password is eight characters.
<graphics type='vnc' port='-1' autoport='yes' passwd='PASSWORD'/>
ration. VM Guests that have not been restarted will not use password au-
thentication for VNC connects.
Warning: Security of the VNC Protocol
The VNC protocol is not considered to be safe. Although the password is sent en-
crypted, it might be vulnerable when an attacker can sniff both the encrypted
password and the encryption key. Therefore, it is recommended to use VNC with
TLS/SSL or tunneled over SSH. virt-viewer , Virtual Machine Manager and
vinagre from version 2.30 onward support both methods.

To connect to a hypervisor with libvirt , you need to specify a uniform resource
identifier (URI). This URI is needed with virsh and virt-viewer (except when
working as root on the VM Host Server) and is optional for the Virtual Machine
Manager. Although the latter can be called with a connection parameter (for exam-
ple, virt-manager -c qemu:///system ), it also offers a graphical interface to
create connection URIs. See Section 11.2.2, “Managing Connections with Virtual Ma-
chine Manager” for details.
HYPERVISOR 1 +PROTOCOL 2 ://USER@REMOTE 3 /CONNECTION_TYPE 4
83 z 366 2017-05-27 13:21

1 Specify the hypervisor. SUSE Linux Enterprise Server currently supports the
following hypervisors: test (dummy for testing), qemu (KVM), and xen
(Xen). This parameter is mandatory.
2 When connecting to a remote host, specify the protocol here. It can be one of:
ssh (connection via SSH tunnel), tcp (TCP connection with SASL/Kerberos
authentication), tls (TLS/SSL encrypted connection with authentication via
x509 certificates).
3 When connecting to a remote host, specify the user name and the remote
host name. If no user name is specified, the user name that has called the
command ( $USER ) is used. See below for more information. For TLS connec-
tions, the host name needs to be specified exactly as in the x509 certificate.
4 When connecting to the QEMU/KVM hypervisor, two connection types are ac-
cepted: system for full access rights, or session for restricted access. Since
session access is not supported on SUSE Linux Enterprise Server, this docu-
mentation focuses on system access.
EXAMPLE HYPERVISOR CONNECTION URIS

test:///default
Connect to the local dummy hypervisor. Useful for testing.
qemu:///system or xen:///system
Connect to the QEMU/Xen hypervisor on the local host having full access (type
system).
qemu+ssh://tux@mercury.example.com/system or
xen+ssh://tux@mercury.example.com/system
Connect to the QEMU/Xen hypervisor on the remote host mercury.exam-
ple.com. The connection is established via an SSH tunnel.
qemu+tls://saturn.example.com/system or
xen+tls://saturn.example.com/system
Connect to the QEMU/Xen hypervisor on the remote host mercury.exam-
ple.com. The connection is established using TLS/SSL.
For more details and examples, refer to the libvirt documentation at

http://libvirt.org/uri.html↗.
84 z 366 2017-05-27 13:21

Note: User Names in URIs
A user name needs to be specified when using Unix socket authentication (re-
gardless of whether using the user/password authentication scheme or PolKit).
This applies to all SSH and local connections.
There is no need to specify a user name when using SASL authentication (for TCP
or TLS connections) or when doing no additional server-side authentication for
TLS connections. With SASL the user name will not be evaluated—you will be
prompted for an SASL user/password combination in any case.
11.2.1 “system” Access for Non-Privileged Users

As mentioned above, a connection to the QEMU hypervisor can be established using
two different protocols: session and system . A “session” connection is spawned
with the same privileges as the client program. Such a connection is intended for
desktop virtualization, since it is restricted (for example no USB/PCI device assign-
ments, no virtual network setup, limited remote access to libvirtd ).
The “system” connection intended for server virtualization has no functional restric-
tions but is, by default, only accessible by root . However, with the addition of the
DAC (Discretionary Access Control) driver to libvirt it is now possible to grant
non-privileged users “system” access. To grant “system” access to the user tux , pro-
ceed as follows:
PROCEDURE 11.3: GRANTING “SYSTEM” ACCESS TO A REGULAR USER
1. Enable access via Unix sockets as described in Section 11.1.1.1, “Access Con-
trol for Unix Sockets with Permissions and Group Ownership”. In that exam-
ple access to libvirt is granted to all members of the group libvirt and
tux made a member of this group. This ensures that tux can connect us-
ing virsh or Virtual Machine Manager.
2. Edit /etc/libvirt/qemu.conf and change the configuration as follows:
user = "tux"
group = "libvirt"
dynamic_ownership = 1
This ensures that the VM Guests are started by tux and that resources
bound to the guest (for example virtual disks) can be accessed and modified
by tux .
3. Make tux a member of the group kvm :
85 z 366 2017-05-27 13:21

usermod --append --groups kvm tux
This step is needed to grant access to /dev/kvm , which is required to start

VM Guests.
11.2.2 Managing Connections with Virtual Machine Manager

The Virtual Machine Manager uses a Connection for every VM Host Server it man-
ages. Each connection contains all VM Guests on the respective host. By default, a
connection to the local host is already configured and connected.
All configured connections are displayed in the Virtual Machine Manager main win-
dow. Active connections are marked with a small triangle, which you can click to fold
or unfold the list of VM Guests for this connection.
Inactive connections are listed gray and are marked with Not Connected . Either
double-click or right-click it and choose Connect from the context menu. You can
also Delete an existing connection from this menu.
Note: Editing Existing Connections
It is not possible to edit an existing connection. To change a connection, create a

new one with the desired parameters and delete the “old” one.
To add a new connection in the Virtual Machine Manager, proceed as follows:
1. Choose File › Add Connection
2. Choose the host's Hypervisor (Xen or QEMU/KVM)
3. (Optional) To set up a remote connection, choose Connect to remote host. For

more information, see Section 11.3, “Configuring Remote Connections”.
In case of a remote connection, specify the Hostname of the remote machine

in the format USERNAME@REMOTE _HOST .
86 z 366 2017-05-27 13:21

Important: Specifying a User Name
There is no need to specify a user name for TCP and TLS connections: In
these cases, it will not be evaluated. However, in the case of SSH connec-
tions, specifying a user name is necessary when you want to connect as a
user other than root .
4. If you do not want the connection to be automatically started when starting

the Virtual Machine Manager, deactivate Autoconnect.
5. Finish the configuration by clicking Connect.

A major benefit of libvirt is the ability to manage VM Guests on different remote
hosts from a central location. This section gives detailed instructions on how to con-
figure server and client to allow remote connections.
11.3.1 Remote Tunnel over SSH ( qemu+ssh or xen+ssh )

Enabling a remote connection that is tunneled over SSH on the VM Host Server only
requires the ability to accept SSH connections. Make sure the SSH daemon is started
( systemctl status sshd ) and that the ports for service SSH are opened in the
firewall.
User authentication for SSH connections can be done using traditional file
user/group ownership and permissions as described in Section 11.1.1.1, “Access
Control for Unix Sockets with Permissions and Group Ownership”. Connecting as
user tux ( qemu+ssh://tuxsIVname;/system or
xen+ssh://tuxsIVname;/system ) works out of the box and does not require ad-
ditional configuration on the libvirt side.
When connecting via SSH qemu+ssh://USER@SYSTEM or

xen+ssh://USER@SYSTEM you need to provide the password for USER . This can
be avoided by copying your public key to ~ USER /.ssh/authorized_keys on the
VM Host Server as explained in Book “Security Guide”, Chapter 14 “SSH: Secure Net-
work Operations”, Section 14.5.2 “Copying an SSH Key”. Using an ssh-agent on the ma-
chine from which you are connecting adds even more convenience. For more infor-
mation, see Book “Security Guide”, Chapter 14 “SSH: Secure Network Operations”, Sec-
tion 14.5.3 “Using the ssh-agent ”.
87 z 366 2017-05-27 13:21

11.3.2 Remote TLS/SSL Connection with x509 Certificate (

qemu+tls or xen+tls )
Using TCP connections with TLS/SSL encryption and authentication via x509 certifi-
cates is much more complicated to set up than SSH, but it is a lot more scalable. Use
this method if you need to manage several VM Host Servers with a varying number
of administrators.
11.3.2.1 Basic Concept

TLS (Transport Layer Security) encrypts the communication between two computers
by using certificates. The computer starting the connection is always considered the
“client”, using a “client certificate”, while the receiving computer is always considered
the “server ”, using a “server certificate”. This scenario applies, for example, if you
manage your VM Host Servers from a central desktop.
If connections are initiated from both computers, each needs to have a client and a
server certificate. This is the case, for example, if you migrate a VM Guest from one
host to another.
Each x509 certificate has a matching private key file. Only the combination of certifi-
cate and private key file can identify itself correctly. To assure that a certificate was is-
sued by the assumed owner, it is signed and issued by a central certificate called cer-
tificate authority (CA). Both the client and the server certificates must be issued by
the same CA.
Important: User Authentication
Using a remote TLS/SSL connection only ensures that two computers are allowed
to communicate in a certain direction. Restricting access to certain users can indi-
rectly be achieved on the client side by restricting access to the certificates. For
more information, see Section 11.3.2.5, “Restricting Access (Security Considera-
tions)”.
libvirt also supports user authentication on the server with SASL. For more
information, see Section 11.3.2.6, “Central User Authentication with SASL for TLS
Sockets”.
11.3.2.2 Configuring the VM Host Server
88 z 366 2017-05-27 13:21

The VM Host Server is the machine receiving connections. Therefore, the server cer-
tificates need to be installed. The CA certificate needs to be installed, too. When the
certificates are in place, TLS support can be turned on for libvirt .
1. Create the server certificate and export it together with the CA certificate as
described in Section B.2, “Generating x509 Client/Server Certificates”.
2. Create the following directories on the VM Host Server:
mkdir -p /etc/pki/CA/ /etc/pki/libvirt/private/
Install the certificates as follows:
/etc/pki/CA/cacert.pem
/etc/pki/libvirt/servercert.pem
/etc/pki/libvirt/private/serverkey.pem
Important: Restrict Access to Certificates
Make sure to restrict access to certificates as explained in Sec-

tion 11.3.2.5, “Restricting Access (Security Considerations)”.
3. Enable TLS support by editing /etc/libvirt/libvirtd.conf and setting

listen_tls = 1 . Restart libvirtd :
4. By default, libvirt uses the TCP port 16514 for accepting secure TLS con-
nections. Open this port in the firewall.
Important: Restarting libvirtd with TLS enabled
If you enable TLS for libvirt , the server certificates need to be in place, other-
wise restarting libvirtd will fail. You also need to restart libvirtd in case
you change the certificates.
11.3.2.3 Configuring the Client and Testing the Setup

The client is the machine initiating connections. Therefore the client certificates need
to be installed. The CA certificate needs to be installed, too.
1.
89 z 366 2017-05-27 13:21

Create the client certificate and export it together with the CA certificate as
described in Section B.2, “Generating x509 Client/Server Certificates”.
2. Create the following directories on the client:
mkdir -p /etc/pki/CA/ /etc/pki/libvirt/private/
Install the certificates as follows:
/etc/pki/libvirt/clientcert.pem
/etc/pki/libvirt/private/clientkey.pem
Make sure to restrict access to certificates as explained in Sec-

tion 11.3.2.5, “Restricting Access (Security Considerations)”.
3. Test the client/server setup by issuing the following command. Replace

mercury.example.com with the name of your VM Host Server. Specify the
same fully qualified host name as used when creating the server certificate.
#QEMU/KVM
virsh -c qemu+tls://mercury.example.com/system list --all
#Xen
virsh -c xen+tls://mercury.example.com/system list --all
If your setup is correct, you will see a list of all VM Guests registered with
libvirt on the VM Host Server.
11.3.2.4 Enabling VNC for TLS/SSL connections

Currently, VNC communication over TLS is only supported by a few tools. Common
VNC viewers such as tightvnc or tigervnc do not support TLS/SSL. The only
supported alternative to Virtual Machine Manager and virt-viewer is vinagre .
11.3.2.4.1 VNC over TLS/SSL: VM Host Server Configuration

To access the graphical console via VNC over TLS/SSL, you need to configure the VM
Host Server as follows:
1. Open ports for the service VNC in your firewall.
2. Create a directory /etc/pki/libvirt-vnc and link the certificates into

this directory as follows:
90 z 366 2017-05-27 13:21

mkdir -p /etc/pki/libvirt-vnc && cd /etc/pki/libvirt-vnc

ln -s /etc/pki/CA/cacert.pem ca-cert.pem
ln -s /etc/pki/libvirt/servercert.pem server-cert.pem
ln -s /etc/pki/libvirt/private/serverkey.pem server-key.pem
3. Edit /etc/libvirt/qemu.conf and set the following parameters:
vnc_tls = 1
vnc_tls_x509_verify = 1
4. Restart the libvirtd :
Important: VM Guests Need to be Restarted
The VNC TLS setting is only set when starting a VM Guest. Therefore, you
need to restart all machines that have been running prior to making the
configuration change.
11.3.2.4.2 VNC over TLS/SSL: Client Configuration

The only action needed on the client side is to place the x509 client certificates in a
location recognized by the client of choice. Unfortunately, each supported client
—Virtual Machine Manager, virt-viewer , and vinagre —expects the certificates
in a different location. However, Virtual Machine Manager and vinagre can either
read from a system-wide location applying to all users, or from a per-user location.
Virtual Machine Manager ( virt-manager )

To connect to the remote host, Virtual Machine Manager requires the setup ex-
plained in Section 11.3.2.3, “Configuring the Client and Testing the Setup”. To be
able to connect via VNC, the client certificates also need to be placed in the fol-
lowing locations:
System-wide location
/etc/pki/libvirt-vnc/clientcert.pem
/etc/pki/libvirt-vnc/private/clientkey.pem
Per-user location
91 z 366 2017-05-27 13:21

~/.pki/libvirt-vnc/clientcert.pem
~/.pki/libvirt-vnc/private/clientkey.pem
virt-viewer
virt-viewer only accepts certificates from a system-wide location:
Make sure to restrict access to certificates as explained in Section 11.3.2.5, “Re-

stricting Access (Security Considerations)”.
11.3.2.5 Restricting Access (Security Considerations)

Each x509 certificate consists of two pieces: the public certificate and a private key. A
client can only authenticate using both pieces. Therefore, any user that has read ac-
cess to the client certificate and its private key can access your VM Host Server. On
the other hand, an arbitrary machine equipped with the full server certificate can
pretend to be the VM Host Server. Since this is probably not desirable, access to at
least the private key files needs to be restricted as much as possible. The easiest way
to control access to a key file is to use access permissions.
Server Certificates
Server certificates need to be readable for QEMU processes. On SUSE Linux
Enterprise Server QEMU, processes started from libvirt tools are owned by
root , so it is sufficient if the root can read the certificates:
chmod 700 /etc/pki/libvirt/private/

chmod 600 /etc/pki/libvirt/private/serverkey.pem
If you change the ownership for QEMU processes in

/etc/libvirt/qemu.conf , you also need to adjust the ownership of the key
file.
System Wide Client Certificates
92 z 366 2017-05-27 13:21

To control access to a key file that is available system-wide, restrict read access to
a certain group, so that only members of that group can read the key file. In the
following example, a group libvirt is created, and group ownership of the
clientkey.pem file and its parent directory is set to libvirt . Afterward, the
access permissions are restricted to owner and group. Finally the user tux is
added to the group libvirt , and thus can access the key file.
CERTPATH="/etc/pki/libvirt/"
# create group libvirt
groupadd libvirt
# change ownership to user root and group libvirt
chown root.libvirt $CERTPATH/private $CERTPATH/clientkey.pem
# restrict permissions
chmod 750 $CERTPATH/private
chmod 640 $CERTPATH/private/clientkey.pem
# add user tux to group libvirt
usermod --append --groups libvirt tux
Per-User Certificates
User-specific client certificates for accessing the graphical console of a VM Guest
via VNC need to be placed in the user's home directory in ~/.pki . Contrary to
SSH, for example, the VNC viewer using these certificates do not check the ac-
cess permissions of the private key file. Therefore, it is solely the user's responsi-
bility to make sure the key file is not readable by others.
11.3.2.5.1 Restricting Access from the Server Side

By default, every client that is equipped with appropriate client certificates may con-
nect to a VM Host Server accepting TLS connections. Therefore, it is possible to use
additional server-side authentication with SASL as described in Section 11.1.1.3,
“User name and Password Authentication with SASL”.
It is also possible to restrict access with a whitelist of DNs (distinguished names), so

only clients with a certificate matching a DN from the list can connect.
Add a list of allowed DNs to tls_allowed_dn_list in

/etc/libvirt/libvirtd.conf . This list may contain wild cards. Do not specify an
empty list, since that would result in refusing all connections.
tls_allowed_dn_list = [
"C=US,L=Provo,O=SUSE Linux Products GmbH,OU=*,CN=venus.example.com,EMAIL=*",
"C=DE,L=Nuremberg,O=SUSE Linux Products GmbH,OU=Documentation,CN=*"]
Get the distinguished name of a certificate with the following command:
certtool -i --infile /etc/pki/libvirt/clientcert.pem | grep "Subject:"
Restart libvirtd after having changed the configuration:
93 z 366 2017-05-27 13:21

11.3.2.6 Central User Authentication with SASL for TLS Sockets

A direct user authentication via TLS is not possible—this is handled indirectly on
each client via the read permissions for the certificates as explained in Sec-
tion 11.3.2.5, “Restricting Access (Security Considerations)”. However, if a central,
server-based user authentication is needed, libvirt also allows to use SASL (Sim-
ple Authentication and Security Layer) on top of TLS for direct user authentication.
See Section 11.1.1.3, “User name and Password Authentication with SASL” for config-
uration details.
11.3.2.7 Troubleshooting
11.3.2.7.1 Virtual Machine Manager/ virsh Cannot Connect to Server
Check the following in the given order:

Is it a firewall issue (TCP port 16514 needs to be open on the server)?
Is the client certificate (certificate and key) readable by the user that has started Vir-
tual Machine Manager/ virsh ?
Has the same full qualified host name as in the server certificate been specified with
the connection?
Is TLS enabled on the server ( listen_tls = 1 )?
Has libvirtd been restarted on the server?
11.3.2.7.2 VNC Connection fails

Ensure that you can connect to the remote server using Virtual Machine Manager. If
so, check whether the virtual machine on the server has been started with TLS sup-
port. The virtual machine's name in the following example is “sles”.
ps ax | grep qemu | grep "\-name sles" | awk -F" -vnc " '{ print FS $2 }'
If the output does not begin with a string similar to the following, the machine has
not been started with TLS support and must be restarted.
-vnc 0.0.0.0:0,tls,x509verify=/etc/pki/libvirt
94 z 366 2017-05-27 13:21

12 Managing Storage
12.1 Managing Storage with Virtual Machine Manager
12.3 Locking Disk Files and Block Devices with virtlockd
12.5 Sharing Directories between Host and Guests (File System
Pass-Through)
When managing a VM Guest on the VM Host Server itself, you can access the com-
plete file system of the VM Host Server to attach or create virtual hard disks or to at-
tach existing images to the VM Guest. However, this is not possible when managing
VM Guests from a remote host. For this reason, libvirt supports so called “Stor-
age Pools”, which can be accessed from remote machines.
Tip: CD/DVD ISO images
To be able to access CD/DVD ISO images on the VM Host Server from remote,
they also need to be placed in a storage pool.
libvirt knows two different types of storage: volumes and pools.
Storage Volume
A storage volume is a storage device that can be assigned to a guest—a virtual
disk or a CD/DVD/floppy image. Physically (on the VM Host Server) it can be a
block device (a partition, a logical volume, etc.) or a file.
Storage Pool
A storage pool is a storage resource on the VM Host Server that can be used for
storing volumes, similar to network storage for a desktop machine. Physically it
can be one of the following types:
95 z 366 2017-05-27 13:21

File System Directory (dir)

A directory for hosting image files. The files can be either one of the sup-
ported disk formats (raw, qcow2, or qed), or ISO images.
Physical Disk Device (disk)

Use a complete physical disk as storage. A partition is created for each vol-
ume that is added to the pool.
Pre-Formatted Block Device (fs)

Specify a partition to be used in the same way as a file system directory pool
(a directory for hosting image files). The only difference to using a file system
directory is that libvirt takes care of mounting the device.
iSCSI Target (iscsi)

Set up a pool on an iSCSI target. You need to have been logged in to the vol-
ume once before, to use it with libvirt (use the YaST iSCSI Initiator to de-
tect and log in to a volume, see Book “Storage Administration Guide” for
details). Volume creation on iSCSI pools is not supported, instead each exist-
ing Logical Unit Number (LUN) represents a volume. Each volume/LUN also
needs a valid (empty) partition table or disk label before you can use it. If
missing, use fdisk to add it:
~ # fdisk -cu /dev/disk/by-path/ip-192.168.2.100:3260-iscsi-iqn.2010-10.com.exam

Device contains neither a valid DOS partition table, nor Sun, SGI
or OSF disklabel
Building a new DOS disklabel with disk identifier 0xc15cdc4e.
Changes will remain in memory only, until you decide to write them.
After that, of course, the previous content won't be recoverable.
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)
Command (m for help): w

The partition table has been altered!
Calling ioctl() to re-read partition table.

Syncing disks.
LVM Volume Group (logical)

Use an LVM volume group as a pool. You may either use a predefined vol-
ume group, or create a group by specifying the devices to use. Storage vol-
umes are created as partitions on the volume.
96 z 366 2017-05-27 13:21

Warning: Deleting the LVM-Based Pool
When the LVM-based pool is deleted in the Storage Manager, the volume
group is deleted as well. This results in a non-recoverable loss of all data
stored on the pool!
Multipath Devices (mpath)

At the moment, multipathing support is limited to assigning existing devices
to the guests. Volume creation or configuring multipathing from within
libvirt is not supported.
Network Exported Directory (netfs)

Specify a network directory to be used in the same way as a file system di-
rectory pool (a directory for hosting image files). The only difference to using
a file system directory is that libvirt takes care of mounting the directory.
Supported protocols are NFS and GlusterFS.
SCSI Host Adapter (scsi)

Use an SCSI host adapter in almost the same way as an iSCSI target. It is rec-
ommended to use a device name from /dev/disk/by-* rather than
/dev/sd X , since the latter can change (for example, when adding or re-
moving hard disks). Volume creation on iSCSI pools is not supported; in-
stead, each existing LUN (Logical Unit Number) represents a volume.
Warning: Security Considerations
To avoid data loss or data corruption, do not attempt to use resources such as
LVM volume groups, iSCSI targets, etc., that are also used to build storage pools
on the VM Host Server. There is no need to connect to these resources from the
VM Host Server or to mount them on the VM Host Server— libvirt takes care
of this.
Do not mount partitions on the VM Host Server by label. Under certain circum-
stances it is possible that a partition is labeled from within a VM Guest with a
name already existing on the VM Host Server.
12.1 Managing Storage with Virtual Machine

Manager
97 z 366 2017-05-27 13:21

The Virtual Machine Manager provides a graphical interface—the Storage Man-

ager—to manage storage volumes and pools. To access it, either right-click a connec-
tion and choose Details, or highlight a connection and choose Edit › Connection De-
tails. Select the Storage tab.
12.1.1 Adding a Storage Pool

To add a storage pool, proceed as follows:
1. Click Add in the bottom left corner. The dialog Add a New Storage Pool ap-
pears.
2. Provide a Name for the pool (consisting of alphanumeric characters and

_-. ) and select a Type. Proceed with Forward.
98 z 366 2017-05-27 13:21

3. Specify the required details in the following window. The data that needs to
be entered depends on the type of pool you are creating:
Type dir
Target Path: Specify an existing directory.
Type disk
Target Path: The directory that hosts the devices. The default
value /dev should usually fit.
Format: Format of the device's partition table. Using auto should

usually work. If not, get the required format by running the com-
mand parted -l on the VM Host Server.
Source Path: Path to the device. It is recommended to use a de-

vice name from /dev/disk/by-* rather than the simple
/dev/sd X , since the latter can change (for example, when
adding or removing hard disks). You need to specify the path
that resembles the whole disk, not a partition on the disk (if ex-
isting).
Build Pool: Activating this option formats the device. Use with
care—all data on the device will be lost!
99 z 366 2017-05-27 13:21

Type fs
Target Path: Mount point on the VM Host Server file system.
Format: File system format of the device. The default value auto
should work.
Source Path: Path to the device file. It is recommended to use a

device name from /dev/disk/by-* rather than /dev/sd X ,
because the latter can change (for example, when adding or re-
moving hard disks).
Type iscsi
Get the necessary data by running the following command on the VM
Host Server:
iscsiadm --mode node
It will return a list of iSCSI volumes with the following format. The ele-
ments in bold text are required:
IP_ADDRESS:PORT,TPGT TARGET_NAME_(IQN)
Target Path: The directory containing the device file. Use

/dev/disk/by-path (default) or /dev/disk/by-id .
Host Name: Host name or IP address of the iSCSI server.
Source Path: The iSCSI target name (IQN).
Type logical
Target Path: In case you use an existing volume group, specify

the existing device path. When building a new LVM volume
group, specify a device name in the /dev directory that does
not already exist.
Source Path: Leave empty when using an existing volume group.

When creating a new one, specify its devices here.
Build Pool: Only activate when creating a new volume group.
Type mpath
Target Path: Support for multipathing is currently limited to mak-

ing all multipath devices available. Therefore, specify an arbitrary
string here that will then be ignored. The path is required, other-
wise the XML parser will fail.
100 z 366 2017-05-27 13:21

Type netfs
Target Path: Mount point on the VM Host Server file system.
Host Name: IP address or host name of the server exporting the

network file system.
Source Path: Directory on the server that is being exported.
Type scsi
Target Path: The directory containing the device file. Use

/dev/disk/by-path (default) or /dev/disk/by-id .
Source Path: Name of the SCSI adapter.
Note: File Browsing
Using the file browser by clicking Browse is not possible when operating
from remote.
4. Click Finish to add the storage pool.
12.1.2 Managing Storage Pools

Virtual Machine Manager's Storage Manager lets you create or delete volumes in a
pool. You may also temporarily deactivate or permanently delete existing storage
pools. Changing the basic configuration of a pool is currently not supported by
SUSE.
12.1.2.1 Starting, Stopping and Deleting Pools

The purpose of storage pools is to provide block devices located on the VM Host
Server that can be added to a VM Guest when managing it from remote. To make a
pool temporarily inaccessible from remote, click Stop in the bottom left corner of the
Storage Manager. Stopped pools are marked with State: Inactive and are grayed out
in the list pane. By default, a newly created pool will be automatically started On Boot
of the VM Host Server.
To start an inactive pool and make it available from remote again, click Start in the
bottom left corner of the Storage Manager.
101 z 366 2017-05-27 13:21

Note: A Pool's State Does not Affect Attached Volumes
Volumes from a pool attached to VM Guests are always available, regardless of

the pool's state (Active (stopped) or Inactive (started)). The state of the pool solely
affects the ability to attach volumes to a VM Guest via remote management.
To permanently make a pool inaccessible, click Delete in the bottom left corner of the
Storage Manager. You may only delete inactive pools. Deleting a pool does not phys-
ically erase its contents on VM Host Server—it only deletes the pool configuration.
However, you need to be extra careful when deleting pools, especially when deleting
LVM volume group-based tools:
Warning: Deleting Storage Pools
Deleting storage pools based on local file system directories, local partitions or
disks has no effect on the availability of volumes from these pools currently at-
tached to VM Guests.
Volumes located in pools of type iSCSI, SCSI, LVM group or Network Exported Di-
rectory will become inaccessible from the VM Guest if the pool is deleted. Al-
though the volumes themselves will not be deleted, the VM Host Server will no
longer have access to the resources.
Volumes on iSCSI/SCSI targets or Network Exported Directory will become acces-

sible again when creating an adequate new pool or when mounting/accessing
these resources directly from the host system.
When deleting an LVM group-based storage pool, the LVM group definition will
be erased and the LVM group will no longer exist on the host system. The config-
uration is not recoverable and all volumes from this pool are lost.
12.1.2.2 Adding Volumes to a Storage Pool

Virtual Machine Manager lets you create volumes in all storage pools, except in
pools of types Multipath, iSCSI, or SCSI. A volume in these pools is equivalent to a
LUN and cannot be changed from within libvirt .
1. A new volume can either be created using the Storage Manager or while
adding a new storage device to a VM Guest. In either case, select a storage
pool from the left panel, then click Create new volume.
2. Specify a Name for the image and choose an image format.
102 z 366 2017-05-27 13:21

Note that SUSE currently only supports raw , qcow2 , or qed images. The
latter option is not available on LVM group-based pools.
Next to Max Capacity, specify the amount maximum size that the disk image
is allowed to reach. Unless you are working with a qcow2 image, you can
also set an amount for Allocation that should be allocated initially. If both
values differ, a sparse image file will be created which grows on demand.
For qcow2 images, you can use a Backing Store (also called “backing file”)
which constitutes a base image. The newly created qcow2 image will then
only record the changes that are made to the base image.
3. Start the volume creation by clicking Finish.
12.1.2.3 Deleting Volumes From a Storage Pool

Deleting a volume can only be done from the Storage Manager, by selecting a vol-
ume and clicking Delete Volume. Confirm with Yes.
Warning: Volumes Can Be Deleted Even While in Use
Volumes can be deleted even if they are currently used in an active or inactive VM
Guest. There is no way to recover a deleted volume.
Whether a volume is used by a VM Guest is indicated in the Used By column in the

Storage Manager.

Managing storage from the command line is also possible by using virsh . How-
ever, creating storage pools is currently not supported by SUSE. Therefore, this sec-
tion is restricted to documenting functions like starting, stopping and deleting pools
and volume management.
A list of all virsh subcommands for managing pools and volumes is available by
running virsh help pool and virsh help volume , respectively.
12.2.1 Listing Pools and Volumes
103 z 366 2017-05-27 13:21

List all pools currently active by executing the following command. To also list inac-
tive pools, add the option --all :
virsh pool-list --details
Details about a specific pool can be obtained with the pool-info subcommand:
virsh pool-info POOL
Volumes can only be listed per pool by default. To list all volumes from a pool, enter
the following command.
virsh vol-list --details POOL
At the moment virsh offers no tools to show whether a volume is used by a guest
or not. The following procedure describes a way to list volumes from all pools that
are currently used by a VM Guest.
PROCEDURE 12.1: LISTING ALL STORAGE VOLUMES CURRENTLY USED ON A VM HOST SERVER
1. Create an XSLT style sheet by saving the following content to a file, for exam-
ple, ~/libvirt/guest_storage_list.xsl:
<?xml version="1.0" encoding="UTF-8"?>

<xsl:stylesheet version="1.0"
xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:output method="text"/>
<xsl:template match="text()"/>
<xsl:strip-space elements="*"/>
<xsl:template match="disk">
<xsl:text> </xsl:text>
<xsl:value-of select="(source/@file|source/@dev|source/@dir)[1]"/>
<xsl:text>
</xsl:text>
</xsl:template>
</xsl:stylesheet>
2. Run the following commands in a shell. It is assumed that the guest's XML
definitions are all stored in the default location ( /etc/libvirt/qemu ).
xsltproc is provided by the package libxslt .
SSHEET="$HOME/libvirt/guest_storage_list.xsl"
cd /etc/libvirt/qemu
for FILE in *.xml; do
basename $FILE .xml
xsltproc $SSHEET $FILE
done
12.2.2 Starting, Stopping and Deleting Pools
104 z 366 2017-05-27 13:21

Use the virsh pool subcommands to start, stop or delete a pool. Replace POOL
with the pool's name or its UUID in the following examples:
Stopping a Pool
virsh pool-destroy POOL
Note: A Pool's State Does not Affect Attached Volumes
Volumes from a pool attached to VM Guests are always available, regardless

of the pool's state (Active (stopped) or Inactive (started)). The state of the pool
solely affects the ability to attach volumes to a VM Guest via remote manage-
ment.
Deleting a Pool
virsh pool-delete POOL
Warning: Deleting Storage Pools
See Warning: Deleting Storage Pools
Starting a Pool
virsh pool-start POOL
Enable Autostarting a Pool
virsh pool-autostart POOL
Only pools that are marked to autostart will automatically be started if the VM
Host Server reboots.
Disable Autostarting a Pool
virsh pool-autostart POOL --disable
12.2.3 Adding Volumes to a Storage Pool
105 z 366 2017-05-27 13:21

virsh offers two ways to add volumes to storage pools: either from an XML defini-
tion with vol-create and vol-create-from or via command line arguments
with vol-create-as . The first two methods are currently not supported by SUSE,
therefore this section focuses on the subcommand vol-create-as .
To add a volume to an existing pool, enter the following command:
virsh vol-create-as POOL 1 NAME 2 12G --format 3 raw|qcow2|qed 4 --allocation 4G
1 Name of the pool to which the volume should be added

2 Name of the volume
3 Size of the image, in this example 12 gigabytes. Use the suffixes k, M, G, T for
kilobyte, megabyte, gigabyte, and terabyte, respectively.
4 Format of the volume. SUSE currently supports raw , qcow2 , and qed .
5 Optional parameter. By default virsh creates a sparse image file that grows
on demand. Specify the amount of space that should be allocated with this
parameter (4 gigabytes in this example). Use the suffixes k, M, G, T for kilobyte,
megabyte, gigabyte, and terabyte, respectively.
When not specifying this parameter, a sparse image file with no allocation will
be generated. To create a non-sparse volume, specify the whole image size
with this parameter (would be 12G in this example).
12.2.3.1 Cloning Existing Volumes

Another way to add volumes to a pool is to clone an existing volume. The new in-
stance is always created in the same pool as the original.
virsh vol-clone NAME_EXISTING_VOLUME 1 NAME_NEW_VOLUME 2 --pool POOL 3
1 Name of the existing volume that should be cloned

2 Name of the new volume
3 Optional parameter. libvirt tries to locate the existing volume automati-
cally. If that fails, specify this parameter.
12.2.4 Deleting Volumes from a Storage Pool

To permanently delete a volume from a pool, use the subcommand vol-delete :
virsh vol-delete NAME --pool POOL
--pool is optional. libvirt tries to locate the volume automatically. If that fails,
specify this parameter.
106 z 366 2017-05-27 13:21

Warning: No Checks Upon Volume Deletion
A volume will be deleted in any case, regardless of whether it is currently used in

an active or inactive VM Guest. There is no way to recover a deleted volume.
Whether a volume is used by a VM Guest can only be detected by using by the

method described in Procedure 12.1, “Listing all Storage Volumes Currently Used
on a VM Host Server”.
12.3 Locking Disk Files and Block Devices with

virtlockd
Locking block devices and disk files prevents concurrent writes to these resources
from different VM Guests. It provides protection against starting the same VM Guest
twice, or adding the same disk to two different virtual machines. This will reduce the
risk of a virtual machine's disk image becoming corrupted because of a wrong con-
figuration.
The locking is controlled by a daemon called virtlockd . Since it operates indepen-

dently from the libvirtd daemon, locks will endure a crash or a restart of
libvirtd . Locks will even persist in the case of an update of the virtlockd itself,
since it can re-execute itself. This ensures that VM Guests do not need to be restarted
upon a virtlockd update. virtlockd is supported for KVM, QEMU, and Xen.
12.3.1 Enable Locking

Locking virtual disks is not enabled by default on SUSE Linux Enterprise Server. To
enable and automatically start it upon rebooting, perform the following steps:
1. Edit /etc/libvirt/qemu.conf and set
lock_manager = "lockd"
2. Start the virtlockd daemon with the following command:
systemctl start virtlockd
3. Restart the libvirtd daemon with:
107 z 366 2017-05-27 13:21

4. Make sure virtlockd is automatically started when booting the system:
systemctl enable virtlockd
12.3.2 Configure Locking

By default virtlockd is configured to automatically lock all disks configured for
your VM Guests. The default setting uses a "direct" lockspace, where the locks are ac-
quired against the actual file paths associated with the VM Guest <disk> devices. For
example, flock(2) will be called directly on
/var/lib/libvirt/images/my-server/disk0.raw when the VM Guest con-
tains the following <disk> device:
<disk type='file' device='disk'>

<driver name='qemu' type='raw'/>
<source file='/var/lib/libvirt/images/my-server/disk0.raw'/>
<target dev='vda' bus='virtio'/>
</disk>
The virtlockd configuration can be changed by editing the file

/etc/libvirt/qemu-lockd.conf . It also contains detailed comments with fur-
ther information. Make sure to activate configuration changes by reloading
virtlockd :
systemctl reload virtlockd
Note: Locking Currently Only Available for All Disks
Currently, locking can only be activated globally, so that all virtual disks are
locked. Support for locking selected disks is planned for future releases.
12.3.2.1 Enabling an Indirect Lockspace

The default configuration of virtlockd uses a “direct” lockspace. This means that
the locks are acquired against the actual file paths associated with the <disk> de-
vices.
108 z 366 2017-05-27 13:21

If the disk file paths are not accessible to all hosts, virtlockd can be configured to
allow an “indirect” lockspace. This means that a hash of the disk file path is used to
create a file in the indirect lockspace directory. The locks are then held on these hash
files instead of the actual disk file paths. Indirect lockspace is also useful if the file
system containing the disk files does not support fcntl() locks. An indirect
lockspace is specified with the file_lockspace_dir setting:
file_lockspace_dir = "/MY_LOCKSPACE_DIRECTORY"
12.3.2.2 Enable Locking on LVM or iSCSI Volumes

When wanting to lock virtual disks placed on LVM or iSCSI volumes shared by sev-
eral hosts, locking needs to be done by UUID rather than by path (which is used by
default). Furthermore, the lockspace directory needs to be placed on a shared file
system accessible by all hosts sharing the volume. Set the following options for LVM
and/or iSCSI:
lvm_lockspace_dir = "/MY_LOCKSPACE_DIRECTORY"
iscsi_lockspace_dir = "/MY_LOCKSPACE_DIRECTORY"

Sometimes you need to change—extend or shrink—the size of the block device
used by your guest system. For example, when the disk space originally allocated is
no longer enough, it is time to increase its size. If the guest disk resides on a logical
volume, you can resize it while the guest system is running. This is a big advantage
over an offline disk resizing (see the virt-resize command from the Section 17.3,
“Guestfs Tools” package) as the service provided by the guest is not interrupted by
the resizing process. To resize a VM Guest disk, follow these steps:
PROCEDURE 12.2: ONLINE RESIZING OF GUEST DISK
1. Inside the guest system, check the current size of the disk (for example
/dev/vda ).
root # fdisk -l /dev/vda

Disk /dev/sda: 160.0 GB, 160041885696 bytes, 312581808 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
2. On the host, resize the logical volume holding the /dev/vda disk of the
guest to the required size, for example 200 GB.
109 z 366 2017-05-27 13:21

root # lvresize -L 2048M /dev/mapper/vg00-home

Extending logical volume home to 2.00 GiB
Logical volume home successfully resized
3. On the host, resize the block device related to the disk

/dev/mapper/vg00-home of the guest. Note that you can find the
domain_id with virsh list .
root # virsh blockresize --path /dev/vg00/home --size 2048M domain_id
Block device '/dev/vg00/home' is resized
4. Check that the new disk size is accepted by the guest.
root # fdisk -l /dev/vda

Disk /dev/sda: 200.0 GB, 200052357120 bytes, 390727260 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
12.5 Sharing Directories between Host and

Guests (File System Pass-Through)
libvirt allows to share directories between host and guests using QEMU's file system
pass-through (also called VirtFS) feature. Such a directory can be also be accessed by
several VM Guests at once and therefore be used to exchange files between VM
Guests.
Note: Windows Guests and File System Pass-Through
Note that sharing directories between VM Host Server and Windows guests via
File System Pass-Through does not work, because Windows lacks the drivers re-
quired to mount the shared directory.
To make a shared directory available on a VM Guest, proceed as follows:
1. Open the guest's console in Virtual Machine Manager and either choose
View › Details from the menu or click Show virtual hardware details in the
toolbar. Choose Add Hardware › Filesystem to open the Filesystem
Passthrough dialog.
2.
110 z 366 2017-05-27 13:21

Driver allows you to choose between a Handle or Path base driver. The de-
fault setting is Path. Mode lets you choose the security model, which influ-
ences the way file permissions are set on the host. Three options are avail-
able:
Passthrough (Default)
Files on the file system are directly created with the client-user's creden-
tials. This is very similar to what NFSv3 is using.
Squash
Same as Passthrough, but failure of privileged operations like chown are
ignored. This is required when KVM is not run with root privileges.
Mapped
Files are created with the file server's credentials ( qemu.qemu ). The user
credentials and the client-user's credentials are saved in extended at-
tributes. This model is recommended when host and guest domains
should be kept completely isolated.
3. Specify the path to the directory on the VM Host Server with Source Path. En-
ter a string at Target Path that will be used as a tag to mount the shared di-
rectory. Note that the string of this field is a tag only, not a path on the VM
Guest.
4. Apply the setting. If the VM Guest is currently running, you need to shut it
down to apply the new setting (rebooting the guest is not sufficient).
5. Boot the VM Guest. To mount the shared directory, enter the following com-
mand:
sudo mount -t 9p -o trans=virtio,version=9p2000.L,rw TAG /MOUNT_POINT
To make the shared directory permanently available, add the following line
to the /etc/fstab file:
TAG /MOUNT_POINT 9p trans=virtio,version=9p2000.L,rw 0 0
111 z 366 2017-05-27 13:21

Abstract
This chapter introduces common networking configurations supported
by libvirt . It does not depend on the hypervisor used. It is valid for all
hypervisors supported by libvirt , such as KVM or Xen. These setups
can be achieved using both the graphical interface of Virtual Machine
Manager and the command line tool virsh .

There are two common network setups to provide a VM Guest with a network con-
nection:
A virtual network for the guest
A network bridge over a host's physical network interface that the guest can
use

A virtual network is a computer network which does not consist of a physical net-
work link, but rather uses a virtual network link. Each host can have several virtual
networks defined. Virtual networks are based on virtual devices that connect virtual
machines inside a hypervisor. They allow outgoing traffic translated to the LAN and
are provided with DHCP and DNS services. Virtual networks can be either isolated, or
forwarded to a physical network.
112 z 366 2017-05-27 13:21

Guests inside an isolated virtual network can communicate with each other, but can-
not communicate with guests outside the virtual network. Also, guests not belong-
ing to the isolated virtual network cannot communicate with guests inside.
On the other hand, guests inside a forwarded (NAT, network address translation) vir-
tual network can make any outgoing network connection they request. Incoming
connections are allowed from VM Host Server, and from other guests connected to
the same virtual network. All other incoming connections are blocked by iptables
rules.
A standard libvirt installation on SUSE Linux Enterprise Server already comes with a
predefined virtual network providing DHCP server and network address translation
(NAT) named "default".
13.1.1 Managing Virtual Networks with Virtual Machine

Manager
You can define, configure, and operate both isolated and forwarded virtual net-
works with Virtual Machine Manager.
13.1.1.1 Defining Virtual Networks
1. Start Virtual Machine Manager. In the list of available connections, right-click

the name of the connection for which you need to configure the virtual net-
work, and then select Details.
2. In the Connection Details window, click the Virtual Networks tab. You can see
the list of all virtual networks available for the current connection. On the
right, there are details of the selected virtual network.
113 z 366 2017-05-27 13:21

FIGURE 13.1: CONNECTION DETAILS
3. To add a new virtual network, click Add.
4. Specify a name for the new virtual network and click Forward.
114 z 366 2017-05-27 13:21

FIGURE 13.2: CREATE VIRTUAL NETWORK
5. To specify an IPv4 network address space definition, activate the relevant

option and type it into the Network text entry.
115 z 366 2017-05-27 13:21

6. libvirt can provide your virtual network with a DHCP server. If you need
it, activate Enable DHCPv4, then type the start and end IP address range of
assignable addresses.
7. To enable static routing for the new virtual network, activate the relevant op-
tion and type the network and gateway addresses.
8. Click Forward to proceed.
9. To specify IPv6-related options—network address space, DHCPv6 server, or

static route—activate Enable IPv6 network address space definition and acti-
vate the relevant options and fill in the relevant boxes.
10. Click Forward to proceed.
11. Select whether you want isolated or forwarded virtual network.
116 z 366 2017-05-27 13:21

For forwarded networks, specify the network interface to which the re-
quests will be forwarded, and one of the forwarding modes: While NAT (net-
work address translation) remaps the virtual network address space and al-
lows sharing a single IP address, Routed connects the virtual switch to the
physical host LAN with no network translation.
12. If you did not specify IPv6 network address space definition earlier, you can
enable IPv6 internal routing between virtual machines.
13. (Optional) Optionally, change the DNS domain name.
14.
117 z 366 2017-05-27 13:21

Click Finish to create the new virtual network. On the VM Host Server, a new
virtual network bridge virbrX is available, which corresponds to the newly
created virtual network. You can check with brctl show . libvirt auto-
matically adds iptables rules to allow traffic to/from guests attached to the
new virbr X device.
13.1.1.2 Starting Virtual Networks

To start a virtual network that is temporarily stopped, follow these steps:

the list of all virtual networks available for the current connection.
3. To start the virtual network, click Start.
13.1.1.3 Stopping Virtual Networks

To stop an active virtual network, follow these steps:

3. Select the virtual network to be stopped, then click Stop.
13.1.1.4 Deleting Virtual Networks

To delete a virtual network from VM Host Server, follow these steps:

118 z 366 2017-05-27 13:21

3. Select the virtual network to be deleted, then click Delete.
13.1.1.5 Obtaining IP Addresses with nsswitch for NAT Networks

(in KVM)
On VM Host Server, install libvirt-nss, which provides NSS support for libvirt:
zypper in libvirt-nss
Add libvirt to /etc/nsswitch.conf :
...
hosts: files libvirt mdns_minimal [NOTFOUND=return] dns
...
If NSCD is running, restart it:
systemctl restart nscd
Now you can reach the guest system by name from the host.
The NSS module has limited functionality. It reads

/var/lib/libvirt/dnsmasq/*.status files to find the host name and corre-
sponding IP addresses in a JSON record describing each lease provided by dnsmasq
. Host name translation can only be done on those VM Host Servers using a libvirt-
managed bridged network backed by dnsmasq .
For more information, see http://wiki.libvirt.org/page/NSS_module↗.
13.1.2 Managing Virtual Networks with virsh

You can manage libvirt -provided virtual networks with the virsh command
line tool. To view all network related virsh commands, run
# virsh help network

Networking (help keyword 'network'):
net-autostart autostart a network
net-create create a network from an XML file
net-define define (but don't start) a network from an XML fi
net-destroy destroy (stop) a network
net-dumpxml network information in XML
net-edit edit XML configuration for a network
net-event Network Events
net-info network information
net-list list networks
net-name convert a network UUID to network name
net-start start a (previously defined) inactive network
net-undefine undefine an inactive network
119 z 366 2017-05-27 13:21

net-update update parts of an existing network's configurati

net-uuid convert a network name to network UUID
To view brief help information for a specific virsh command, run

virsh help virsh_command :
# virsh help net-create

NAME
net-create - create a network from an XML file
SYNOPSIS
net-create <file>
DESCRIPTION
Create a network.
OPTIONS
[--file] <string> file containing an XML network description
13.1.2.1 Creating a Network

To create a new running virtual network, run
root # virsh net-create vnet_definition.xml
The vnet_definition.xml XML file includes the definition of the virtual network
that libvirt accepts.
To define a new virtual network without activating it, run
root # virsh net-define vnet_definition.xml
The following examples illustrate definitions of different types of virtual networks.
EXAMPLE 13.1: NAT BASED NETWORK
The following configuration allows VM Guests outgoing connectivity if it is avail-

able on VM Host Server. In the absence of VM Host Server networking, it allows
guests to talk directly to each other.
<network>
<name>vnet_nated</name> 1
<bridge name="virbr1" /> 2
<forward mode="nat"/> 3
<ip address="192.168.122.1" netmask="255.255.255.0"> 4
<dhcp>
<range start="192.168.122.2" end="192.168.122.254" /> 5
<host mac="52:54:00:c7:92:da" name="host1.testing.com" \
ip="192.168.1.23.101" /> 6
<host mac="52:54:00:c7:92:db" name="host2.testing.com" \
ip="192.168.1.23.102" />
120 z 366 2017-05-27 13:21

<host mac="52:54:00:c7:92:dc" name="host3.testing.com" \

ip="192.168.1.23.103" />
</dhcp>
</ip>
</network>
1 The name of the new virtual network.

2 The name of the bridge device used to construct the virtual network.
When defining a new network with a <forward> mode of "nat" or "route"
(or an isolated network with no <forward> element), libvirt will auto-
matically generate a unique name for the bridge device if none is given.
3 Inclusion of the <forward> element indicates that the virtual network will
be connected to the physical LAN. The mode attribute specifies the for-
warding method. The most common modes are "nat" (default, network
address translation), "route" (direct forwarding to the physical network,
no address translation), and "bridge" (network bridge configured outside
of libvirt ). If the <forward> element is not specified, the virtual net-
work will be isolated from other networks. For a complete list of forward-
ing modes, see http://libvirt.org/formatnetwork.html#elementsConnect↗.
4 The IP address and netmask for the network bridge.
5 Enable DHCP server for the virtual network, offering IP addresses ranging
from the specified start and end attribute.
6 The optional <host> elements specify hosts which will be given names and
predefined IP addresses by the built-in DHCP server. Any IPv4 host ele-
ment must specify the MAC address of the host to be assigned a given
name, the IP to be assigned to that host, and the name to be given that
host by the DHCP server. An IPv6 host element differs slightly from that
for IPv4: there is no mac attribute since a MAC address has no defined
meaning in IPv6. Instead, the name attribute is used to identify the host to
be assigned the IPv6 address. For DHCPv6, the name is the plain name of
the client host sent by the client to the server. Note that this method of as-
signing a specific IP address can also be used instead of the mac attribute
for IPv4.
EXAMPLE 13.2: ROUTED NETWORK
The following configuration routes traffic from the virtual network to the LAN
without applying any NAT. The IP address range must be preconfigured in the
routing tables of the router on the VM Host Server network.
<network>
<name>vnet_routed</name>
<bridge name="virbr1" />
121 z 366 2017-05-27 13:21

<forward mode="route" dev="eth1"/> 1

<ip address="192.168.122.1" netmask="255.255.255.0">
<dhcp>
<range start="192.168.122.2" end="192.168.122.254" />
</dhcp>
</ip>
</network>
1 The guest traffic may only go out via the eth1 network device on the VM
Host Server.
EXAMPLE 13.3: ISOLATED NETWORK
This configuration provides a completely isolated private network. The guests

can talk to each other, and to VM Host Server, but cannot reach any other ma-
chines on the LAN, as the <forward> element is missing in the XML description.
<network>
<name>vnet_isolated</name>
<bridge name="virbr3" />
<ip address="192.168.152.1" netmask="255.255.255.0">
<dhcp>
<range start="192.168.152.2" end="192.168.152.254" />
</dhcp>
</ip>
</network>
EXAMPLE 13.4: USING AN EXISTING BRIDGE ON VM HOST SERVER
This configuration shows how to use an existing VM Host Server's network

bridge br0 . VM Guests are directly connected to the physical network. Their IP
addresses will all be on the subnet of the physical network, and there will be no
restrictions on incoming or outgoing connections.
<network>
<name>host-bridge</name>
<forward mode="bridge"/>
<bridge name="br0"/>
</network>
13.1.2.2 Listing Networks

To list all virtual networks available to libvirt , run:
122 z 366 2017-05-27 13:21

root # virsh net-list --all
Name State Autostart Persistent

----------------------------------------------------------
crowbar active yes yes
vnet_nated active yes yes
vnet_routed active yes yes
vnet_isolated inactive yes yes
To list available domains, run:
root # virsh list

Id Name State
----------------------------------------------------
1 nated_sles12sp1 running
...
To get a list of interfaces of a running domain, run domifaddr DOMAIN , or option-

ally specify the interface to limit the output to this interface. By default, it addition-
ally outputs their IP and MAC addresses:
root # virsh domifaddr nated_sles12sp1 --interface vnet0 --source lease

Name MAC address Protocol Address
-------------------------------------------------------------------------------
vnet0 52:54:00:9e:0d:2b ipv6 fd00:dead:beef:55::140/64
- - ipv4 192.168.100.168/24
To print brief information of all virtual interfaces associated with the specified do-
main, run:
root # virsh domiflist nated_sles12sp1

Interface Type Source Model MAC
---------------------------------------------------------
vnet0 network vnet_nated virtio 52:54:00:9e:0d:2b
13.1.2.3 Getting Details about a Network

To get detailed information about a network, run:
root # virsh net-info vnet_routed

Name: vnet_routed
UUID: 756b48ff-d0c6-4c0a-804c-86c4c832a498
Active: yes
Persistent: yes
Autostart: yes
Bridge: virbr5
13.1.2.4 Starting a Network
123 z 366 2017-05-27 13:21

To start an inactive network that was already defined, find its name (or unique iden-
tifier, UUID) with:
root # virsh net-list --inactive

----------------------------------------------------------
vnet_isolated inactive yes yes
Then run:
root # virsh net-start vnet_isolated

Network vnet_isolated started
13.1.2.5 Stopping a Network

To stop an active network, find its name (or unique identifier, UUID) with:
root # virsh net-list --inactive

----------------------------------------------------------
vnet_isolated active yes yes
Then run:
root # virsh net-destroy vnet_isolated

Network vnet_isolated destroyed
13.1.2.6 Removing a Network

To remove the definition of an inactive network from VM Host Server permanently,
run:
root # virsh net-undefine vnet_isolated

Network vnet_isolated has been undefined

A network bridge is used to connect two or more network segments. It behaves like
a virtual network switch, and guest machines treat it transparently as a physical net-
work interface. Any physical or virtual device can be connected to the bridge.
If there is a network bridge present on VM Host Server, you can connect a VM Guest
to it directly. This provides the VM Guest with full incoming and outgoing network
access.
124 z 366 2017-05-27 13:21

13.2.1 Managing Network Bridges with YaST

This section includes procedures to add or remove network bridges with YaST.
13.2.1.1 Adding a Network Bridge

To add a network bridge on VM Host Server, follow these steps:
1. Start YaST › System › Network Settings.
2. Activate the Overview tab and click Add.
3. Select Bridge from the Device Type list and enter the bridge device interface
name in the Configuration Name entry. Proceed with Next.
4. In the Address tab, specify networking details such as DHCP/static IP address,

subnet mask or host name.
Using Dynamic Address is only useful when also assigning a device to a

bridge that is connected to some DHCP server.
If you intend to create a virtual bridge that has no connection to a real Eth-
ernet device, use Statically assigned IP Address. In this case, it is a good idea
to use addresses from the private IP address ranges, for example,
192.168.x.x or 10.x.x.x .
To create a bridge that should only serve as a connection between the differ-
ent guests without connection to the host system, set the IP address to
0.0.0.0 and the subnet mask to 255.255.255.255 . The network scripts
handle this special address as an unset IP address.
5. Activate the Bridged Devices tab and activate the network devices you want
to include in the network bridge.
6. Click Next to return to the Overview tab and confirm with OK. The new net-
work bridge should be active on VM Host Server now.
13.2.1.2 Deleting a Network Bridge

To delete an existing network bridge, follow these steps:
125 z 366 2017-05-27 13:21

1. Start YaST › System › Network Settings.
2. Select the bridge device you want to delete from the list in the Overview tab.
3. Delete the bridge with Delete and confirm with OK.
13.2.2 Managing Network Bridges with brctl

This section includes procedures to add or remove network bridges with the brctl
command line tool.
13.2.2.1 Adding a Network Bridge

To add a new network bridge device on VM Host Server with brctl , follow these
steps:
1. Log in as root on the VM Host Server where you want to create a new net-
work bridge.
2. Choose a name for the new bridge— virbr_test in our example— and
run
root # brctl addbr virbr_test
3. Check if the bridge was created on VM Host Server:
root # brctl show

bridge name bridge id STP enabled interfaces
br0 8000.e06995ec09e8 no eth0
virbr0 8000.525400b37ec9 yes virbr0-nic
virbr_test 8000.000000000000 no
virbr_test is present, but is not associated with any physical network in-
terface.
4. Add a network interface to the bridge:
root # brctl addif eth1
Important: Network Interface Must Be Unused
You can only enslave a network interface that is not yet used by other
network bridge.
5. Optionally, enable STP (see Spanning Tree Protocol↗):
126 z 366 2017-05-27 13:21

root # brctl stp virbr_test on
13.2.2.2 Deleting a Network Bridge

To delete an existing network bridge device on VM Host Server with brctl , follow
these steps:
1. Log in as root on the VM Host Server where you want to delete an existing
network bridge.
2. List existing network bridges to identify the name of the bridge to remove:
root # brctl show

br0 8000.e06995ec09e8 no eth0
virbr0 8000.525400b37ec9 yes virbr0-nic
virbr_test 8000.000000000000 yes eth1
3. Delete the bridge:
root # brctl delbr virbr_test
13.2.3 Using VLAN Interfaces

Sometimes, it is necessary to create a private connection either between two VM
Host Servers or between VM Guest systems. For example, to migrate VM Guest to
hosts in a different network segment, or to create a private bridge that only VM
Guest systems may connect to, even when running on different VM Host Server sys-
tems. An easy way to build such connections is to set up VLAN networks.
VLAN interfaces are commonly set up on the VM Host Server and either interconnect
the different VM Host Server systems, or they may be set up as a physical interface
to an otherwise virtual-only bridge. It is even possible to create a bridge with a VLAN
as a physical interface that has no IP address in the VM Host Server. That way, the
guest systems have no possibility to access the host over this network.
Run the YaST module System › Network Settings. Follow this procedure to set up the
VLAN device:
PROCEDURE 13.1: SETTING UP VLAN INTERFACES WITH YAST
1. Press Add to create a new network interface.
2. In the Hardware Dialog, select Device Type VLAN.
127 z 366 2017-05-27 13:21

3. Change the value of Configuration Name to the ID of your VLAN. Note that
VLAN ID 1 is commonly used for management purposes.
4. Press Next.
5. Select the interface that the VLAN device should connect to below Real Inter-
face for VLAN. If the desired interface does not appear in the list, first set up
this interface without an IP Address.
6. Select the desired method for assigning an IP address to the VLAN device.
7. Press Next to finish the configuration.
It is also possible to use the VLAN interface as a physical interface of a bridge. This
makes it possible to connect several VM Host Server-only networks and allows to
live-migrate VM Guest systems that are connected to such a network.
YaST does not always allow to set no IP address. However, this may be a desired fea-
ture especially if VM Host Server-only networks should be connected. In this case,
use the special address 0.0.0.0 with netmask 255.255.255.255 . The system
scripts handle this address as no IP address set.

Abstract
Virtual Machine Manager's Details view offers in-depth information about
the VM Guest's complete configuration and hardware equipment. Using
this view, you can also change the guest configuration or add and modify
virtual hardware. To access this view, open the guest's console in Virtual
Machine Manager and either choose View › Details from the menu, or click
Show virtual hardware details in the toolbar.
14.1 Machine Setup

14.2 Storage
14.3 Controllers
14.4 Networking
128 z 366 2017-05-27 13:21

14.5 Enabling Seamless and Synchronized Mouse Pointer Movement

14.6 Adding a CD/DVD-ROM Device with Virtual Machine Manager
14.7 Adding a Floppy Device with Virtual Machine Manager
14.8 Ejecting and Changing Floppy or CD/DVD-ROM Media with Virtual
Machine Manager
FIGURE 14.1: DETAILS VIEW OF A VM GUEST
129 z 366 2017-05-27 13:21

The left panel of the window lists VM Guest overview and already installed hard-
ware. After clicking an item in the list, you can access its detailed settings in the de-
tails view. You can change the hardware parameters to match your needs, then click
Apply to confirm them. Some changes take effect immediately, while others need a
reboot of the machine—and virt-manager warns you about that fact.
To remove installed hardware from a VM Guest, select the appropriate list entry in
the left panel and then click Remove in the bottom right of the window.
To add new hardware, click Add Hardware below the left panel, then select the type
of the hardware you want to add in the Add New Virtual Hardware window. Modify
its parameters and confirm with Finish.
The following sections describe configuration options for the specific hardware type
being added. They do not focus on modifying an existing piece of hardware as the
options are identical.
14.1 Machine Setup

This section describes the setup of the virtualized processor and memory hardware.
These components are vital to a VM Guest, therefore you cannot remove them. It
also shows how to view the overview and performance information, and how to
change boot options.
14.1.1 Overview
Overview shows basic details about VM Guest and the hypervisor.
130 z 366 2017-05-27 13:21

FIGURE 14.2: OVERVIEW DETAILS
Name, Title, and Description are editable and help you identify VM Guest in the Vir-
tual Machine Manager list of machines.
FIGURE 14.3: VM GUEST TITLE AND DESCRIPTION
UUID shows the universally unique identifier of the virtual machine, while Status
shows its current status—Running, Paused, or Shutoff.
The Hypervisor Details section shows the hypervisor type, CPU architecture, used em-
ulator, and chipset type. None of the hypervisor parameters can be changed.
131 z 366 2017-05-27 13:21

14.1.2 Performance
Performance shows regularly updated charts of CPU and memory usage, and disk
and network I/O.
FIGURE 14.4: PERFORMANCE
Tip: Enabling Disabled Charts
Not all the charts in the Graph view are enabled by default. To enable these
charts, go to File › View Manager, then select Edit › Preferences › Polling, and check
the charts that you want to see regularly updated.
132 z 366 2017-05-27 13:21

FIGURE 14.5: STATISTICS CHARTS
14.1.3 Processor
Processor includes detailed information about VM Guest processor configuration.
133 z 366 2017-05-27 13:21

FIGURE 14.6: PROCESSOR VIEW
In the CPUs section, you can configure several parameters related to the number of
allocated CPUs.
Logical host CPUs

The real number of CPUs installed on VM Host Server.
Current allocation
The number of currently allocated CPUs. You can hotplug more CPUs by increas-
ing this value up to the Maximum allocation value.
134 z 366 2017-05-27 13:21

Maximum allocation
Maximum number of allocable CPUs for the current session. Any change to this
value will take effect after the next VM Guest reboot.
The Configuration section lets you configure the CPU model and topology.
When activated, the Copy host CPU configuration option uses the host CPU model for
VM Guest. Otherwise you need to specify the CPU model from the drop-down box.
After you activate Manually set CPU topology, you can specify a custom number of
sockets, cores and threads for the CPU.
14.1.4 Memory
Memory contains information about the memory that is available to VM Guest.
FIGURE 14.7: MEMORY VIEW
Total host memory

Total amount of memory installed on VM Host Server.
135 z 366 2017-05-27 13:21

Current allocation
The amount of memory currently available to VM Guest. You can hotplug more
memory by increasing this value up to the value of Maximum allocation.
Maximum allocation
The maximum value to which you can hotplug the currently available memory.
Any change to this value will take effect after the next VM Guest reboot.
14.1.5 Boot Options

Boot Options introduces options affecting the VM Guest boot process.
136 z 366 2017-05-27 13:21

FIGURE 14.8: BOOT OPTIONS
In the Autostart section, you can specify whether the virtual machine should auto-
matically start during the VM Host Server boot phase.
In the Boot device order, activate the devices that will be used for booting VM Guest.
You can change their order with the up and down arrow buttons on the right side of
the list. To choose from a list of bootable devices on VM Guest start, activate Enable
boot menu.
To boot a different kernel than the one on the boot device, activate Enable direct ker-
nel boot and specify the paths to the alternative kernel and initrd placed on the VM
Host Server file system. You can also specify kernel arguments that will be passed to
the loaded kernel.
137 z 366 2017-05-27 13:21

14.2 Storage
This section gives you a detailed description of configuration options for storage de-
vices. It includes both hard disks and removable media, such as USB or CD-ROM
drives.
PROCEDURE 14.1: ADDING A NEW STORAGE DEVICE
1. Click Add Hardware below the left panel, then select Storage from the Add
New Virtual Hardware window.
138 z 366 2017-05-27 13:21

FIGURE 14.9: ADD A NEW STORAGE
2. To create a qcow2 disk image in the default location, activate Create a disk
image for the virtual machine and specify its size in gigabytes.
To gain more control over the disk image creation, activate Select or create
custom storage and click Manage to manage storage pools and images. The
window Choose Storage Volume opens which has almost identical functional-
ity as the Storage tab described in Section 12.1, “Managing Storage with Vir-
tual Machine Manager”.
139 z 366 2017-05-27 13:21

Tip: Supported Storage Formats
SUSE only supports the following storage formats: raw , qcow2 , and
qed .
3. After you manage to create and specify the disk image file, specify the Device
type. It can be one of the following options:
Disk device
CDROM device: Does not allow using Create a disk image for the virtual
machine.
Floppy device: Does not allow using Create a disk image for the virtual
machine.
LUN Passthrough: Required to use an existing SCSI storage directly

without adding it into a storage pool.
4. Select the Bus type for your device. The list of available options depends on
the device type you selected in the previous step. The types based on VirtIO
use paravirtualized drivers.
5. In the Advanced options section, select the preferred Cache mode. For more
information on cache modes, see Chapter 15, Disk Cache Modes.
6. Confirm your settings with Finish. A new storage device appears in the left
panel.
14.3 Controllers
This section focuses on adding and configuring new controllers.
PROCEDURE 14.2: ADDING A NEW CONTROLLER
1. Click Add Hardware below the left panel, then select Controller from the Add
140 z 366 2017-05-27 13:21

FIGURE 14.10: ADD A NEW CONTROLLER
2. Select the type of the controller. You can choose from IDE, Floppy, SCSI, SATA,
VirtIO Serial (paravirtualized), USB, or CCID (smart card devices).
3. Optionally, in the case of a USB or SCSI controller, select a controller model.
4. Confirm your settings with Finish. A new controller appears in the left panel.
14.4 Networking
This section describes how to add and configure new network devices.
PROCEDURE 14.3: ADDING A NEW NETWORK DEVICE
1. Click Add Hardware below the left panel, then select Network from the Add
141 z 366 2017-05-27 13:21

FIGURE 14.11: ADD A NEW CONTROLLER
2. From the Network source list, select the source for the network connection.
The list includes VM Host Server's available physical network interfaces, net-
work bridges, or network bonds. You can also assign the VM Guest to an al-
ready defined virtual network. See Chapter 13, Managing Networks for more
information on setting up virtual networks with Virtual Machine Manager.
3. Specify a MAC address for the network device. While Virtual Machine Man-
ager pre-fills a random value for your convenience, it is recommended to
supply a MAC address appropriate for your network environment to avoid
network conflicts.
4. Select a device model from the list. You can either leave the Hypervisor de-
fault, or specify one of e1000, rtl8139, or virtio models. Note that virtio uses
paravirtualized drivers.
142 z 366 2017-05-27 13:21

5. Confirm your settings with Finish. A new network device appears in the left
panel.
14.5 Enabling Seamless and Synchronized

Mouse Pointer Movement
When you click within a VM Guest's console with the mouse, the pointer is captured
by the console window and cannot be used outside the console unless it is explicitly
released (by pressing Alt – Ctrl ). To prevent the console from grabbing the key and
to enable seamless pointer movement between host and guest instead, add a tablet
to the VM Guest.
Adding a tablet has the additional advantage of synchronizing the mouse pointer
movement between VM Host Server and VM Guest when using a graphical environ-
ment on the guest. With no tablet configured on the guest, you will often see two
pointers with one dragging behind the other.
1. Double-click a VM Guest entry in the Virtual Machine Manager to open its

console and switch to the Details view with View › Details.
2. Click Add Hardware and choose Input and then EvTouch USB Graphics Tablet
in the pop-up window. Proceed with Finish.
3. If the guest is running, you will be asked whether to enable the tablet after
the next reboot. Confirm with Yes.
4. When you start or restart the VM Guest, the tablet becomes available in the
VM Guest.
14.6 Adding a CD/DVD-ROM Device with Virtual

Machine Manager
KVM supports CD or DVD-ROMs in VM Guest either by directly accessing a physical
drive on the VM Host Server or by accessing ISO images. To create an ISO image
from an existing CD or DVD, use dd :
dd if=/dev/cd_dvd_device of=my_distro.iso bs=2048
To add a CD/DVD-ROM device to your VM Guest, proceed as follows:
143 z 366 2017-05-27 13:21


2. Click Add Hardware and choose Storage in the pop-up window.
3. Change the Device Type to IDE CDROM.
4. Select Select or create custom storage.
a. To assign the device to a physical medium, enter the path to the VM

Host Server's CD/DVD-ROM device (for example, /dev/cdrom ) next
to Manage. Alternatively, use Manage to open a file browser and then
click Browse Local to select the device. Assigning the device to a physi-
cal medium is only possible when the Virtual Machine Manager was
started on the VM Host Server.
b. To assign the device to an existing image, click Manage to choose an

image from a storage pool. If the Virtual Machine Manager was
started on the VM Host Server, alternatively choose an image from an-
other location on the file system by clicking Browse Local. Select an im-
age and close the file browser with Choose Volume.
5. Save the new virtualized device with Finish.
6. Reboot the VM Guest to make the new device available. For more
information, see Section 14.8, “Ejecting and Changing Floppy or
CD/DVD-ROM Media with Virtual Machine Manager”.
14.7 Adding a Floppy Device with Virtual

Machine Manager
Currently KVM only supports the use of floppy disk images—using a physical floppy
drive is not supported. Create a floppy disk image from an existing floppy using dd :
dd if=/dev/fd0 of=/var/lib/libvirt/images/floppy.img
To create an empty floppy disk image use one of the following commands:
Raw Image
dd if=/dev/zero of=/var/lib/libvirt/images/floppy.img bs=512 count=2880
FAT Formatted Image
144 z 366 2017-05-27 13:21

mkfs.msdos -C /var/lib/libvirt/images/floppy.img 1440
To add a floppy device to your VM Guest, proceed as follows:

2. Click Add Hardware and choose Storage in the pop-up window.
3. Change the Device Type to Floppy Disk.
4. Choose Select or create custom storage and click Manage to choose an exist-
ing image from a storage pool. If Virtual Machine Manager was started on
the VM Host Server, alternatively choose an image from another location on
the file system by clicking Browse Local. Select an image and close the file
browser with Choose Volume.
5. Save the new virtualized device with Finish.
6. Reboot the VM Guest to make the new device available. For more
information, see Section 14.8, “Ejecting and Changing Floppy or
CD/DVD-ROM Media with Virtual Machine Manager”.
14.8 Ejecting and Changing Floppy or

CD/DVD-ROM Media with Virtual Machine
Manager
Regardless of whether you are using the VM Host Server's physical CD/DVD-ROM
device or an ISO/floppy image, before you can change the media or image of an ex-
isting device in the VM Guest, you first need to disconnect the media from the
guest.

2. Choose the Floppy or CD/DVD-ROM device and “eject” the medium by click-
ing Disconnect.
3. To “insert” a new medium, click Connect.
a.
145 z 366 2017-05-27 13:21

If using the VM Host Server's physical CD/DVD-ROM device, first

change the media in the device (this may require unmounting it on the
VM Host Server before it can be ejected). Then choose CD-ROM or DVD
and select the device from the drop-down box.
b. If you are using an ISO image, choose ISO image Location and select an
image by clicking Manage. When connecting from a remote host, you
may only choose images from existing storage pools.
4. Click OK to finish. The new media can now be accessed in the VM Guest.

By default, when installing with the virt-install tool, the machine type for VM
Guest is pc-i440fx. The machine type is stored in the VM Guest's xml configuration
file in /etc/libvirt/qemu/ in the tag type :
<type arch='x86_64' machine='pc-i440fx-2.3'>hvm</type>
As an example, the following procedure shows how to change this value to the ma-
chine type q35 . q35 is an Intel* chipset. It includes PCIe, supports up to 12 USB
ports, and has support for SATA and IOMMU. IRQ routing has also been improved.
1. Check whether your VM Guest is inactive:
virsh list --inactive

Id Name State
----------------------------------------------------
- sles11 shut off
2. Edit the configuration for this VM Guest:
virsh edit sles11
3. Change the value of the machine attribute:
<type arch='x86_64' machine='pc-q35-2.0'>hvm</type>
4. Restart the VM Guest.
root # virsh start sles11
5. Check that the machine type has changed. Log in to the VM Guest as root
and run the following command:
root # dmidecode | grep Product
146 z 366 2017-05-27 13:21

Product Name: Standard PC (Q35 + ICH9, 2009)
Tip: Machine Type Update Recommendations
Whenever the QEMU version on the host system is upgraded, for example, when
upgrading the VM Host Server to a new service pack, it is also recommended to
upgrade the machine type of the VM Guests to the latest available version. To
check, use the command qemu-system-x86_64 -M help on the VM Host
Server.
The default machine type pc-i440fx , for example, is regularly updated. If your
VM Guest still runs with a machine type of pc-i440fx-1.x , an update to
pc-i440fx-2.x is strongly recommended. This allows taking advantage of the
most recent updates and corrections in machine definitions, and ensures better
future compatibility.

You can directly assign host-PCI devices to guests (PCI pass-through). When the PCI
device is assigned to one VM Guest, it cannot be used on the host or by another VM
Guest unless it is re-assigned. A prerequisite for this feature is a VM Host Server con-
figuration as described in Important: Requirements for VFIO and SR-IOV.
14.10.1 Adding a PCI Device with Virtual Machine Manager

The following procedure describes how to add a PCI device to a VM Guest using Vir-
tual Machine Manager:

2. Click Add Hardware and choose the PCI Host Device category in the left panel.
A list of available PCI devices appears in the right part of the window.
147 z 366 2017-05-27 13:21

FIGURE 14.12: ADDING A PCI DEVICE
3. From the list of available PCI devices, choose the one you want to pass to
the guest. Confirm with Finish.
Tip: Assigning a PCI Device Requires a VM Guest Shutdown
Although it is possible to assign a PCI device to a running VM Guest as described

above, the device will not become available until you shut down the VM Guest
and reboot it afterward.
14.10.2 Adding a PCI Device with virsh

To assign a PCI device to VM Guest with virsh , follow these steps:
1. Identify the host PCI device to assign to the guest. In the following example,
we are assigning a DEC network card to the guest:
148 z 366 2017-05-27 13:21

tux > sudo lspci -nn

[...]
03:07.0 Ethernet controller [0200]: Digital Equipment Corporation DECchip \
21140 [FasterNet] [1011:0009] (rev 22)
[...]
Note down the device ID ( 03:07.0 in this case).
2. Gather detailed information about the device using

virsh nodedev-dumpxml ID . To get the ID , you need to replace colon
and period in the device ID ( 03:07.0 ) with underscore and prefix the re-
sult with “pci_0000_” ( pci_0000_03_07_0 ).
tux > virsh nodedev-dumpxml pci_0000_03_07_0

<device>
<name>pci_0000_03_07_0</name>
<path>/sys/devices/pci0000:00/0000:00:14.4/0000:03:07.0</path>
<parent>pci_0000_00_14_4</parent>
<driver>
<name>tulip</name>
</driver>
<capability type='pci'>
<domain>0</domain>
<bus>3</bus>
<slot>7</slot>
<function>0</function>
<product id='0x0009'>DECchip 21140 [FasterNet]</product>
<vendor id='0x1011'>Digital Equipment Corporation</vendor>
<numa node='0'/>
</capability>
</device>
Note down the values for domain, bus, and function.
3. Detach the device from the host system prior to attaching it to VM Guest.
tux > virsh nodedev-detach pci_0000_03_07_0

Device pci_0000_03_07_0 detached
Tip: Multi-Function PCI Devices
When using a multi-function PCI device that does not support FLR (func-
tion level reset) or PM (power management) reset, you need to detach all
its functions from the VM Host Server. The whole device must be reset for
security reasons. libvirt will refuse to assign the device if one of its
functions is still in use by the VM Host Server or another VM Guest.
4.
149 z 366 2017-05-27 13:21

Convert the domain, bus, slot, and function value from decimal to hexadeci-
mal, and prefix with 0x to tell the system that the value is hexadecimal. In
our example, domain = 0, bus = 3, slot = 7, and function = 0. Their hexadeci-
mal values are:
tux > printf %x 0

0
tux > printf %x 3
3
tux > printf %x 7
7
This results in domain = 0x0000, bus = 0x03, slot = 0x07 and function =
0x00.
5. Run virsh edit on your domain, and add the following device entry in
the <devices> section using the values from the previous step:
<hostdev mode='subsystem' type='pci' managed='yes'>

<source>
<address domain='0x0000' bus='0x03' slot='0x07' function='0x00'/>
</source>
</hostdev>
Tip: managed Compared to unmanaged
libvirt recognizes two modes for handling PCI devices: they can be ei-
ther managed or unmanaged . In the managed case, libvirt will han-
dle all the details of unbinding the device from the existing driver if
needed, resetting the device, binding it to vfio-pci before starting the
domain, etc. When the domain is terminated or the device is removed
from the domain, libvirt will unbind from vfio-pci and rebind to
the original driver in the case of a managed device. If the device is un-
managed, the user must take care to ensure all of these management as-
pects of the device are done before assigning it to a domain, and after
the device is no longer used by the domain.
In the example above, the managed='yes' option means that the de-
vice is managed. To switch the device mode to unmanaged, set
managed='no' in the listing above. If you do so, you need to take care
of the related driver with the virsh nodedev-detach and
virsh nodedev-reattach commands. That means you need to run
virsh nodedev-detach pci_0000_03_07_0 prior to starting the VM
Guest to detach the device from the host. In case the VM Guest is not run-
ning, you can make the device available for the host by running
virsh nodedev-reattach pci_0000_03_07_0 .
150 z 366 2017-05-27 13:21

6. Shut down the VM Guest and restart it to make the assigned PCI device avail-
able.
Tip: SELinux
If you are running SELinux on your VM Host Server, you need to disable
it prior to starting the VM Guest with
setsebool -P virt_use_sysfs 1

Single Root I/O Virtualization (SR-IOV) capable PCIe devices can replicate their re-
sources, so they appear to be multiple devices. Each of these "pseudo-devices" can
be assigned to a VM Guest.
SR-IOV is an industry specification that was created by the Peripheral Component In-
terconnect Special Interest Group (PCI-SIG) consortium. It introduces physical func-
tions (PF) and virtual functions (VF). PFs are full PCIe functions used to manage and
configure the device. PFs also can move data. VFs lack the configuration and man-
agement part—they only can move data and a reduced set of configuration func-
tions. Since VFs do not have all PCIe functions, the host operating system or the Hy-
pervisor must support SR-IOV to be able to access and initialize VFs. The theoretical
maximum for VFs is 256 per device (consequently the maximum for a dual-port Eth-
ernet card would be 512). In practice this maximum is much lower, since each VF
consumes resources.
14.11.1 Requirements
The following requirements must be met to be able to use SR-IOV:
An SR-IOV-capable network card (as of SUSE Linux Enterprise Server 12 SP2,

only network cards support SR-IOV)
An AMD64/Intel 64 host supporting hardware virtualization (AMD-V or Intel

VT-x), see Section 7.3, “KVM Hardware Requirements” for more information
A chipset that supports device assignment (AMD-Vi or Intel VT-d)
libvirt-0.9.10 or better
151 z 366 2017-05-27 13:21

SR-IOV drivers must be loaded and configured on the host system
A host configuration that meets the requirements listed at Important: Re-

quirements for VFIO and SR-IOV
A list of the PCI addresses of the VF(s) that will be assigned to VM Guests
Tip: Checking if a Device is SR-IOV-Capable
The information whether a device is SR-IOV-capable can be obtained from its PCI
descriptor by running lspci . A device that supports SR-IOV reports a capability
similar to the following:
Capabilities: [160 v1] Single Root I/O Virtualization (SR-IOV)
Note: Adding an SR-IOV Device at VM Guest Creation
Before adding an SR-IOV device to a VM Guest when initially setting it up, the VM
Host Server already needs to be configured as described in Section 14.11.2,
“Loading and Configuring the SR-IOV Host Drivers”.
14.11.2 Loading and Configuring the SR-IOV Host Drivers

To be able to access and initialize VFs, an SR-IOV-capable driver needs to be loaded
on the host system.
1. Before loading the driver, make sure the card is properly detected by run-
ning lspci . The following example shows the lspci output for the
dual-port Intel 82576NS network card:
tux > sudo /sbin/lspci | grep 82576

01:00.0 Ethernet controller: Intel Corporation 82576NS Gigabit Network Connection
In case the card is not detected, it is likely that the hardware virtualization
support in the BIOS/EFI has not been enabled.
2.
152 z 366 2017-05-27 13:21

Check whether the SR-IOV driver is already loaded by running lsmod . In

the following example a check for the igb driver (for the Intel 82576NS net-
work card) returns a result. That means the driver is already loaded. If the
command returns nothing, the driver is not loaded.
tux > sudo /sbin/lsmod | egrep "^igb "

igb 185649 0
3. Skip this step if the driver is already loaded.
If the SR-IOV driver is not yet loaded, the non-SR-IOV driver needs to be re-
moved first, before loading the new driver. Use rmmod to unload a driver.
The following example unloads the non-SR-IOV driver for the Intel 82576NS
network card:
sudo /sbin/rmmod igbvf
Load the SR-IOV driver subsequently using the modprobe command—the

VF parameter ( max_vfs ) is mandatory:
sudo /sbin/modprobe igb max_vfs=8
Or load the driver via SYSFS:
Find the PCI ID of the physical NIC by listing Ethernet devices:
tux > sudo lspci | grep Eth

06:00.0 Ethernet controller: Emulex Corporation OneConnect NIC (Skyhawk) (rev 10)
To enable VFs, echo the number of desired VFs to load to the

sriov_numvfs parameter:
tux > sudo echo 1 > /sys/bus/pci/devices/0000:06:00.1/sriov_numvfs
Verify that the VF NIC was loaded:
tux > sudo lspci | grep Eth

Obtain the maximum number of VFs available:
tux > sudo lspci -vvv -s 06:00.1 | grep 'Initial VFs'

Initial VFs: 32, Total VFs: 32, Number of VFs: 0,
Function Dependency Link: 01
4. Create a before.service file which loads VF via SYSFS on boot:
[Unit]
153 z 366 2017-05-27 13:21

Before=
[Service]
Type=oneshot
RemainAfterExit=true
ExecStart=/bin/bash -c "echo 1 > /sys/bus/pci/devices/0000:06:00.1/sriov_numvfs"
# beware, executable is run directly, not through a shell, check the man pages
# systemd.service and systemd.unit for full syntax
[Install]
# target in which to start the service
WantedBy=multi-user.target
#WantedBy=graphical.target
And copy it to /etc/systemd/system .
Additionally, it is required to create another service file (

after-local.service ) pointing to /etc/init.d/after.local script
that detaches the NIC prior to starting the VM, otherwise the VM would fail
to start:
[Unit]
Description=/etc/init.d/after.local Compatibility
After=libvirtd.service
Requires=libvirtd.service
[Service]
Type=oneshot
ExecStart=/etc/init.d/after.local
RemainAfterExit=true
[Install]
WantedBy=multi-user.target
And copy it to /etc/systemd/system .
#! /bin/sh
#
# Copyright (c) 2010 SuSE LINUX Products GmbH, Germany. All rights reserved.
# ...
virsh nodedev-detach pci_0000_06_08_0
Then save it as /etc/init.d/after.local .
5. Reboot the machine and check if the SR-IOV driver is loaded by re-running
the lspci command from the first step of this procedure. If the SR-IOV
driver was loaded successfully you should see additional lines for the VFs:

01:10.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
[...]
154 z 366 2017-05-27 13:21

[...]
14.11.3 Adding a VF Network Device to an Existing VM Guest

When the SR-IOV hardware is properly set up on the VM Host Server, you can add
VFs to VM Guests. To do so, you need to collect some data first.
Note: The following procedure is using example data. Make sure to replace it
by appropriate data from your setup.
1. Use the virsh nodedev-list command to get the PCI address of the VF
you want to assign and its corresponding PF. Numerical values from the
lspci output shown in Section 14.11.2, “Loading and Configuring the
SR-IOV Host Drivers” (for example 01:00.0 or 04:00.1 ) are transformed
by adding the prefix "pci_0000_" and by replacing colons and dots with un-
derscores. So a PCI ID listed as "04:00.0" by lspci is listed as
"pci_0000_04_00_0" by virsh. The following example lists the PCI IDs for the
second port of the Intel 82576NS network card:
tux > sudo virsh nodedev-list | grep 0000_04_

pci_0000_04_00_0
pci_0000_04_00_1
pci_0000_04_10_0
pci_0000_04_10_1
pci_0000_04_10_2
pci_0000_04_10_3
pci_0000_04_10_4
pci_0000_04_10_5
pci_0000_04_10_6
pci_0000_04_10_7
pci_0000_04_11_0
pci_0000_04_11_1
pci_0000_04_11_2
pci_0000_04_11_3
pci_0000_04_11_4
pci_0000_04_11_5
The first two entries represent the PFs, whereas the other entries represent
the VFs.
2. Get more data that will be needed by running the command

virsh nodedev-dumpxml on the PCI ID of the VF you want to add:
tux > sudo virsh nodedev-dumpxml pci_0000_04_10_0

<device>
<name>pci_0000_04_10_0</name>
<parent>pci_0000_00_02_0</parent>
<capability type='pci'>
155 z 366 2017-05-27 13:21

<domain>0</domain>
<bus>4</bus>
<slot>16</slot>
<product id='0x10ca'>82576 Virtual Function</product>
<vendor id='0x8086'>Intel Corporation</vendor>
<capability type='phys_function'>
<address domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
</capability>
</capability>
</device>
The following data is needed for the next step:
<domain>0</domain>
<bus>4</bus>
<slot>16</slot>
3. Create a temporary XML file (for example /tmp/vf-interface.xml con-

taining the data necessary to add a VF network device to an existing VM
Guest. The minimal content of the file needs to look like the following:
<interface type='hostdev'> 1
<source>
<address type='pci' domain='0' bus='11' slot='16' function='0'2/>
</source>
</interface>
1 VFs do not get a fixed MAC address; it changes every time the host re-
boots. When adding network devices the “traditional” way with <host-
dev>, it would require to reconfigure the VM Guest's network device
after each reboot of the host, because of the MAC address change. To
avoid this kind of problem, libvirt introduced the “interface
type='hostdev'” directive, which sets up network-specific data before
assigning the device.
2 Specify the data you acquired in the previous step here.
4. In case a device is already attached to the host, it cannot be attached to a

guest. To make it available for guests, detach it from the host first:
virsh nodedev-detach pci_0000_04_10_0
5. Last, add the VF interface to an existing VM Guest:
virsh attach-device GUEST /tmp/vf-interface.xml --OPTION
156 z 366 2017-05-27 13:21

GUEST needs to be replaced by the domain name, id or uuid of the VM

Guest and -- OPTION can be one of the following:
--persistent
This option will always add the device to the domain's persistent XML. In
addition, if the domain is running, it will be hotplugged.
--config
This option will only affect the persistent XML, even if the domain is run-
ning. The device will only show up in the guest on next boot.
--live
This option will only affect a running domain. If the domain is inactive,
the operation will fail. The device is not persisted in the XML and will not
be available in the guest on next boot.
--current
This option affects the current state of the domain. If the domain is inac-
tive, the device is added to the persistent XML and will be available on
next boot. If the domain is active, the device is hotplugged but not
added to the persistent XML.
To detach a VF interface, use the virsh detach-device command, which

also takes the options listed above.
14.11.4 Dynamic Allocation of VFs from a Pool

If you define the PCI address of a VF into a guest's configuration statically as de-
scribed in Section 14.11.3, “Adding a VF Network Device to an Existing VM Guest”, it
is hard to migrate such guest to another host. The host must have identical hard-
ware in the same location on the PCI bus, or the guest configuration must be modi-
fied prior to each start.
Another approach is to create a libvirt network with a device pool that contains
all the VFs of an SR-IOV device. The guest then references this network, and each
time it is started, a single VF is dynamically allocated to it. When the guest is stopped,
the VF is returned to the pool, available for another guest.
14.11.4.1 Defining Network with Pool of VFs on VM Host Server

The following example of network definition creates a pool of all VFs for the SR-IOV
device with its physical function (PF) at the network interface eth0 on the host:
157 z 366 2017-05-27 13:21

<network>
<name>passthrough</name>
<forward mode='hostdev' managed='yes'>
<pf dev='eth0'/>
</forward>
</network>
To use this network on the host, save the above code to a file, for example
/tmp/passthrough.xml , and execute the following commands. Remember to re-
place eth0 with the real network interface name of your SR-IOV device's PF:
virsh net-define /tmp/passthrough.xml

virsh net-autostart passthrough
virsh net-start passthrough
14.11.4.2 Configuring VM Guest to Use VF from the Pool

The following example of guest device interface definition uses a VF of the SR-IOV
device from the pool created in Section 14.11.4.1, “Defining Network with Pool of
VFs on VM Host Server”. libvirt automatically derives the list of all VFs associated
with that PF the first time the guest is started.
<interface type='network'>
<source network='passthrough'>
</interface>
To verify the list of associated VFs, run virsh net-dumpxml passthrough on the
host after the first guest that uses the network with the pool of VFs starts.
<network connections='1'>
<name>passthrough</name>
<uuid>a6a26429-d483-d4ed-3465-4436ac786437</uuid>
<forward mode='hostdev' managed='yes'>
<pf dev='eth0'/>
<address type='pci' domain='0x0000' bus='0x02' slot='0x10' function='0x1'/>
</forward>
</network>
158 z 366 2017-05-27 13:21

Part III Hypervisor-Independent

Features
15 Disk Cache Modes

Keeping the correct time in a VM Guest is one of the more difficult aspects of vir-
tualization. Keeping the correct time is especially important for network applica-
tions and is also a prerequisite to do a live migration of a VM Guest.
17 libguestfs
Virtual Machines consist of disk images and definition files. Manually accessing
and manipulating these guest components (outside of normal hypervisor pro-
cesses) is possible, but inherently dangerous and risks compromising data in-
tegrity. libguestfs is a C library and corresponding set of tools designed for safely
accessing and modifying Virtual Machine disk images - outside of normal hyper-
visor processes, but without the risk normally associated with manual editing.
15 Disk Cache Modes

15.3 Data Integrity Implications of Cache Modes
159 z 366 2017-05-27 13:21


Hypervisors allow for various storage caching strategies to be specified when config-
uring a VM Guest. Each guest disk interface can have one of the following cache
modes specified: writethrough, writeback, none, directsync, or unsafe. If no cache
mode is specified, an appropriate default cache mode is used. These cache modes
influence how host-based storage is accessed, as follows:
Read/write data may be cached in the host page cache.
The guest's storage controller is informed whether a write cache is present,

allowing for the use of a flush command.
Synchronous write mode may be used, in which write requests are reported
complete only when committed to the storage device.
Flush commands (generated by the guest storage controller) may be ignored

for performance reasons.
If a disorderly disconnection between the guest and its storage occurs, the cache
mode in use will affect whether data loss occurs. The cache mode can also affect
disk performance significantly. Additionally, some cache modes are incompatible
with live migration, depending on several factors. There are no simple rules about
what combination of cache mode, disk image format, image placement, or storage
sub-system is best. The user should plan each guest's configuration carefully and ex-
periment with various configurations to determine the optimal performance.

cache mode unspecified
In older QEMU versions, not specifying a cache mode meant that writethrough
would be used as the default. With modern versions—as shipped with SUSE
Linux Enterprise Server— the various guest storage interfaces have been fixed
to handle writeback or writethrough semantics more correctly, allowing for the
default caching mode to be switched to writeback. The guest driver for each of
ide , scsi , and virtio have within their power to disable the write back
cache, causing the caching mode used to revert to writethrough. The typical
guest's storage drivers will maintain the default caching mode as writeback, how-
ever.
160 z 366 2017-05-27 13:21

writethrough
This mode causes the hypervisor to interact with the disk image file or block de-
vice with O_DSYNC semantics, where writes are reported as completed only
when the data has been committed to the storage device. The host page cache is
used in what can be termed a writethrough caching mode. The guest's virtual
storage adapter is informed that there is no writeback cache, so the guest would
not need to send down flush commands to manage data integrity. The storage
behaves as if there is a writethrough cache.
writeback
vice with neither O_DSYNC nor O_DIRECT semantics, so the host page cache is
used and writes are reported to the guest as completed when they are placed in
the host page cache. The normal page cache management will handle commit-
ment to the storage device. Additionally, the guest's virtual storage adapter is in-
formed of the writeback cache, so the guest would be expected to send down
flush commands as needed to manage data integrity. Analogous to a raid con-
troller with RAM cache.
none
vice with O_DIRECT semantics, so the host page cache is bypassed and I/O hap-
pens directly between the hypervisor user space buffers and the storage device.
Because the actual storage device may report a write as completed when placed
in its write queue only, the guest's virtual storage adapter is informed that there
is a writeback cache. The guest would be expected to send down flush com-
mands as needed to manage data integrity. Performance-wise, it is equivalent to
direct access to your host's disk.
unsafe
This mode is similar to the writeback mode discussed above. The key aspect
of this “unsafe” mode, is that all flush commands from the guests are ignored.
Using this mode implies that the user has accepted the trade-off of performance
over risk of data loss in case of a host failure. Useful, for example, during guest
installation, but not for production workloads.
directsync
161 z 366 2017-05-27 13:21

vice with both O_DSYNC and O_DIRECT semantics. This means, writes are re-
ported as completed only when the data has been committed to the storage de-
vice, and when it is also desirable to bypass the host page cache. Like
writethrough, it is helpful to guests that do not send flushes when needed. It
was the last cache mode added, completing the possible combinations of
caching and direct access semantics.
15.3 Data Integrity Implications of Cache

Modes
writethrough, none, directsync
These are the safest modes, and considered equally safe, given that the guest
operating system is “modern and well behaved”, which means that it uses
flushes as needed. If you have a suspect guest, use writethough, or directsync.
Note that some file systems are not compatible with none or directsync , as
they do not support O_DIRECT, which these cache modes rely on.
writeback
This mode informs the guest of the presence of a write cache, and relies on the
guest to send flush commands as needed to maintain data integrity within its
disk image. This is a common storage design which is completely accounted for
within modern file systems. But it should be noted that because there is a win-
dow of time between the time a write is reported as completed, and that write
being committed to the storage device, this mode exposes the guest to data loss
in the unlikely case of a host failure.
unsafe
This mode is similar to writeback caching except the guest flush commands are
ignored, nullifying the data integrity control of these flush commands, and re-
sulting in a higher risk of data loss because of host failure. The name “unsafe”
should serve as a warning that there is a much higher potential for data loss be-
cause of a host failure than with the other modes. Note that as the guest termi-
nates, the cached data is flushed at that time.
162 z 366 2017-05-27 13:21

The choice to make full use of the page cache, or to write through it, or to bypass it
altogether can have dramatic performance implications. Other factors that influence
disk performance include the capabilities of the actual storage system, what disk im-
age format is used, the potential size of the page cache and the IO scheduler used.
Additionally, not flushing the write cache increases performance, but with risk, as
noted above. As a general rule, high-end systems typically perform best with the
cache mode none , because of the reduced data copying that occurs. The potential
benefit of having multiple guests share the common host page cache, the ratio of
reads to writes, and the use of AIO mode native (see below) should also be con-
sidered.

The caching of storage data and metadata restricts the configurations that support
live migration. Currently, only raw , qcow2 and qed image formats can be used for
live migration. If a clustered file system is used, all cache modes support live migra-
tion. Otherwise the only cache mode that supports live migration on read/write
shared storage is none .
The libvirt management layer includes checks for migration compatibility based
on several factors. If the guest storage is hosted on a clustered file system, is
read-only or is marked shareable, then the cache mode is ignored when determin-
ing if migration can be allowed. Otherwise libvirt will not allow migration unless
the cache mode is set to none . However, this restriction can be overridden with the
“unsafe” option to the migration APIs, which is also supported by virsh , as for ex-
ample in
virsh migrate --live --unsafe
Tip
The cache mode none is required for the AIO mode setting native . If another
cache mode is used, then the AIO mode will silently be switched back to the de-
fault threads . The guest flush within the host is implemented using
fdatasync() .
163 z 366 2017-05-27 13:21


Abstract
Keeping the correct time in a VM Guest is one of the more difficult as-
pects of virtualization. Keeping the correct time is especially important for
network applications and is also a prerequisite to do a live migration of a
VM Guest.

Tip: Timekeeping on the VM Host Server
It is strongly recommended to ensure the VM Host Server keeps the correct time
as well, for example, by using NTP (see Book “Administration Guide”, Chapter 22
“Time Synchronization with NTP” for more information).

KVM provides a paravirtualized clock which is currently supported by SUSE Linux En-
terprise Server 10 SP3 and newer and RedHat Enterprise Linux 5.4 and newer via
the kvm_clock driver. It is strongly recommended to use kvm_clock when avail-
able.
Use the following command inside a VM Guest running Linux to check whether the
driver kvm_clock has been loaded:
tux > sudo dmesg | grep kvm-clock

[ 0.000000] kvm-clock: cpu 0, msr 0:7d3a81, boot clock
[ 0.000000] kvm-clock: cpu 0, msr 0:1206a81, primary cpu clock
164 z 366 2017-05-27 13:21

[ 0.012000] kvm-clock: cpu 1, msr 0:1306a81, secondary cpu clock

[ 0.160082] Switching to clocksource kvm-clock
To check which clock source is currently used, run the following command in the VM
Guest. It should output kvm-clock :
cat /sys/devices/system/clocksource/clocksource0/current_clocksource
Important: kvm-clock and NTP
When using kvm-clock , it is recommended to use NTP in the VM Guest, as well.

Using NTP on the VM Host Server is also recommended.
16.1.1 Other Timekeeping Methods

The paravirtualized kvm-clock is currently not for Windows* operating systems.
For Windows*, use the Windows Time Service Tools for time synchronization
(see http://technet.microsoft.com/en-us/library/cc773263%28WS.10%29.aspx↗ for
more information).

When booting, virtual machines get their initial clock time from their host. After get-
ting their initial clock time, fully virtual machines manage their time independently
from the host. Paravirtual machines manage clock time according to their indepen-
dent wallclock setting. If the independent wallclock is enabled, the virtual machine
manages its time independently and does not synchronize with the host. If the inde-
pendent wallclock is disabled, the virtual machine periodically synchronizes its time
with the host clock.
Note
OES 2 NetWare virtual machines manage clock time independently after booting.
They do not synchronize with the host clock time.
165 z 366 2017-05-27 13:21

If a guest operating system is configured for NTP and the virtual machine's indepen-
dent wallclock setting is disabled, it will still periodically synchronize its time with the
host time. This dual type of configuration can result in time drift between virtual ma-
chines that need to be synchronized. To effectively use an external time source, such
as NTP, for time synchronization on a virtual machine, the virtual machine's indepen-
dent wallclock setting must be enabled (set to 1 ). Otherwise, it will continue to syn-
chronize its time with its host.
PROCEDURE 16.1: VIEWING THE INDEPENDENT WALLCLOCK SETTING
1. Log in to the virtual machine’s operating system as root .
2. In the virtual machine environment, enter
cat /proc/sys/xen/independent_wallclock
0 means that the virtual machine is getting its time from the host
and is not using independent wallclock.
1 means that the virtual machine is using independent wallclock

and managing its time independently from the host.
PROCEDURE 16.2: PERMANENTLY CHANGING THE INDEPENDENT WALLCLOCK SETTING
1. Log in to the virtual machine environment as root .
2. Edit the virtual machine’s /etc/sysctl.conf file.
3. Add or change the following entry:
xen.independent_wallclock=1
Enter 1 to enable or 0 to disable the wallclock setting.
4. Save the file and reboot the virtual machine operating system.
While booting, a virtual machine gets its initial clock time from the host.
Then, if the wallclock setting is set to 1 in the sysctl.conf file, it manages
its clock time independently and does not synchronize with the host clock
time.
PROCEDURE 16.3: TEMPORARILY CHANGING THE INDEPENDENT WALLCLOCK SETTING
1. Log in to the virtual machine environment as root .
2. Enter the following command:
166 z 366 2017-05-27 13:21

echo "1" > /proc/sys/xen/independent_wallclock
3. Add or change the following entry:
xen.independent_wallclock=1
Although the current status of the independent wallclock changes immedi-

ately, its clock time might not be immediately synchronized. The setting per-
sists until the virtual machine reboots. Then, it gets its initial clock time from
the host and uses the independent wallclock according to the setting speci-
fied in the sysctl.conf file.
17 libguestfs
Abstract
Virtual Machines consist of disk images and definition files. Manually ac-
cessing and manipulating these guest components (outside of normal hy-
pervisor processes) is possible, but inherently dangerous and risks com-
promising data integrity. libguestfs is a C library and corresponding set of
tools designed for safely accessing and modifying Virtual Machine disk
images - outside of normal hypervisor processes, but without the risk
normally associated with manual editing.

17.3 Guestfs Tools
167 z 366 2017-05-27 13:21

17.1.1 VM Guest Manipulation Risk

As disk images and definition files are simply another type of file in a Linux environ-
ment, it is possible to use many tools to access, edit and write to these files. When
used correctly, such tools can be an important part of guest administration. How-
ever, even correct usage of these tools is not without risk. Risks that should be con-
sidered when manually manipulating guest disk images include:
Data Corruption: Concurrently accessing images, by the host machine or an-

other node in a cluster, can cause changes to be lost or data corruption to
occur if virtualization protection layers are bypassed.
Security: Mounting disk images as loop devices requires root access. While an
image is loop mounted, other users and processes can potentially access the
disk contents.
Administrator Error: Bypassing virtualization layers correctly requires ad-

vanced understanding of virtual components and tools. Failing to isolate the
images or failing to clean up properly after changes have been made can
lead to further problems once back in virtualization control.
17.1.2 libguestfs Design

libguestfs C library has been designed to safely and securely create, access and mod-
ify virtual machine (VM Guest) disk images. It also provides additional language bind-
ings: for Perl↗, Python↗, PHP (only for 64-bit machines), and Ruby↗. libguestfs can
access VM Guest disk images without needing root and with multiple layers of de-
fense against rogue disk images.
libguestfs provides many tools designed for accessing and modifying VM Guest disk
images and contents. These tools provide such capabilities as: viewing and editing
files inside guests, scripting changes to VM Guests, monitoring disk used/free statis-
tics, creating guests, doing V2V or P2V migrations, performing backups, cloning VM
Guests, formatting disks, and resizing disks.
168 z 366 2017-05-27 13:21

Warning: Best Practices
You must not use libguestfs tools on live virtual machines. Doing so will probably
result in disk corruption in the VM Guest. libguestfs tools try to stop you from do-
ing this, but cannot catch all cases.
However most command have the --ro (read-only) option. With this option,
you can attach a command to a live virtual machine. The results might be strange
or inconsistent at times but you will not risk disk corruption.

libguestfs is shipped through 4 packages:
libguestfs0 : which provides the main C library
guestfs-data : which contains the appliance files used when launching im-
ages (stored in /usr/lib64/guestfs )
guestfs-tools : the core guestfs tools, man pages, and the

/etc/libguestfs-tools.conf configuration file.
guestfs-winsupport : provides support for Windows file guests in the

guestfs tools. This package only needs to be installed to handle Windows
guests, for example when converting a Windows guest to KVM.
To install guestfs tools on your system run:
zypper in guestfs-tools
17.3 Guestfs Tools
17.3.1 Modifying Virtual Machines
169 z 366 2017-05-27 13:21

The set of tools found within the guestfs-tools package is used for accessing and
modifying virtual machine disk images. This functionality is provided through a fa-
miliar shell interface with built-in safeguards which ensure image integrity. Guestfs
tools shells expose all capabilities of the guestfs API, and create an appliance on the
fly using the packages installed on the machine and the files found in
/usr/lib4/guestfs .
17.3.2 Supported File Systems and Disk Images

Guestfs tools support various file systems including:
Ext2, Ext3, Ext4
Xfs
Btrfs
Multiple disk image formats are also supported:
raw
qcow2
Warning: Unsupported File System
Guestfs may also support Windows* file systems (VFAT, NTFS), BSD* and Apple*
file systems, and other disk image formats (VMDK, VHDX...). However, these file
systems and disk image formats are unsupported on SUSE Linux Enterprise.
17.3.3 virt-rescue
virt-rescue is similar to a rescue CD, but for virtual machines, and without the
need for a CD. virt-rescue presents users with a rescue shell and some simple recov-
ery tools which can be used to examine and correct problems within a virtual ma-
chine or disk image.
tux > virt-rescue -a sles.qcow2

Welcome to virt-rescue, the libguestfs rescue shell.
Note: The contents of / are the rescue appliance.

You need to mount the guest's partitions under /sysroot
before you can examine them. A helper script for that exists:
mount-rootfs-and-do-chroot.sh /dev/sda2
170 z 366 2017-05-27 13:21

><rescue>
[ 67.194384] EXT4-fs (sda1): mounting ext3 file system
using the ext4 subsystem
[ 67.199292] EXT4-fs (sda1): mounted filesystem with ordered data
mode. Opts: (null)
mount: /dev/sda1 mounted on /sysroot.
mount: /dev bound on /sysroot/dev.
mount: /dev/pts bound on /sysroot/dev/pts.
mount: /proc bound on /sysroot/proc.
mount: /sys bound on /sysroot/sys.
Directory: /root
Thu Jun 5 13:20:51 UTC 2014
(none):~ #
You are now running the VM Guest in rescue mode:
(none):~ # cat /etc/fstab

devpts /dev/pts devpts mode=0620,gid=5 0 0
proc /proc proc defaults 0 0
sysfs /sys sysfs noauto 0 0
debugfs /sys/kernel/debug debugfs noauto 0 0
usbfs /proc/bus/usb usbfs noauto 0 0
tmpfs /run tmpfs noauto 0 0
/dev/disk/by-id/ata-QEMU_HARDDISK_QM00001-part1 / ext3 defaults 1 1
17.3.4 virt-resize
virt-resize is used to resize a virtual machine disk, making it larger or smaller
overall, and resizing or deleting any partitions contained within.
PROCEDURE 17.1: EXPANDING A DISK
Full step-by-step example: How to expand a virtual machine disk
1. First, with virtual machine powered off, determine the size of the partitions
available on this virtual machine:
tux > virt-filesystems --long --parts --blkdevs -h -a sles.qcow2

Name Type MBR Size Parent
/dev/sda1 partition 83 16G /dev/sda
/dev/sda device - 16G -
2. virt-resize cannot do in-place disk modifications—there must be suffi-

cient space to store the resized output disk. Use the truncate command
to create a file of suitable size:
tux > truncate -s 32G outdisk.img
3. Use virt-resize to resize the disk image. virt-resize requires two

mandatory parameters for the input and output images:
171 z 366 2017-05-27 13:21

tux > virt-resize --expand /dev/sda1 sles.qcow2 outdisk.img

Examining sles.qcow2 ...
**********
Summary of changes:
/dev/sda1: This partition will be resized from 16,0G to 32,0G. The

filesystem ext3 on /dev/sda1 will be expanded using the 'resize2fs'
method.
**********
Setting up initial partition table on outdisk.img ...
Copying /dev/sda1 ...
◐ 84%
⟦▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒════════
Expanding /dev/sda1 using the 'resize2fs' method ...
Resize operation completed with no errors. Before deleting the old

disk, carefully check that the resized disk boots and works correctly.
4. Confirm the image was resized properly:
tux > virt-filesystems --long --parts --blkdevs -h -a outdisk.img

Name Type MBR Size Parent
/dev/sda1 partition 83 32G /dev/sda
/dev/sda device - 32G -
5. Bring up the VM Guest using the new disk image and confirm correct opera-
tion before deleting the old image.
17.3.5 Other virt-* Tools

There are guestfs tools to simplify administrative tasks—such as viewing and editing
files, or obtaining information on the virtual machine.
17.3.5.1 virt-filesystems
This tool is used to report information regarding file systems, partitions, and logical
volumes in a disk image or virtual machine.
tux > virt-filesystems -l -a sles.qcow2

Name Type VFS Label Size Parent
/dev/sda1 filesystem ext3 - 17178820608 -
17.3.5.2 virt-ls
172 z 366 2017-05-27 13:21

virt-ls lists file names, file sizes, checksums, extended attributes and more from
a virtual machine or disk image. Multiple directory names can be given, in which
case the output from each is concatenated. To list directories from a libvirt guest, use
the -d option to specify the name of the guest. For a disk image, use the -a op-
tion.
tux > virt-ls -h -lR -a sles.qcow2 /var/log/

d 0755 4,0K /var/log/
d 0700 4,0K /var/log//YaST2
- 0644 1,9K /var/log//YaST2/mkinitrd.log
- 0644 496 /var/log//YaST2/perl-BL-standalone-log
- 0600 3,2K /var/log//faillog
d 0700 4,0K /var/log//krb5
- 0644 29K /var/log//lastlog
- 0644 496 /var/log//pbl.log
- 0664 0 /var/log//wtmp
d 0755 4,0K /var/log//zypp
17.3.5.3 virt-cat
virt-cat is a command line tool to display the contents of a file that exists in the
named virtual machine (or disk image). Multiple file names can be given, in which
case they are concatenated together. Each file name must be a full path, starting at
the root directory (starting with '/').
tux > virt-cat -a sles.qcow2 /etc/fstab

17.3.5.4 virt-df
virt-df is a command line tool to display free space on virtual machine file sys-
tems. Unlike other tools, it does not just display the size of disk allocated to a virtual
machine, but can look inside disk images to show how much space is actually being
used.
tux > virt-df -a sles.qcow2

Filesystem 1K-blocks Used Available Use%
sles.qcow2:/dev/sda1 16381864 520564 15022492 4%
17.3.5.5 virt-edit
virt-edit is a command line tool capable of editing files that reside in the named
virtual machine (or disk image).
173 z 366 2017-05-27 13:21

17.3.5.6 virt-tar-in/out
virt-tar-in unpacks an uncompressed TAR archive into a virtual machine disk
image or named libvirt domain. virt-tar-out packs a virtual machine disk image
directory into a TAR archive.
tux > virt-tar-out -a sles.qcow2 /home homes.tar
17.3.5.7 virt-copy-in/out
virt-copy-in copies files and directories from the local disk into a virtual ma-
chine disk image or named libvirt domain. virt-copy-out copies files and directo-
ries out of a virtual machine disk image or named libvirt domain.
tux > virt-copy-in -a sles.qcow2 data.tar /tmp/

virt-ls -a sles.qcow2 /tmp/
.ICE-unix
.X11-unix
data.tar
17.3.5.8 virt-log
virt-log shows the log files of the named libvirt domainvirtual machine or disk
image. If the package guestfs-winsupport is installed it can also show the events
log of a Windows virtual machine disk image.
tux > virt-log -a windows8.qcow2

<?xml version="1.0" encoding="utf-8" standalone="yes" ?>
<Events>
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"><System><Provider N
<EventID Qualifiers="32768">6011</EventID>
<Level>4</Level>
<Task>0</Task>
<Keywords>0x0080000000000000</Keywords>
<TimeCreated SystemTime="2014-09-12 05:47:21"></TimeCreated>
<EventRecordID>1</EventRecordID>
<Channel>System</Channel>
<Computer>windows-uj49s6b</Computer>
<Security UserID=""></Security>
</System>
<EventData><Data><string>WINDOWS-UJ49S6B</string>
<string>WIN-KG190623QG4</string>
</Data>
<Binary></Binary>
</EventData>
</Event>
...
174 z 366 2017-05-27 13:21

17.3.6 guestfish
guestfish is a shell and command line tool for examining and modifying virtual
machine file systems. It uses libguestfs and exposes all of the functionality of the
guestfs API.
Examples of usage:
tux > guestfish -a disk.img <<EOF

run
list-filesystems
EOF
guestfish
Welcome to guestfish, the guest filesystem shell for

editing virtual machine filesystems and disk images.
Type: 'help' for help on commands

'man' to read the manual
'quit' to quit the shell
><fs> add sles.qcow2

><fs> run
><fs> list-filesystems
/dev/sda1: ext3
><fs> mount /dev/sda1 /
cat /etc/fstab
sysfs /sys sysfs noauto 0 0
debugfs /sys/kernel/debug debugfs noauto 0 0
usbfs /proc/bus/usb usbfs noauto 0 0
tmpfs /run tmpfs noauto 0 0
/dev/disk/by-id/ata-QEMU_HARDDISK_QM00001-part1 / ext3 defaults 1 1
17.3.7 Converting a Physical Machine into a KVM Guest

Libguestfs provides tools to help converting Xen virtual machines or physical ma-
chines into KVM guests. The Xen to KVM conversion scenario is covered by the Article
“Xen to KVM Migration Guide”. The following section will cover a special use case: con-
verting a bare metal machine into a KVM one.
Converting a physical machine into a KVMone is not yet supported in SUSE Linux
Enterprise Server. This feature is released as a technology preview only.
Converting a physical machine requires collecting information about it and transmit-

ting this to a conversion server. This is achieved by running a live system prepared
with virt-p2v and kiwi tools on the machine.
175 z 366 2017-05-27 13:21

PROCEDURE 17.2: USING VIRT-P2V
1. Install the needed packages with the command:
root # zypper in virt-p2v kiwi-desc-isoboot
Note
These steps will document how to create an ISO image to create a

bootable DVD. Alternatively, you can create a PXE boot image instead; for
more information about building PXE images with kiwi, see
man virt-p2v-make-kiwi .
2. Create a kiwi configuration:
tux > virt-p2v-make-kiwi -o /tmp/p2v.kiwi
The -o defines where to create the kiwi configuration.
3. Edit the config.xml file in the generated configuration if needed. For ex-
ample, in config.xml adjust the keyboard layout of the live system.
4. Build the ISO image with kiwi :
tux > kiwi --build /tmp/p2v.kiwi 1 \

-d /tmp/build 2 \
--ignore-repos \
--add-repo http://url/to/SLE/repositories 3 \
--type iso
1 The directory where the kiwi configuration was generated in the previ-
ous step.
2 The directory where kiwi will place the generated ISO image and other
intermediary build results.
3 The URLs to the package repositories as found with zypper lr -d .
Use one --add-repo parameter per repository.
5. Burn the ISO on a DVD or a USB stick. With such a medium, boot the ma-
chine to be converted.
6. Once the system is started, you will be asked for the connection details of
the conversion server. This server is a machine with the virt-v2v package
installed.
If the network setup is more complex than a DHCP client, click the Configure
network button to open the YaST network configuration dialog.
176 z 366 2017-05-27 13:21

Click the Test connection button to allow moving to the next page of the wiz-
ard.
7. Select the disks and network interfaces to be converted and define the VM
data like the amount of allocated CPUs, memory and the Virtual Machine
name.
Note
If not defined, the created disk image format will be raw by default. This
can be changed by entering the desired format in the Output format field.
There are two possibilities to generate the virtual machine: either using the
local or the libvirt output. The first one will place the Virtual Machine disk im-
age and configuration in the path defined in the Output storage field. These
can then be used to define a new libvirt-handled guest using virsh . The
second method will create a new libvirt-handled guest with the disk image
placed in the pool defined in the Output storage field.
Click the Start conversion to start it.
17.4.1 Btrfs-related Problems

When using the guestfs tools on an image with Btrfs root partition (the default with
SUSE Linux Enterprise Server 12) the following error message could be provided:
tux > virt-ls -a /path/to/sles12sp2.qcow2 /

virt-ls: multi-boot operating systems are not supported
If using guestfish '-i' option, remove this option and instead

use the commands 'run' followed by 'list-filesystems'.
You can then mount filesystems you want by hand using the
'mount' or 'mount-ro' command.
If using guestmount '-i', remove this option and choose the

filesystem(s) you want to see by manually adding '-m' option(s).
Use 'virt-filesystems' to see what filesystems are available.
If using other virt tools, multi-boot operating systems won't work

with these tools. Use the guestfish equivalent commands
(see the virt tool manual page).
177 z 366 2017-05-27 13:21

This is usually caused by the presence of snapshots in the guests. In this case guestfs
does not know what snapshot to bootstrap. To force the use of a snapshot, use the
-m parameter as following:
tux > virt-ls -m /dev/sda2:/:subvol=@/.snapshots/2/snapshot -a /path/to/sles12sp2.qcow2
17.4.2 Environment
When troubleshooting problems within a libguestfs appliance, the environment vari-
able LIBGUESTFS_DEBUG=1 can be used to enable debug messages. To output each
command/API call in a format that is similar to guestfish commands, use the envi-
ronment variable LIBGUESTFS_TRACE=1.
17.4.3 libguestfs-test-tool
libguestfs-test-tool is a test program that checks if basic libguestfs functional-
ity is working. It will print a large amount of diagnostic messages and details of the
guestfs environment, then create a test image and try to start it. If it runs to comple-
tion successfully, the following message should be seen near the end:
===== TEST FINISHED OK =====

libguestfs.org↗
libguestfs FAQ↗
Part IV Managing Virtual Machines

with Xen
178 z 366 2017-05-27 13:21

This section documents how to set up and use SUSE Linux Enterprise Server 12
SP2 as a virtual machine host.
A VM Guest system needs some means to communicate either with other VM
Guest systems or with a local network. The network interface to the VM Guest
system is made of a split device driver, which means that any virtual Ethernet de-
vice has a corresponding network interface in Dom0. This interface is s…
20 Managing a Virtualization Environment

Apart from using the recommended libvirt library (Part II, “Managing Virtual
Machines with libvirt ”), you can manage Xen guest domains with the xl
tool from the command line.
22 Virtualization: Configuration Options and Settings

The documentation in this section, describes advanced management tasks and
configuration options that might help technology innovators implement lead-
ing-edge virtualization solutions. It is provided as a courtesy and does not imply
that all documented options and tasks are supported by Novell, Inc.
24 XenStore: Configuration Database Shared between Domains

This section introduces basic information about XenStore, its role in the Xen en-
vironment, the directory structure of files used by XenStore, and the description
of XenStore's commands.
25 Xen as a High-Availability Virtualization Host

Setting up two Xen hosts as a failover system has several advantages compared
to a setup where every server runs on dedicated hardware.

179 z 366 2017-05-27 13:21


This section documents how to set up and use SUSE Linux Enterprise Server 12 SP2
as a virtual machine host.
Usually, the hardware requirements for the Dom0 are the same as those for the
SUSE Linux Enterprise Server operating system, but additional CPU, disk, memory,
and network resources should be added to accommodate the resource demands of
all planned VM Guest systems.
Tip: Resources
Remember that VM Guest systems, like physical machines, perform better when
they run on faster processors and have access to more system memory.
The virtual machine host requires several software packages and their dependencies
to be installed. To install all necessary packages, run YaST Software Management, se-
lect View › Patterns and choose Xen Virtual Machine Host Server for installation. The in-
stallation can also be performed with YaST using the module Virtualization › Install
Hypervisor and Tools.
After the Xen software is installed, restart the computer and, on the boot screen,
choose the newly added option with the Xen kernel.
Updates are available through your update channel. To be sure to have the latest
updates installed, run YaST Online Update after the installation has finished.

When installing and configuring the SUSE Linux Enterprise operating system on the
host, be aware of the following best practices and suggestions:
If the host should always run as Xen host, run YaST System › Boot Loader and
activate the Xen boot entry as default boot section.
In YaST, click System > Boot Loader.
180 z 366 2017-05-27 13:21

Change the default boot to the Xen label, then click Set as Default.
Click Finish.
For best performance, only the applications and processes required for vir-
tualization should be installed on the virtual machine host.
When using both iSCSI and OCFS2 to host Xen images, the latency required
for OCFS2 default timeouts in SUSE Linux Enterprise Server 12 may not be
met. To reconfigure this timeout, run systemctl configure o2cb or edit
O2CB_HEARTBEAT_THRESHOLD in the system configuration.
If you intend to use a watchdog device attached to the Xen host, use only
one at a time. It is recommended to use a driver with actual hardware inte-
gration over a generic software one.
Note: Hardware Monitoring
The Dom0 Kernel is running virtualized, so tools like irqbalance or lscpu will
not reflect the real hardware characteristics.

When the host is set up, a percentage of system memory is reserved for the hypervi-
sor, and all remaining memory is automatically allocated to Dom0.
A better solution is to set a default amount of memory for Dom0, so the memory
can be allocated appropriately to the hypervisor. An adequate amount would be 20
percent of the total system memory up to 4 GiB. A recommended minimum amount
would be 512 MiB
Warning: Minimum amount of Memory
The minimum amount of memory heavily depends on how many VM Guest(s) the
host should handle. So be sure you have enough memory to support all your VM
Guests. If the value is too low, the host system may hang when multiple VM
Guests use most of the memory.
181 z 366 2017-05-27 13:21

18.2.1 Setting a Maximum Amount of Memory

1. Determine the amount of memory to set for Dom0.
2. At Dom0, type xl info to view the amount of memory that is available on

the machine. The memory that is currently allocated by Dom0 can be deter-
mined with the command xl list .
3. Run YaST › Boot Loader.
4. Select the Xen section.
5. In Additional Xen Hypervisor Parameters, add dom0_mem= mem_amount

where mem_amount is the maximum amount of memory to allocate to
Dom0. Add K , M , or G , to specify the size, for example, dom0_mem=768M .
6. Restart the computer to apply the changes.
Warning: Xen Dom0 Memory
When using the XL tool stack and the dom0_mem= option for the Xen hypervisor
in GRUB 2 you need to disable xl autoballoon in etc/xen/xl.conf , otherwise
launching VMs will fail with errors about not being able to balloon down Dom0.
So add autoballoon=0 to xl.conf if you have the dom0_mem= option specified
for Xen. Also see Xen dom0 memory↗

In a fully virtualized guest, the default network card is an emulated Realtek network
card. However, it also possible to use the split network driver to run the communica-
tion between Dom0 and a VM Guest. By default, both interfaces are presented to
the VM Guest, because the drivers of some operating systems require both to be
present.
When using SUSE Linux Enterprise, only the paravirtualized network cards are avail-
able for the VM Guest by default. The following network options are available:
emulated
182 z 366 2017-05-27 13:21

To use an emulated network interface like an emulated Realtek card, specify

type=ioemu in the vif device section of the domain xl configuration. An ex-
ample configuration would look like:
vif = [ 'type=ioemu,mac=00:16:3e:5f:48:e4,bridge=br0' ]
Find more details about the xl configuration in the xl.conf manual page
man 5 xl.conf .
paravirtualized
When you specify type=vif and do not specify a model or type, the paravirtu-
alized network interface is used:
vif = [ 'type=vif,mac=00:16:3e:5f:48:e4,bridge=br0,backen=0' ]
emulated and paravirtualized

If the administrator should be offered both options, simply specify both type
and model. The xl configuration would look like:
vif = [ 'type=ioemu,mac=00:16:3e:5f:48:e4,model=rtl8139,bridge=br0' ]
In this case, one of the network interfaces should be disabled on the VM Guest.

If virtualization software is correctly installed, the computer boots to display the
GRUB 2 boot loader with a Xen option on the menu. Select this option to start the
virtual machine host.
Note: Xen and Kdump
In Xen, the hypervisor manages the memory resource. If you need to reserve sys-
tem memory for a recovery kernel in Dom0, this memory need to be reserved by
the hypervisor. Thus, it is necessary to add the parameter crashkernel=size
to the kernel line instead of using the line with the other boot options.
For more information on the crashkernel parameter, see Book “System Analysis
and Tuning Guide”, Chapter 17 “Kexec and Kdump”, Section 17.4 “Calculating
crashkernel Allocation Size”.
183 z 366 2017-05-27 13:21

If the Xen option is not on the GRUB 2 menu, review the steps for installation and
verify that the GRUB 2 boot loader has been updated. If the installation has been
done without selecting the Xen pattern, run the YaST Software Management, select
the filter Patterns and choose Xen Virtual Machine Host Server for installation.
After booting the hypervisor, the Dom0 virtual machine starts and displays its
graphical desktop environment. If you did not install a graphical desktop, the com-
mand line environment appears.
Tip: Graphics Problems
Sometimes it may happen that the graphics system does not work properly. In
this case, add vga=ask to the boot parameters. To activate permanent settings,
use vga=mode-0x??? where ??? is calculated as 0x100 + VESA mode from
http://en.wikipedia.org/wiki/VESA_BIOS_Extensions↗, for example
vga=mode-0x361 .
Before starting to install virtual guests, make sure that the system time is correct. To
do this, configure NTP (Network Time Protocol) on the controlling domain:
1. In YaST select Network Services › NTP Configuration.
2. Select the option to automatically start the NTP daemon during boot. Pro-
vide the IP address of an existing NTP time server, then click Finish.
Note: Time Services on Virtual Guests
Hardware clocks commonly are not very precise. All modern operating systems
try to correct the system time compared to the hardware time by means of an ad-
ditional time source. To get the correct time on all VM Guest systems, also activate
the network time services on each respective guest or make sure that the guest
uses the system time of the host. For more about Independent Wallclocks in
SUSE Linux Enterprise Server see Section 16.2, “Xen Virtual Machine Clock Set-
tings”.
For more information about managing virtual machines, see Chapter 20, Managing a
Virtualization Environment.
184 z 366 2017-05-27 13:21

To take full advantage of VM Guest systems, it is sometimes necessary to assign spe-

cific PCI devices to a dedicated domain. When using fully virtualized guests, this func-
tionality is only available if the chipset of the system supports this feature, and if it is
activated from the BIOS.
This feature is available from both AMD* and Intel*. For AMD machines, the feature
is called IOMMU; in Intel speak, this is VT-d. Note that Intel-VT technology is not suffi-
cient to use this feature for fully virtualized guests. To make sure that your computer
supports this feature, ask your supplier specifically to deliver a system that supports
PCI Pass-Through.
LIMITATIONS
Some graphics drivers use highly optimized ways to access DMA. This is not
supported, and thus using graphics cards may be difficult.
When accessing PCI devices behind a PCIe bridge, all of the PCI devices must
be assigned to a single guest. This limitation does not apply to PCIe devices.
Guests with dedicated PCI devices cannot be migrated live to a different host.
The configuration of PCI Pass-Through is twofold. First, the hypervisor must be in-
formed at boot time that a PCI device should be available for reassigning. Second,
the PCI device must be assigned to the VM Guest.
18.5.1 Configuring the Hypervisor for PCI Pass-Through

1. Select a device to reassign to a VM Guest. To do this, run lspci and read
the device number. For example, if lspci contains the following line:
06:01.0 Ethernet controller: Digital Equipment Corporation DECchip 21142/43 (rev 4
In this case, the PCI number is (06:01.0) .
2. Run YaST › System › Boot Loader.
3. Select the Xen section and press Edit.
4. Add the PCI number to the Optional Kernel Command Line Parameter line:
pciback.hide=(06:01.0)
5. Press OK and exit YaST.
6. Reboot the system.
7. Check if the device is in the list of assignable devices with the command
185 z 366 2017-05-27 13:21

xl pci-assignable-list
18.5.1.1 Dynamic Assignment with xl

To avoid restarting the host system, you can use dynamic assignment with xl to use
PCI Pass-Through.
Begin by making sure that dom0 has the pciback module loaded:
modprobe pciback
Then make a device assignable by using xl pci-assignable-add . For example, to

make the device 06:01.0 available for guests, run the command:
xl pci-assignable-add 06:01.0
18.5.2 Assigning PCI Devices to VM Guest Systems

There are several possibilities to dedicate a PCI device to a VM Guest:
Adding the device while installing:

During installation, add the pci line to the configuration file:
pci=['06:01.0']
Hotplugging PCI devices to VM Guest systems

The command xl can be used to add or remove PCI devices on the fly. To add
the device with number 06:01.0 to a guest with name sles12 use:
xl pci-attach sles12 06:01.0
Adding the PCI device to Xend

To add the device to the guest permanently, add the following snippet to the
guest configuration file:
pci = [ '06:01.0,power_mgmt=1,permissive=1' ]
After assigning the PCI device to the VM Guest, the guest system must care for the
configuration and device drivers for this device.
18.5.3 VGA Pass-Through
186 z 366 2017-05-27 13:21

Xen 4.0 and newer supports VGA graphics adapter pass-through on fully virtualized
VM Guests. The guest can take full control of the graphics adapter with high-perfor-
mance full 3D and video acceleration.
LIMITATIONS
VGA Pass-Through functionality is similar to PCI Pass-Through and as such
also requires IOMMU (or Intel VT-d) support from the mainboard chipset and
BIOS.
Only the primary graphics adapter (the one that is used when you power on
the computer) can be used with VGA Pass-Through.
VGA Pass-Through is supported only for fully virtualized guests. Paravirtual

guests (PV) are not supported.
The graphics card cannot be shared between multiple VM Guests using VGA
Pass-Through — you can dedicate it to one guest only.
To enable VGA Pass-Through, add the following settings to your fully virtualized
guest configuration file:
gfx_passthru=1
pci=['yy:zz.n']
where yy:zz.n is the PCI controller ID of the VGA graphics adapter as found with
lspci -v on Dom0.
18.5.4 Troubleshooting
In some circumstances, problems may occur during the installation of the VM Guest.
This section describes some known problems and their solutions.
During boot, the system hangs

The software I/O translation buffer allocates a large chunk of low memory early
in the bootstrap process. If the requests for memory exceed the size of the buf-
fer it usually results in a hung boot process. To check if this is the case, switch to
console 10 and check the output there for a message similar to
kernel: PCI-DMA: Out of SW-IOMMU space for 32768 bytes at device 000:01:02.0
In this case you need to increase the size of the swiotlb . Add swiotlb=128
on the cmdline of Dom0. Note that the number can be adjusted up or down to
find the optimal size for the machine.
187 z 366 2017-05-27 13:21

Note: swiotlb a PV guest
The swiotlb=force kernel parameter is required for DMA access to work for
PCI devices on a PV guest. For more information about IOMMU and the
swiotlb option see the file boot-options.txt from the package
kernel-source .
18.5.5 For More Information

There are several resources on the Internet that provide interesting information
about PCI Pass-Through:
http://wiki.xensource.com/xenwiki/VTdHowTo↗
http://software.intel.com/en-us/articles/intel-virtualization-technology-
for-directed-io-vt-d-enhancing-intel-platforms-for-efficient-virtualization-
of-io-devices/↗
http://support.amd.com/TechDocs/48882_IOMMU.pdf↗

PVUSB is a new high performance method for USB Pass-Through from dom0 to the
virtualized guests. PVUSB supports both, USB 2.0 and USB 1.1 devices. PVUSB uses a
paravirtualized front- and back-end interface. With PVUSB, there are two ways to
add USB devices to a guest:
via the config file at domain creation time
via hot-plug while the VM is running
PVUSB uses a paravirtualized front- and back-end interface. PVUSB supports USB
1.1 and USB 2.0, and it works for both, PV and HVM guests. In order to use PVUSB,
you need usbfront in your guest OS, and usbback in dom0 or usb backend in qemu.
With SUSE Linux Enterprise Server, usb backend comes with qemu.
Use lsusb to list the USB devices on the system:
188 z 366 2017-05-27 13:21

root # lsusb
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 003: ID 0461:4d15 Primax Electronics, Ltd Dell Optical Mouse
Bus 002 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
To pass through the Dell mouse, for instance, specify either 0461:4d15 or 2.3 (you
must remove leading zeros, otherwise xl would interpret the numbers as octal val-
ues).
As of Xen 4.7, xl PVUSB support and hot-plug support is introduced.
In the configuration file, specify USB controllers and USB host devices with
usbctrl and usbdev . For example, in case of HVM guests:
usbctrl=['type=qubs,version=2,ports=4', 'type=qubs,version=1,ports=4', ]
usbdev=['hostbus=2, hostaddr=1, controller=0,port=1', ]
Note
It is important to specify type=qusb for the controller of HVM guests.
To manage hot-pluggin PVUSB devices, use the usbctrl-attach ,

usbctrl-detach , usb-list , usbdev-attach and usb-detach subcommands.
For example:
Create a USB controller which is version USB 1.1 and has 8 ports:
root # xl usbctrl-attach test_vm version=1 ports=8 type=qusb
Find the first available controller:port in the domain, and attach USB device whose
busnum:devnum is 2:3 to it; you can also specify controller and port :
root # xl usbdev-attach test_vm hostbus=2 hostaddr=3
Show all USB controllers and USB devices in the domain:
root # xl usb-list test_vm

Devid Type BE state usb-ver ports
0 qusb 0 1 1 8
Port 1: Bus 002 Device 003
Port 2:
Port 3:
Port 4:
Port 5:
Port 6:
Port 7:
Port 8:
Detach the USB device under controller 0 port 1:
189 z 366 2017-05-27 13:21

root # xl usbdev-detach test_vm 0 1
Remove the USB controller with the indicated dev_id , and all USB devices under it:
root # xl usbctrl-detach test_vm dev_id
For more information, see https://wiki.xenproject.org/wiki/Xen_USB_Passthrough↗.
A VM Guest system needs some means to communicate either with other VM Guest
systems or with a local network. The network interface to the VM Guest system is
made of a split device driver, which means that any virtual Ethernet device has a cor-
responding network interface in Dom0. This interface is set up to access a virtual
network that is run in Dom0. The bridged virtual network is fully integrated into the
system configuration of SUSE Linux Enterprise Server and can be configured with
YaST.
When installing a Xen VM Host Server, a bridged network configuration will be pro-
posed during normal network configuration. The user can choose to change the
configuration during the installation and customize it to the local needs.
If desired, Xen VM Host Server can be installed after performing a default Physical
Server installation using the Install Hypervisor and Tools module in YaST.
This module will prepare the system for hosting virtual machines, including invoca-
tion of the default bridge networking proposal.
In case the necessary packages for a Xen VM Host Server are installed manually with
rpm or zypper , the remaining system configuration needs to be done by the ad-
ministrator manually or with YaST.
190 z 366 2017-05-27 13:21

The network scripts that are provided by Xen are not used by default in SUSE Linux
Enterprise Server. They are only delivered for reference but disabled. The network
configuration that is used in SUSE Linux Enterprise Server is done by means of the
YaST system configuration similar to the configuration of network interfaces in SUSE
Linux Enterprise Server.
For more general information about managing network bridges, see Section 13.2,
“Bridged Networking”.

The Xen hypervisor can provide different types of network interfaces to the VM
Guest systems. The preferred network device should be a paravirtualized network
interface. This yields the highest transfer rates with the lowest system requirements.
Up to eight network interfaces may be provided for each VM Guest.
Systems that are not aware of paravirtualized hardware may not have this option. To
connect systems to a network that can only run fully virtualized, several emulated
network interfaces are available. The following emulations are at your disposal:
Realtek 8139 (PCI). This is the default emulated network card.
AMD PCnet32 (PCI)
NE2000 (PCI)
NE2000 (ISA)
Intel e100 (PCI)
Intel e1000 and its variants e1000-82540em, e1000-82544gc,

e1000-82545em (PCI)
All these network interfaces are software interfaces. Because every network inter-
face must have a unique MAC address, an address range has been assigned to Xen-
source that can be used by these interfaces.
191 z 366 2017-05-27 13:21

Tip: Virtual Network Interfaces and MAC Addresses
The default configuration of MAC addresses in virtualized environments creates a

random MAC address that looks like 00:16:3E:xx:xx:xx. Normally, the amount of
available MAC addresses should be big enough to get only unique addresses.
However, if you have a very big installation, or to make sure that no problems
arise from random MAC address assignment, you can also manually assign these
addresses.
For debugging or system management purposes, it may be useful to know which

virtual interface in Dom0 is connected to which Ethernet device in a running guest.
This information may be read from the device naming in Dom0. All virtual devices
follow the rule vif<domain number>.<interface_number> .
For example, if you want to know the device name for the third interface (eth2) of
the VM Guest with id 5, the device in Dom0 would be vif5.2 . To obtain a list of all
available interfaces, run the command ip a .
The device naming does not contain any information about which bridge this inter-
face is connected to. However, this information is available in Dom0. To get an over-
view about which interface is connected to which bridge, run the command
brctl show . The output may look like the following:
# brctl show
br0 8000.001cc0309083 no eth0
vif2.1
br1 8000.000476f060cc no eth1
vif2.0
br2 8000.000000000000 no
In this example, there are three configured bridges: br0 , br1 and br2 . Currently,
br0 and br1 each have a real Ethernet device added: eth0 and eth1 , respec-
tively. There is one VM Guest running with the ID 2 that has two Ethernet devices
available. eth0 on the VM Guest is bridged with eth1 on the VM Host Server and
eth1 on the VM Guest is connected to eth0 on the VM Host Server. The third bridge
with the name br2 is not connected to any VM Guest nor any real Ethernet device.
192 z 366 2017-05-27 13:21

Xen can be set up to use host-based routing in the controlling Dom0. Unfortunately,
this is not yet well supported from YaST and requires quite an amount of manual
editing of configuration files. Thus, this is a task that requires an advanced adminis-
trator.
The following configuration will only work when using fixed IP addresses. Using
DHCP is not practicable with this procedure, because the IP address must be known
to both, the VM Guest and the VM Host Server system.
The easiest way to create a routed guest is to change the networking from a bridged
to a routed network. As a requirement to the following procedures, a VM Guest with
a bridged network setup must be installed. For example, the VM Host Server is
named earth with the IP 192.168.1.20, and the VM Guest has the name alice with the
IP 192.168.1.21.
PROCEDURE 19.1: CONFIGURING A ROUTED IPV4 VM GUEST
1. Make sure that alice is shut down. Use xl commands to shut down and
check.
2. Prepare the network configuration on the VM Host Server earth:
a. Create a hotplug interface that will be used to route the traffic. To ac-
complish this, create a file named
/etc/sysconfig/network/ifcfg-alice.0 with the following con-
tent:
NAME="Xen guest alice"

BOOTPROTO="static"
STARTMODE="hotplug"
b. Edit the file /etc/sysconfig/SuSEfirewall2 and add the follow-

ing configurations:
Add alice.0 to the devices in FW_DEV_EXT:
FW_DEV_EXT="br0 alice.0"
Switch on the routing in the firewall:
FW_ROUTE="yes"
Tell the firewall which address should be forwarded:
FW_FORWARD="192.168.1.21/32,0/0"
Finally, restart the firewall with the command:
sudo systemctl restart SuSEfirewall2
193 z 366 2017-05-27 13:21

c. Add a static route to the interface of alice. To accomplish this, add the
following line to the end of /etc/sysconfig/network/routes :
192.168.1.21 - - alice.0
d. To make sure that the switches and routers that the VM Host Server is
connected to know about the routed interface, activate proxy_arp
on earth. Add the following lines to /etc/sysctl.conf :
net.ipv4.conf.default.proxy_arp = 1
net.ipv4.conf.all.proxy_arp = 1
e. Activate all changes with the commands:
sudo systemctl restart systemd-sysctl wicked
3. Proceed with configuring the Xen configuration of the VM Guest by chang-

ing the vif interface configuration for alice as described in Section 20.1,
“XL—Xen Management Tool”. Make the following changes to the text file you
generate during the process:
a. Remove the snippet
bridge=br0
b. And add the following one:
vifname=vifalice.0
or
vifname=vifalice.0=emu
for a fully virtualized domain.
c. Change the script that is used to set up the interface to the following:
script=/etc/xen/scripts/vif-route-ifup
d. Activate the new configuration and start the VM Guest.
4. The remaining configuration tasks must be accomplished from inside the

VM Guest.
a. Open a console to the VM Guest with xl console domain and log

in.
b. Check that the guest IP is set to 192.168.1.21.
c.
194 z 366 2017-05-27 13:21

Provide VM Guest with a host route and a default gateway to the VM

Host Server. Do this by adding the following lines to
/etc/sysconfig/network/routes :
192.168.1.20 - - eth0
default 192.168.1.20 - -
5. Finally, test the network connection from the VM Guest to the world outside
and from the network to your VM Guest.

Creating a masqueraded network setup is quite similar to the routed setup. How-
ever, there is no proxy_arp needed, and some firewall rules are different. To create a
masqueraded network to a guest dolly with the IP address 192.168.100.1 where the
host has its external interface on br0 , proceed as follows. For easier configuration,
only the already installed guest is modified to use a masqueraded network:
PROCEDURE 19.2: CONFIGURING A MASQUERADED IPV4 VM GUEST
1. Shut down the VM Guest system with xl shutdown domain .
2. Prepare the network configuration on the VM Host Server:
a. Create a hotplug interface that will be used to route the traffic. To ac-
complish this, create a file named
/etc/sysconfig/network/ifcfg-dolly.0 with the following con-
tent:
NAME="Xen guest dolly"

BOOTPROTO="static"
STARTMODE="hotplug"
b. Edit the file /etc/sysconfig/SuSEfirewall2 and add the follow-

ing configurations:
Add dolly.0 to the devices in FW_DEV_DMZ:
FW_DEV_DMZ="dolly.0"
Switch on the routing in the firewall:
FW_ROUTE="yes"
Switch on masquerading in the firewall:
195 z 366 2017-05-27 13:21

FW_MASQUERADE="yes"
Tell the firewall which network should be masqueraded:
FW_MASQ_NETS="192.168.100.1/32"
Remove the networks from the masquerading exceptions:
FW_NOMASQ_NETS=""
Finally, restart the firewall with the command:
sudo systemctl restart SuSEfirewall2
c. Add a static route to the interface of dolly. To accomplish this, add the
following line to the end of /etc/sysconfig/network/routes :
192.168.100.1 - - dolly.0
d. Activate all changes with the command:
sudo systemctl restart wicked
3. Proceed with configuring the Xen configuration of the VM Guest.
a. Change the vif interface configuration for dolly as described in Sec-

tion 20.1, “XL—Xen Management Tool”.
b. Remove the entry:
bridge=br0
c. And add the following one:
vifname=vifdolly.0
d. Change the script that is used to set up the interface to the following:
script=/etc/xen/scripts/vif-route-ifup
e. Activate the new configuration and start the VM Guest.
4. The remaining configuration tasks need to be accomplished from inside the

VM Guest.
a. Open a console to the VM Guest with xl console domain and log

in.
b. Check whether the guest IP is set to 192.168.100.1.
196 z 366 2017-05-27 13:21

c. Provide VM Guest with a host route and a default gateway to the VM

Host Server. Do this by adding the following lines to
/etc/sysconfig/network/routes :
192.168.1.20 - - eth0
default 192.168.1.20 - -
5. Finally, test the network connection from the VM Guest to the outside world.

There are many network configuration possibilities available to Xen. The following
configurations are not activated by default:
19.4.1 Bandwidth Throttling in Virtual Networks

With Xen, you may limit the network transfer rate a virtual guest may use to access a
bridge. To configure this, you need to modify the VM Guest configuration as de-
scribed in Section 20.1, “XL—Xen Management Tool”.
In the configuration file, first search for the device that is connected to the virtual
bridge. The configuration looks like the following:
vif = [ 'mac=00:16:3e:4f:94:a9,bridge=br0' ]
To add a maximum transfer rate, add a parameter rate to this configuration as in:
vif = [ 'mac=00:16:3e:4f:94:a9,bridge=br0,rate=100Mb/s' ]
Note that the rate is either Mb/s (megabits per second) or MB/s (megabytes per
second). In the above example, the maximum transfer rate of the virtual interface is
100 megabits. By default, there is no limitation to the bandwidth of a guest to the
virtual bridge.
It is even possible to fine-tune the behavior by specifying the time window that is
used to define the granularity of the credit replenishment:
vif = [ 'mac=00:16:3e:4f:94:a9,bridge=br0,rate=100Mb/s@20ms' ]
19.4.2 Monitoring the Network Traffic
197 z 366 2017-05-27 13:21

To monitor the traffic on a specific interface, the little application iftop is a nice
program that displays the current network traffic in a terminal.
When running a Xen VM Host Server, you need to define the interface that is moni-
tored. The interface that Dom0 uses to get access to the physical network is the
bridge device, for example br0 . This, however, may vary on your system. To moni-
tor all traffic to the physical interface, run a terminal as root and use the com-
mand:
iftop -i br0
To monitor the network traffic of a special network interface of a specific VM Guest,

supply the correct virtual interface. For example, to monitor the first Ethernet device
of the domain with id 5, use the command:
ftop -i vif5.0
To quit iftop , press the key Q . More options and possibilities are available in the
manual page man 8 iftop .
20 Managing a Virtualization
Environment
20.3 Event Actions
Apart from using the recommended libvirt library (Part II, “Managing Virtual Ma-
chines with libvirt ”), you can manage Xen guest domains with the xl tool from
the command line.
198 z 366 2017-05-27 13:21


The xl program is a tool for managing Xen guest domains. It is part of the
xen-tools package. xl is based on the LibXenlight library, and can be used for
general domain management, such as domain creation, listing, pausing, or shutting
down. Usually you need to be root to execute xl commands.
Note
xl can only manage running guest domains specified by their configuration file.
If a guest domain is not running, you cannot manage it with xl .
Tip
To allow users to continue to have managed guest domains in the way the obso-
lete xm command allowed, we now recommend using libvirt 's virsh and
virt-manager tools. For more information, see Part II, “Managing Virtual Ma-
chines with libvirt ”.
xl operations rely upon xenstored and xenconsoled services. Make sure you
start
systemctl start xencommons
at boot time to initialize all the daemons required by xl .
Tip: Set up a xenbr0 Network Bridge in the Host Domain
In the most common network configuration, you need to set up a bridge in the
host domain named xenbr0 to have a working network for the guest domains.
The basic structure of every xl command is:
xl <subcommand> [options] domain_id
where <subcommand> is the xl command to run, domain_id is the ID number as-

signed to a domain or the name of the virtual machine, and OPTIONS indicates sub-
command-specific options.
199 z 366 2017-05-27 13:21

For a complete list of the available xl subcommands, run xl help . For each com-
mand, there is a more detailed help available that is obtained with the extra parame-
ter --help . More information about the respective subcommands is available in
the manual page of xl .
For example, the xl list --help displays all options that are available to the list
command. As an example, the xl list command displays the status of all virtual
machines.
# xl list
Name ID Mem VCPUs State Time(s)
Domain-0 0 457 2 r----- 2712.9
sles12 7 512 1 -b---- 16.3
opensuse 512 1 12.9
The State information indicates if a machine is running, and in which state it is. The
most common flags are r (running) and b (blocked) where blocked means it is ei-
ther waiting for IO, or sleeping because there is nothing to do. For more details
about the state flags, see man 1 xl .
Other useful xl commands include:
xl create creates a virtual machine from a given configuration file.
xl reboot reboots a virtual machine.
xl destroy immediately terminates a virtual machine.
xl block-list displays all virtual block devices attached to a virtual ma-

chine.
20.1.1 Guest Domain Configuration File

When operating domains, xl requires a domain configuration file for each domain.
The default directory to store such configuration files is /etc/xen/ .
A domain configuration file is a plain text file. It consists of several key = value
pairs. Some keys are mandatory, some are general and apply to any guest, and
some apply only to a specific guest type (para or fully virtualized). A value can either
be a "string" surrounded by single or double quotes, a number, a boolean value,
or a list of several values enclosed in brackets [ value1, value2, ... ] .
EXAMPLE 20.1: GUEST DOMAIN CONFIGURATION FILE: /etc/xen/sled12.cfg
name= "sled12"
builder = "hvm"
vncviewer = 1
memory = 512
200 z 366 2017-05-27 13:21

disk = [ '/var/lib/xen/images/sled12.raw,,hda', '/dev/cdrom,,hdc,cdrom' ]

vif = [ 'mac=00:16:3e:5f:48:e4,model=rtl8139,bridge=br0' ]
boot = "n"
To start such domain, run xl create /etc/xen/sled12.cfg .

To make a guest domain start automatically after the host system boots, follow these
steps:
1. Create the domain configuration file if it does not exist, and save it in the
/etc/xen/ directory, for example /etc/xen/domain_name.cfg .
2. Make a symbolic link of the guest domain configuration file in the auto/
subdirectory.
ln -s /etc/xen/domain_name.cfg /etc/xen/auto/domain_name.cfg
3. On the next system boot, the guest domain defined in domain_name.cfg

will be started.
20.3 Event Actions

In the guest domain configuration file, you can define actions to be performed on a
predefined set of events. For example, to tell the domain to restart itself after it is
powered off, include the following line in its configuration file:
on_poweroff="restart"
A list of predefined events for a guest domain follows:
LIST OF EVENTS
on_poweroff
Specifies what should be done with the domain if it shuts itself down.
on_reboot
Action to take if the domain shuts down with a reason code requesting a reboot.
on_watchdog
Action to take if the domain shuts down because of a Xen watchdog timeout.
201 z 366 2017-05-27 13:21

on_crash
Action to take if the domain crashes.
For these events, you can define one of the following actions:
LIST OF RELATED ACTIONS

destroy
Destroy the domain.
restart
Destroy the domain and immediately create a new domain with the same con-
figuration.
rename-restart
Rename the domain that terminated, and then immediately create a new do-
main with the same configuration as the original.
preserve
Keep the domain. It can be examined, and later destroyed with xl destroy .
coredump-destroy
Write a core dump of the domain to /var/xen/dump/NAME and then destroy
the domain.
coredump-restart
Write a core dump of the domain to /var/xen/dump/NAME and then restart
the domain.

The Time Stamp Counter (TSC) may be specified for each domain in the guest do-
main configuration file (for more information, see Section 20.1.1, “Guest Domain
Configuration File”).
With the tsc_mode setting, you specify whether rdtsc instructions are executed “na-
tively ” (fast, but TSC-sensitive applications may sometimes run incorrectly) or emu-
lated (always run correctly, but performance may suffer).
tsc_mode=0 (default)
202 z 366 2017-05-27 13:21

Use this to ensure correctness while providing the best performance possi-
ble—for more information, see https://xenbits.xen.org/docs/4.3-testing
/misc/tscmode.txt↗.
tsc_mode=1 (always emulate)

Use this when TSC-sensitive apps are running and worst-case performance
degradation is known and acceptable.
tsc_mode=2 (never emulate)

Use this when all applications running in this VM are TSC-resilient and highest
performance is required.
tsc_mode=3 (PVRDTSCP)
High-TSC-frequency applications may be paravirtualized (modified) to obtain
both correctness and highest performance—any unmodified applications must
be TSC-resilient.
For background information, see https://xenbits.xen.org/docs/4.3-testing

/misc/tscmode.txt↗.

PROCEDURE 20.1: SAVE A VIRTUAL MACHINE’S CURRENT STATE
1. Make sure the virtual machine to be saved is running.
2. In the host environment, enter
xl save ID state-file
where ID is the virtual machine ID you want to save, and state-file is

the name you specify for the memory state file. By default, the domain will
no longer be running after you create its snapshot. Use -c to keep it run-
ning even after you create the snapshot.

PROCEDURE 20.2: RESTORE A VIRTUAL MACHINE’S CURRENT STATE
1.
203 z 366 2017-05-27 13:21

Make sure the virtual machine to be restored has not been started since you
ran the save operation.
2. In the host environment, enter
xl restore state-file
where state-file is the previously saved memory state file. By default,

the domain will be running after it is restored. To pause it after the restore,
use -p .

A virtual machine’s state can be displayed by viewing the results of the xl list
command, which abbreviates the state using a single character.
r - running - The virtual machine is currently running and consuming allo-

cated resources.
b - blocked - The virtual machine’s processor is not running and not able to
run. It is either waiting for I/O or has stopped working.
p - paused - The virtual machine is paused. It does not interact with the hy-
pervisor but still maintains its allocated resources, such as memory.
s - shutdown - The guest operating system is in the process of being shut

down, rebooted, or suspended, and the virtual machine is being stopped.
c - crashed - The virtual machine has crashed and is not running.
d - dying - The virtual machine is in the process of shutting down or crash-

ing.

204 z 366 2017-05-27 13:21


21.3 File-Backed Virtual Disks and Loopback Devices
21.5 Scripts for Managing Advanced Storage Scenarios

The disk(s) specification for a Xen domain in the domain configuration file is as
straightforward as the following example:
disk = [ 'format=raw,vdev=hdc,access=ro,devtype=cdrom,target=/root/image.iso' ]
It defines a disk block device based on the /root/image.iso disk image file. The
disk will be seen as hdc by the guest, with read-only ( ro ) access. The type of the
device is cdrom with raw format.
The following example defines an identical device, but using simplified positional
syntax:
disk = [ '/root/image.iso,raw,hdc,ro,cdrom' ]
You can include more disk definitions in the same line, each one separated by a
comma. If a parameter is not specified, then its default value is taken:
disk = [ '/root/image.iso,raw,hdc,ro,cdrom','/dev/vg/guest-volume,,hda','...' ]
LIST OF PARAMETERS
target
Source block device or disk image file path.
format
The format of the image file. Default is raw .
vdev
Virtual device as seen by the guest. Supported values are hd[x], xvd[x], sd[x] etc.
See /usr/share/doc/packages/xen/misc/vbd-interface.txt for more
details. This parameter is mandatory.
205 z 366 2017-05-27 13:21

access
Whether the block device is provided to the guest in read-only or read-write
mode. Supported values are ro or r for read-only, and rw or w for
read/write access. Default is ro for devtype=cdrom , and rw for other device
types.
devtype
Qualifies virtual device type. Supported value is cdrom .
backendtype
The back-end implementation to use. Supported values are phy , tap , and
qdisk . Normally this option should not be specified as the back-end type is au-
tomatically determined.
script
Specifies that target is not a normal host path, but rather information to be
interpreted by the executable program. The specified script file is looked for in
/etc/xen/scripts if it does not point to an absolute path. These scripts are
normally called block-<script_name> .
For more information about specifying virtual disks, see

/usr/share/doc/packages/xen/misc/xl-disk-configuration.txt .

Similar to mapping a local disk image (see Section 21.1, “Mapping Physical Storage to
Virtual Disks”), you can map a network disk as a virtual disk as well.
The following example shows mapping of an RBD (RADOS Block Device) disk with
multiple Ceph monitors and cephx authentication enabled:
disk = [ 'vdev=hdc, backendtype=qdisk, \

target=rbd:libvirt-pool/new-libvirt-image:\
id=libvirt:key=AQDsPWtW8JoXJBAAyLPQe7MhCC+JPkI3QuhaAw==:auth_supported=cephx;none:\
mon_host=137.65.135.205\\:6789;137.65.135.206\\:6789;137.65.135.207\\:6789' ]
Following is an example of an NBD (Network Block Device) disk mapping:
disk = [ 'vdev=hdc, backendtype=qdisk, target=nbd:151.155.144.82:5555' ]
21.3 File-Backed Virtual Disks and Loopback
206 z 366 2017-05-27 13:21

Devices
When a virtual machine is running, each of its file-backed virtual disks consumes a
loopback device on the host. By default, the host allows up to 64 loopback devices
to be consumed.
To simultaneously run more file-backed virtual disks on a host, you can increase the
number of available loopback devices by adding the following option to the host’s
/etc/modprobe.conf.local file.
options loop max_loop=x
where x is the maximum number of loopback devices to create.
Changes take effect after the module is reloaded.
Tip
Enter rmmod loop and modprobe loop to unload and reload the module. In
case rmmod does not work, unmount all existing loop devices or reboot the com-
puter.

While it is always possible to add new block devices to a VM Guest system, it is some-
times more desirable to increase the size of an existing block device. In case such a
system modification is already planned during deployment of the VM Guest, some
basic considerations should be done:
Use a block device that may be increased in size. LVM devices and file system
images are commonly used.
Do not partition the device inside the VM Guest, but use the main device di-
rectly to apply the file system. For example, use /dev/xvdb directly instead
of adding partitions to /dev/xvdb .
Make sure that the file system to be used can be resized. Sometimes, for ex-
ample with Ext3, some features must be switched off to be able to resize the
file system. A file system that can be resized online and mounted is XFS . Use
the command xfs_growfs to resize that file system after the underlying
block device has been increased in size. For more information about XFS ,
see man 8 xfs_growfs .
207 z 366 2017-05-27 13:21

When resizing an LVM device that is assigned to a VM Guest, the new size is automat-
ically known to the VM Guest. No further action is needed to inform the VM Guest
about the new size of the block device.
When using file system images, a loop device is used to attach the image file to the
guest. For more information about resizing that image and refreshing the size infor-
mation for the VM Guest, see Section 23.2, “Sparse Image Files and Disk Space”.
21.5 Scripts for Managing Advanced Storage

Scenarios
There are scripts that can help with managing advanced storage scenarios such as
disk environments provided by dmmd (“device mapper—multi disk”) including LVM
environments built upon a software RAID set, or a software RAID set built upon an
LVM environment. These scripts are part of the xen-tools package. After installa-
tion, they can be found in /etc/xen/scripts :
block-dmmd
block-drbd-probe
block-npiv
The scripts allow for external commands to perform some action, or series of ac-
tions of the block devices prior to serving them up to a guest.
These scripts could formerly only be used with xl or libxl using the disk config-
uration syntax script= . They can now be used with libvirt by specifying the base
name of the block script in the <source> element of the disk. For example:
<source dev='dmmd:md;/dev/md0;lvm;/dev/vgxen/lv-vm01'/>
22 Virtualization: Configuration
Options and Settings
208 z 366 2017-05-27 13:21


22.3 VNC Viewer
22.6 HVM Features
The documentation in this section, describes advanced management tasks and con-
figuration options that might help technology innovators implement leading-edge
virtualization solutions. It is provided as a courtesy and does not imply that all docu-
mented options and tasks are supported by Novell, Inc.

Virtual CD readers can be set up when a virtual machine is created or added to an
existing virtual machine. A virtual CD reader can be based on a physical CD/DVD, or
based on an ISO image. Virtual CD readers work differently depending on whether
they are paravirtual or fully virtual.
22.1.1 Virtual CD Readers on Paravirtual Machines

A paravirtual machine can have up to 100 block devices composed of virtual CD
readers and virtual disks. On paravirtual machines, virtual CD readers present the
CD as a virtual disk with read-only access. Virtual CD readers cannot be used to write
data to a CD.
After you have finished accessing a CD on a paravirtual machine, it is recommended

that you remove the virtual CD reader from the virtual machine.
Paravirtualized guests can use the device type devtype=cdrom . This partly emu-
lates the behavior of a real CD reader, and allows CDs to be changed. It is even pos-
sible to use the eject command to open the tray of the CD reader.
22.1.2 Virtual CD Readers on Fully Virtual Machines
209 z 366 2017-05-27 13:21

A fully virtual machine can have up to four block devices composed of virtual CD
readers and virtual disks. A virtual CD reader on a fully virtual machine interacts with
an inserted CD in the way you would expect a physical CD reader to interact. For ex-
ample, in a Windows* XP* virtual machine, the inserted CD appears in the
Devices with Removable Storage section of My Computer .
When a CD is inserted in the physical CD reader on the host computer, all virtual
machines with virtual CD readers based on the physical CD reader, such as
/dev/cdrom/ , can read the inserted CD. Assuming the operating system has auto-
mount functionality, the CD should automatically appear in the file system. Virtual
CD readers cannot be used to write data to a CD. They are configured as read-only
devices.
22.1.3 Adding Virtual CD Readers

Virtual CD readers can be based on a CD inserted into the CD reader or on an ISO
image file.
1. Make sure that the virtual machine is running and the operating system has
finished booting.
2. Insert the desired CD into the physical CD reader or copy the desired ISO
image to a location available to Dom0.
3. Select a new, unused block device in your VM Guest, such as /dev/xvdb .
4. Choose the CD reader or ISO image that you want to assign to the guest.
5. When using a real CD reader, use the following command to assign the CD
reader to your VM Guest. In this example, the name of the guest is alice:
xl block-attach alice target=/dev/sr0,vdev=xvdb,access=ro
6. When assigning an image file, use the following command:
xl block-attach alice target=/path/to/file.iso,vdev=xvdb,access=ro
7. A new block device, such as /dev/xvdb , is added to the virtual machine.
8. If the virtual machine is running Linux, complete the following:
a. Open a terminal in the virtual machine and enter fdisk -l to verify

that the device was properly added. You can also enter
ls /sys/block to see all disks available to the virtual machine.
The CD is recognized by the virtual machine as a virtual disk with a

drive designation, for example:
210 z 366 2017-05-27 13:21

/dev/xvdb
b. Enter the command to mount the CD or ISO image using its drive des-
ignation. For example,
mount -o ro /dev/xvdb /mnt
mounts the CD to a mount point named /mnt .
The CD or ISO image file should be available to the virtual machine at

the specified mount point.
9. If the virtual machine is running Windows, reboot the virtual machine.
Verify that the virtual CD reader appears in its My Computer section.
22.1.4 Removing Virtual CD Readers

1. Make sure that the virtual machine is running and the operating system has
finished booting.
2. If the virtual CD reader is mounted, unmount it from within the virtual ma-
chine.
3. Enter xl block-list alice on the host view of the guest block devices.
4. Enter xl block-detach alice block_dev_id to remove the virtual de-

vice from the guest. If that fails, try to add -f to force the removal.
5. Press the hardware eject button to eject the CD.

Some configurations, such as those that include rack-mounted servers, require a
computer to run without a video monitor, keyboard, or mouse. This type of configu-
ration is often called headless and requires the use of remote administration tech-
nologies.
Typical configuration scenarios and technologies include:
Graphical Desktop with X Window Server
211 z 366 2017-05-27 13:21

If a graphical desktop, such as GNOME, is installed on the virtual machine host,

you can use a remote viewer, such as a VNC viewer. On a remote computer, log
in and manage the remote guest environment by using graphical tools, such as
tigervnc or virt-viewer .
Text Only
You can use the ssh command from a remote computer to log in to a virtual
machine host and access its text-based console. You can then use the xl com-
mand to manage virtual machines, and the virt-install command to create
new virtual machines.
22.3 VNC Viewer

VNC viewer is used to view the environment of the running guest system in a graphi-
cal way. You can use it from Dom0 (known as local access or on-box access), or from
a remote computer.
You can use the IP address of a VM Host Server and a VNC viewer to view the display
of this VM Guest. When a virtual machine is running, the VNC server on the host as-
signs the virtual machine a port number to be used for VNC viewer connections. The
assigned port number is the lowest port number available when the virtual machine
starts. The number is only available for the virtual machine while it is running. After
shutting down, the port number might be assigned to other virtual machines.
For example, if ports 1 and 2 and 4 and 5 are assigned to the running virtual ma-
chines, the VNC viewer assigns the lowest available port number, 3. If port number 3
is still in use the next time the virtual machine starts, the VNC server assigns a differ-
ent port number to the virtual machine.
To use the VNC viewer from a remote computer, the firewall must permit access to
as many ports as VM Guest systems run from. This means from port 5900 and up.
For example, to run 10 VM Guest systems, you need to open the TCP ports
5900:5910.
To access the virtual machine from the local console running a VNC viewer client, en-
ter one of the following commands:
vncviewer ::590#
vncviewer :#
# is the VNC viewer port number assigned to the virtual machine.
212 z 366 2017-05-27 13:21

When accessing the VM Guest from a machine other than Dom0, use the following
syntax:
vncviewer 192.168.1.20::590#
In this case, the IP address of Dom0 is 192.168.1.20.
22.3.1 Assigning VNC Viewer Port Numbers to Virtual

Machines
Although the default behavior of VNC viewer is to assign the first available port num-
ber, you should assign a specific VNC viewer port number to a specific virtual ma-
chine.
To assign a specific port number on a VM Guest, edit the xl setting of the virtual ma-
chine and change the vnclisten to the desired value. Note that for example for
port number 5902, specify 2 only, as 5900 is added automatically:
vfb = [ 'vnc=1,vnclisten="localhost:2"' ]
For more information regarding editing the xl settings of a guest domain, see Sec-
tion 20.1, “XL—Xen Management Tool”.
Tip
Assign higher port numbers to avoid conflict with port numbers assigned by the
VNC viewer, which uses the lowest available port number.
22.3.2 Using SDL instead of a VNC Viewer

If you access a virtual machine's display from the virtual machine host console
(known as local or on-box access), you should use SDL instead of VNC viewer. VNC
viewer is faster for viewing desktops over a network, but SDL is faster for viewing
desktops from the same computer.
To set the default to use SDL instead of VNC, change the virtual machine's configura-
tion information to the following. For instructions, see Section 20.1, “XL—Xen Man-
agement Tool”.
vfb = [ 'sdl=1' ]
Remember that, unlike a VNC viewer window, closing an SDL window terminates the
virtual machine.
213 z 366 2017-05-27 13:21


When a virtual machine is started, the host creates a virtual keyboard that matches
the keymap entry according to the virtual machine's settings. If there is no keymap
entry specified, the virtual machine's keyboard defaults to English (US).
To view a virtual machine's current keymap entry, enter the following command on
the Dom0:
xl list -l vm_name | grep keymap
To configure a virtual keyboard for a guest, use the following snippet:
vfb = [ 'keymap="de"' ]
For a complete list of supported keyboard layouts, see the Keymaps section of the
xl.cfg manual page man 5 xl.cfg .

In Xen it is possible to specify how many and which CPU cores the Dom0 or VM
Guest should use to retain its performance. The performance of Dom0 is important
for the overall system, as the disk and network drivers are running on it. Also I/O in-
tensive guests' workloads may consume lots of Dom0s' CPU cycles. On the other
hand, the performance of VM Guests is also important, to be able to accomplish the
task they were set up for.
22.5.1 Dom0
Dedicating CPU resources to Dom0 results in a better overall performance of the vir-
tualized environment because Dom0 has free CPU time to process I/O requests from
VM Guests. Failing to dedicate exclusive CPU resources to Dom0 usually results in a
poor performance and can cause the VM Guests to function incorrectly.
Dedicating CPU resources involves three basic steps: modifying Xen boot line, bind-
ing Dom0's VCPUs to a physical processor, and configuring CPU-related options on
VM Guests:
1. First you need to append the dom0_max_vcpus=X to the Xen boot line. Do
so by adding the following line to /etc/default/grub :
214 z 366 2017-05-27 13:21

GRUB_CMDLINE_XEN="dom0_max_vcpus=X"
If /etc/default/grub already contains a line setting

GRUB_CMDLINE_XEN , rather append dom0_max_vcpus=X to this line.
X needs to be replaced by the number of VCPUs dedicated to Dom0.
2. Update the GRUB 2 configuration file by running the following command:
grub2-mkconfig -o /boot/grub2/grub.cfg
3. Reboot for the change to take effect.
4. The next step is to bind (or “pin”) each Dom0's VCPU to a physical processor.
xl vcpu-pin Domain-0 0 0
xl vcpu-pin Domain-0 1 1
The first line binds Dom0's VCPU number 0 to the physical processor num-
ber 0, while the second line binds Dom0's VCPU number 1 to the physical
processor number 1.
5. Lastly, you need to make sure no VM Guest uses the physical processors
dedicated to VCPUs of Dom0. Assuming you are running an 8-CPU system,
you need to add
cpus="2-8"
to the configuration file of the relevant VM Guest.
22.5.2 VM Guests
It is often necessary to dedicate specific CPU resources to a virtual machine. By de-
fault, a virtual machine uses any available CPU core. Its performance can be im-
proved by assigning a reasonable number of physical processors to it as other VM
Guests are not allowed to use them after that. Assuming a machine with 8 CPU cores
while a virtual machine needs to use 2 of them, change its configuration file as fol-
lows:
vcpus=2
cpus="2,3"
The above example dedicates 2 processors to the VM Guest, and these being the
3rd and 4th one, ( 2 and 3 counted from zero). If you need to assign more physical
processors, use the cpus="2-8" syntax.
215 z 366 2017-05-27 13:21

If you need to change the CPU assignment for a guest named “alice” in a hotplug
manner, do the following on the related Dom0:
xl vcpu-set alice 2
xl vcpu-pin alice 0 2
xl vcpu-pin alice 1 3
The example will dedicate 2 physical processors to the guest, and bind its VCPU 0 to
physical processor 2 and VCPU 1 to physical processor 3. Now check the assignment:
xl vcpu-list alice
Name ID VCPUs CPU State Time(s) CPU Affinity
alice 4 0 2 -b- 1.9 2-3
alice 4 1 3 -b- 2.8 2-3
22.6 HVM Features

In Xen some features are only available for fully virtualized domains. They are not
very often used, but still may be interesting in some environments.
22.6.1 Specify Boot Device on Boot

Just as with physical hardware, it is sometimes desirable to boot a VM Guest from a
different device than its own boot device. For fully virtual machines, it is possible to
select a boot device with the boot parameter in a domain xl configuration file:
boot = boot_device
boot_device can be one of c for hard disk, d for CD-ROM, or n for Net-
work/PXE. You can specify multiple options, and they will be attempted in the given
order. For example,
boot = dc
boots from CD-ROM, and falls back to the hard disk if CD-ROM is not bootable.
22.6.2 Changing CPUIDs for Guests
216 z 366 2017-05-27 13:21

To be able to migrate a VM Guest from one VM Host Server to a different VM Host

Server, it is mandatory, that the VM Guest system only uses CPU features that are
available on both VM Host Server systems. If the actual CPUs are different on both
hosts, it may be necessary to hide some features before the VM Guest is started to
maintain the possibility to migrate the VM Guest between both hosts. For fully virtu-
alized guests, this can be achieved by configuring the cpuid that is available to the
guest.
To gain an overview of the current CPU, have a look at /proc/cpuinfo . This con-
tains all the important information that defines the current CPU.
To redefine a CPU, first have a look at the respective cpuid definitions of the CPU
vendor. These are available from:
Intel
http://www.intel.com/Assets/PDF/appnote/241618.pdf↗
cpuid = "host,tm=0,sse3=0"
The syntax is a comma-separated list of key=value pairs, preceded by the word

"host". A few keys take a numerical value, while all others take a single character
which describes what to do with the feature bit. See man 5 xl.cfg for a complete
list of cpuid keys. The respective bits may be changed by using the following values:
1
Force the corresponding bit to 1
0
Force the corresponding bit to 0
x
Use the values of the default policy
k
Use the values defined by the host
s
Like k , but preserve the value over migrations
Note that counting bits is done from right to left, starting with bit 0 .
22.6.3 Increasing the Number of PCI-IRQs
217 z 366 2017-05-27 13:21

In case you need to increase the default number of PCI-IRQs available to Dom0
and/or VM Guest, you can do so by modifying the Xen kernel command line. Use the
command extra_guest_irqs= domu_irgs,dom0_irgs . The optional first num-
ber domu_irgs is common for all VM Guests, while the optional second number
dom0_irgs (preceded by a comma) is for Dom0. Changing the setting for VM Guest
has no impact on Dom0 and vice versa. For example to change Dom0 without
changing VM Guest, use
extra_guest_irqs=,512
23.4 Monitoring Xen
23.5 Providing Host Information for VM Guest Systems

The boot loader controls how the virtualization software boots and runs. You can
modify the boot loader properties by using YaST, or by directly editing the boot
loader configuration file.
The YaST boot loader program is located at YaST › System › Boot Loader. Click the
Bootloader Options tab and select the line containing the Xen kernel as the Default
Boot Section.
218 z 366 2017-05-27 13:21

FIGURE 23.1: BOOT LOADER SETTINGS
Confirm with OK. Next time you boot the host, it will be ready to provide the Xen vir-
tualization environment.
You can use the Boot Loader program to specify functionality, such as:
Pass kernel command line parameters.
Specify the kernel image and initial RAM disk.
Select a specific hypervisor.
Pass additional parameters to the hypervisor. See http://xenbits.xen.org

/docs/unstable/misc/xen-command-line.html↗ for their complete list.
You can customize your virtualization environment by editing the

/etc/default/grub file. Add the following line to this file:
GRUB_CMDLINE_XEN="<boot_parameters>" . Do not forget to run
grub2-mkconfig -o /boot/grub2/grub.cfg after editing the file.
219 z 366 2017-05-27 13:21

If the host’s physical disk reaches a state where it has no available space, a virtual
machine using a virtual disk based on a sparse image file cannot write to its disk.
Consequently, it reports I/O errors.
If this situation occurs, you should free up available space on the physical disk, re-
mount the virtual machine’s file system, and set the file system back to read-write.
To check the actual disk requirements of a sparse image file, use the command
du -h <image file> .
To increase the available space of a sparse image file, first increase the file size and
then the file system.
Warning: Back Up Before Resizing
Touching the sizes of partitions or sparse files always bears the risk of data fail-
ure. Do not work without a backup.
The resizing of the image file can be done online, while the VM Guest is running. In-
crease the size of a sparse image file with:
dd if=/dev/zero of=<image file> count=0 bs=1M seek=<new size in MB>
For example, to increase the file /var/lib/xen/images/sles/disk0 to a size of

16GB, use the command:
dd if=/dev/zero of=/var/lib/xen/images/sles/disk0 count=0 bs=1M seek=16000
Note: Increasing Non-Sparse Images
It is also possible to increase the image files of devices that are not sparse files.
However, you must know exactly where the previous image ends. Use the seek
parameter to point to the end of the image file and use a command similar to the
following:
dd if=/dev/zero of=/var/lib/xen/images/sles/disk0 seek=8000 bs=1M count=2000
Be sure to use the right seek, else data loss may happen.
If the VM Guest is running during the resize operation, also resize the loop device
that provides the image file to the VM Guest. First detect the correct loop device with
the command:
losetup -j /var/lib/xen/images/sles/disk0
220 z 366 2017-05-27 13:21

Then resize the loop device, for example /dev/loop0 , with the following com-
mand:
losetup -c /dev/loop0
Finally check the size of the block device inside the guest system with the command
fdisk -l /dev/xvdb . The device name depends on the actually increased device.
The resizing of the file system inside the sparse file involves tools that are depending
on the actual file system. This is described in detail in the Book “Storage
Administration Guide”.

With Xen it is possible to migrate a VM Guest system from one VM Host Server to an-
other with almost no service interruption. This could be used for example to move a
busy VM Guest to a VM Host Server that has stronger hardware or is not yet loaded.
Or, if a service of a VM Host Server is required, all VM Guest systems running on this
machine can be migrated to other machines to avoid interruption of service. These
are only two examples—many more reasons may apply to your personal situation.
Before starting, some preliminary considerations regarding the VM Host Server

should be taken into account:
All VM Host Server systems should use a similar CPU. The frequency is not so
important, but they should be using the same CPU family. To get more infor-
mation about the used CPU, see cat /proc/cpuinfo .
All resources that are used by a specific guest system must be available on all
involved VM Host Server systems—for example all used block devices must
exist on both VM Host Server systems.
If the hosts included in the migration process run in different subnets, make
sure that either DHCP relay is available to the guests, or for guests with static
network configuration, set up the network manually.
Using special features like PCI Pass-Through may be problematic. Do not

implement these when deploying for an environment that should migrate
VM Guest systems between different VM Host Server systems.
For fast migrations, a fast network is mandatory. If possible, use GB Ethernet

and fast switches. Deploying VLAN might also help avoid collisions.
221 z 366 2017-05-27 13:21

23.3.1 Preparing Block Devices for Migrations

The block devices needed by the VM Guest system must be available on all involved
VM Host Server systems. This is done by implementing some kind of shared storage
that serves as container for the root file system of the migrated VM Guest system.
Common possibilities include:
iSCSI can be set up to give access to the same block devices from different
systems at the same time. For more information about iSCSI, see Book
“Storage Administration Guide”, Chapter 14 “Mass Storage over IP Networks:
iSCSI”.
NFS is a widely used root file system that can easily be accessed from differ-
ent locations. For more information, see Book “Administration Guide”, Chap-
ter 25 “Sharing File Systems with NFS”.
DRBD can be used if only two VM Host Server systems are involved. This
gives some extra data security, because the used data is mirrored over the
network. For more information, see the SUSE Linux Enterprise High Availability
Extension 12 SP2 documentation at http://www.suse.com/doc/↗.
SCSI can also be used if the available hardware permits shared access to
the same disks.
NPIV is a special mode to use Fibre channel disks. However, in this case all
migration hosts must be attached to the same Fibre channel switch. For
more information about NPIV, see Section 21.1, “Mapping Physical Storage to
Virtual Disks”. Commonly, this works if the Fibre channel environment sup-
ports 4 Gbit or faster connections.
23.3.2 Migrating VM Guest Systems

The actual migration of the VM Guest system is done with the command:
xl migrate <domain_name> <host>
The speed of the migration depends on how fast the memory print can be saved to
disk, sent to the new VM Host Server and loaded there. This means that small VM
Guest systems can be migrated faster than big systems with a lot of memory.
23.4 Monitoring Xen
222 z 366 2017-05-27 13:21

For a regular operation of many virtual guests, having a possibility to check the san-
ity of all the different VM Guest systems is indispensable. Xen offers several tools be-
sides the system tools to gather information about the system.
Tip: Monitoring the VM Host Server
Basic monitoring of the VM Host Server (I/O and CPU) is available via the Virtual
Machine Manager. Refer to Section 10.8.1, “Monitoring with Virtual Machine
Manager” for details.
23.4.1 Monitor Xen with xentop

The preferred terminal application to gather information about Xen virtual environ-
ment is xentop . Unfortunately, this tool needs a rather broad terminal, else it in-
serts line breaks into the display.
xentop has several command keys that can give you more information about the
system that is monitored. Some of the more important are:
D
Change the delay between the refreshes of the screen.
N
Also display network statistics. Note, that only standard configurations will be
displayed. If you use a special configuration like a routed network, no network
will be displayed.
B
Display the respective block devices and their cumulated usage count.
For more information about xentop see the manual page man 1 xentop .
Tip: virt-top
libvirt offers the hypervisor-agnostic tool virt-top , which is recommended for

monitoring VM Guests. See Section 10.8.2, “Monitoring with virt-top ” for de-
tails.
223 z 366 2017-05-27 13:21

23.4.2 Additional Tools

There are many system tools that also help monitoring or debugging a running
SUSE Linux Enterprise system. Many of these are covered in the official SUSE Linux
Enterprise documentation. Especially useful for monitoring a virtualization environ-
ment are the following tools:
ip
The command line utility ip may be used to monitor arbitrary network inter-
faces. This is especially useful if you have set up a network that is routed or ap-
plied a masqueraded network. To monitor a network interface with the name
alice.0 , run the following command:
watch ip -s link show alice.0
brctl
In a standard setup, all the Xen VM Guest systems are attached to a virtual net-
work bridge. brctl allows you to determine the connection between the
bridge and the virtual network adapter in the VM Guest system. For example,
the output of brctl show may look like the following:

br0 8000.000476f060cc no eth0
vif1.0
br1 8000.00001cb5a9e7 no vlan22
This shows that there are two virtual bridges defined on the system. One is con-
nected to the physical Ethernet device eth0 , the other one is connected to a
VLAN interface vlan22 .
There is only one guest interface active in this setup, vif1.0 . This means that
the guest with ID 1 has an Ethernet interface eth0 assigned, that is connected
to br0 in the VM Host Server.
iptables-save
Especially when using masquerade networks, or if several Ethernet interfaces
are set up together with a firewall setup, it may be helpful to check the current
firewall rules.
The command iptables may be used to check all the different firewall set-
tings. To list all the rules of a chain, or even of the complete setup, you may use
the commands iptables-save or iptables -S .
224 z 366 2017-05-27 13:21

23.5 Providing Host Information for VM Guest

Systems
In a standard Xen environment, the VM Guest systems have only very limited infor-
mation about the VM Host Server system they are running on. If a guest should
know more about the VM Host Server it runs on, vhostmd can provide more infor-
mation to selected guests. To set up your system to run vhostmd , proceed as fol-
lows:
1. Install the package vhostmd on the VM Host Server.
2. To add or remove metric sections from the configuration, edit the file
/etc/vhostmd/vhostmd.conf . However, the default works well.
3. Check the validity of the vhostmd.conf configuration file with the com-
mand:
cd /etc/vhostmd
xmllint --postvalid --noout vhostmd.conf
4. Start the vhostmd daemon with the command

sudo systemctl start vhostmd .
If vhostmd should be started automatically during start-up of the system,

run the command:
sudo systemctl enable vhostmd
5. Attach the image file /dev/shm/vhostmd0 to the VM Guest system named

alice with the command:
xl block-attach opensuse /dev/shm/vhostmd0,,xvdb,ro
6. Log on the VM Guest system.
7. Install the client package vm-dump-metrics .
8. Run the command vm-dump-metrics . To save the result to a file, use the
option -d <filename> .
The result of the vm-dump-metrics is an XML output. The respective metric entries
follow the DTD /etc/vhostmd/metric.dtd .
225 z 366 2017-05-27 13:21

For more information, see the manual pages man 8 vhostmd and
/usr/share/doc/vhostmd/README on the VM Host Server system. On the guest,
see the manual page man 1 vm-dump-metrics .
24 XenStore: Configuration Database

Shared between Domains
24.1 Introduction
This section introduces basic information about XenStore, its role in the Xen envi-
ronment, the directory structure of files used by XenStore, and the description of
XenStore's commands.
24.1 Introduction
XenStore is a database of configuration and status information shared between VM
Guests and the management tools running in Dom0. VM Guests and the manage-
ment tools read and write to XenStore to convey configuration information, status
updates, and state changes. The XenStore database is managed by Dom0 and sup-
ports simple operations such as reading and writing a key. VM Guests and manage-
ment tools can be notified of any changes in XenStore by watching entries of inter-
est. Note that the xenstored daemon is managed by the xencommons service.
XenStore is located on Dom0 in a single database file /var/lib/xenstored/tdb (

tdb represents tree database).
226 z 366 2017-05-27 13:21

XenStore database content is represented by a virtual file system similar to /proc

(for more information on /proc , see Book “System Analysis and Tuning Guide”, Chap-
ter 2 “System Monitoring Utilities”, Section 2.6 “The /proc File System”). The tree has
three main paths: /vm , /local/domain , and /tool .
/vm - stores information about the VM Guest configuration.
/local/domain - stores information about VM Guest on the local node.
/tool - stores general information about various tools.
Tip
Each VM Guest has two different ID numbers. The universal unique identifier
(UUID) remains the same even if the VM Guest is migrated to another machine.
The domain identifier (DOMID) is an identification number that represents a par-
ticular running instance. It typically changes when the VM Guest is migrated to an-
other machine.
24.2.1 XenStore Commands

The file system structure of the XenStore database can be operated with the follow-
ing commands:
xenstore-ls
Displays the full dump of the XenStore database.
xenstore-read path_to_xenstore_entry
Displays the value of the specified XenStore entry.
xenstore-exists xenstore_path
Reports whether the specified XenStore path exists.
xenstore-list xenstore_path
Displays all the children entries of the specified XenStore path.
xenstore-write path_to_xenstore_entry
Updates the value of the specified XenStore entry.
xenstore-rm xenstore_path
Removes the specified XenStore entry or directory.
227 z 366 2017-05-27 13:21

xenstore-chmod xenstore_path mode

Updates the read/write permission on the specified XenStore path.
xenstore-control
Sends a command to the xenstored back-end, such as triggering an integrity
check.
24.2.2 /vm
The /vm path is indexed by the UUID of each VM Guest, and stores configuration in-
formation such as the number of virtual CPUs and the amount of allocated memory.
There is a /vm/<uuid> directory for each VM Guest. To list the directory content,
use xenstore-list .
# xenstore-list /vm
00000000-0000-0000-0000-000000000000
9b30841b-43bc-2af9-2ed3-5a649f466d79-1
The first line of the output belongs to Dom0, and the second one to a running VM
Guest. The following command lists all the entries related to the VM Guest:
# xenstore-list /vm/9b30841b-43bc-2af9-2ed3-5a649f466d79-1
image
rtc
device
pool_name
shadow_memory
uuid
on_reboot
start_time
on_poweroff
bootloader_args
on_crash
vcpus
vcpu_avail
bootloader
name
To read a value of an entry, for example the number of virtual CPUs dedicated to the
VM Guest, use xenstore-read :
# xenstore-read /vm/9b30841b-43bc-2af9-2ed3-5a649f466d79-1/vcpus
1
A list of selected /vm/<uuid> entries follows:
uuid
UUID of the VM Guest. It does not change during the migration process.
228 z 366 2017-05-27 13:21

on_reboot
Specifies whether to destroy or restart the VM Guest in response to a reboot re-
quest.
on_poweroff
Specifies whether to destroy or restart the VM Guest in response to a halt re-
quest.
on_crash
Specifies whether to destroy or restart the VM Guest in response to a crash.
vcpus
Number of virtual CPUs allocated to the VM Guest.
vcpu_avail
Bitmask of active virtual CPUs for the VM Guest. The bitmask has several bits
equal to the value of vcpus , with a bit set for each online virtual CPU.
name
The name of the VM Guest.
Regular VM Guests (not Dom0) use the /vm/<uuid>/image path:
# xenstore-list /vm/9b30841b-43bc-2af9-2ed3-5a649f466d79-1/image
ostype
kernel
cmdline
ramdisk
dmargs
device-model
display
An explanation of the used entries follows:
ostype
The OS type of the VM Guest.
kernel
The path on Dom0 to the kernel for the VM Guest.
cmdline
The kernel command line for the VM Guest used when booting.
ramdisk
The path on Dom0 to the RAM disk for the VM Guest.
229 z 366 2017-05-27 13:21

dmargs
Shows arguments passed to the QEMU process. If you look at the QEMU process
with ps , you should see the same arguments as in
/vm/<uuid>/image/dmargs .
24.2.3 /local/domain/<domid>
This path is indexed by the running domain (VM Guest) ID, and contains information
about the running VM Guest. Remember that the domain ID changes during VM
Guest migration. The following entries are available:
vm
The path of the /vm directory for this VM Guest.
on_reboot, on_poweroff, on_crash, name

See identical options in Section 24.2.2, “ /vm ”
domid
Domain identifier for the VM Guest.
cpu
The current CPU to which the VM Guest is pinned.
cpu_weight
The weight assigned to the VM Guest for scheduling purposes. Higher weights
use the physical CPUs more often.
Apart from the individual entries described above, there are also several subdirecto-
ries under /local/domain/<domid> , containing specific entries. To see all entries
available, refer to XenStore Reference↗.
/local/domain/<domid>/memory
Contains memory information. /local/domain/<domid>/memory/target
contains target memory size for the VM Guest (in kilobytes).
/local/domain/<domid>/console
Contains information about a console used by the VM Guest.
/local/domain/<domid>/backend
Contains information about all back-end devices used by the VM Guest. The
path has subdirectories of its own.
230 z 366 2017-05-27 13:21

/local/domain/<domid>/device
Contains information about the front-end devices for the VM Guest.
/local/domain/<domid>/device-misc
Contains miscellaneous information about devices.
/local/domain/<domid>/store
Contains information about the VM Guest's store.
25 Xen as a High-Availability
Virtualization Host
Setting up two Xen hosts as a failover system has several advantages compared to a
setup where every server runs on dedicated hardware.
Failure of a single server does not cause major interruption of the service.
A single big machine is normally way cheaper than multiple smaller ma-
chines.
Adding new servers as needed is a trivial task.
The utilization of the server is improved, which has positive effects on the
power consumption of the system.
The setup of migration for Xen hosts is described in Section 23.3, “Migrating Xen VM
Guest Systems”. In the following, several typical scenarios are described.
231 z 366 2017-05-27 13:21


Xen can directly provide several remote block devices to the respective Xen guest
systems. These include iSCSI, NPIV, and NBD. All of these may be used to do live mi-
grations. When a storage system is already in place, first try to use the same device
type you already used in the network.
If the storage system cannot be used directly but provides a possibility to offer the
needed space over NFS, it is also possible to create image files on NFS. If the NFS file
system is available on all Xen host systems, this method also allows live migrations
of Xen guests.
When setting up a new system, one of the main considerations is whether a dedi-
cated storage area network should be implemented. The following possibilities are
available:
TABLE 25.1: XEN REMOTE STORAGE
Com-
Method Comments
plexity
Note that all block device traffic goes over the same Ether-
Ethernet low net interface as the network traffic. This may be limiting
the performance of the guest.
Running the storage traffic over a dedicated Ethernet in-
Ethernet terface may eliminate a bottleneck on the server side.
dedicated to medium However, planning your own network with your own IP
storage. address range and possibly a VLAN dedicated to storage
requires numerous considerations.
NPIV is a method to virtualize Fibre channel connections.
This is available with adapters that support a data rate of
NPIV high
at least 4 Gbit/s and allows the setup of complex storage
systems.
Typically, a 1 Gbit/s Ethernet device can fully use a typical hard disk or storage sys-
tem. When using very fast storage systems, such an Ethernet device will probably
limit the speed of the system.
232 z 366 2017-05-27 13:21

For space or budget reasons, it may be necessary to rely on storage that is local to
the Xen host systems. To still maintain the possibility of live migrations, it is neces-
sary to build block devices that are mirrored to both Xen hosts. The software that al-
lows this is called Distributed Replicated Block Device (DRBD).
If a system that uses DRBD to mirror the block devices or files between two Xen
hosts should be set up, both hosts should use the identical hardware. If one of the
hosts has slower hard disks, both hosts will suffer from this limitation.
During the setup, each of the required block devices should use its own DRBD de-
vice. The setup of such a system is quite a complex task.

When using several guest systems that need to communicate between each other, it
is possible to do this over the regular interface. However, for security reasons it may
be advisable to create a bridge that is only connected to guest systems.
In an HA environment that also should support live migrations, such a private

bridge must be connected to the other Xen hosts. This is possible by using dedicated
physical Ethernet devices and a dedicated network.
A different implementation method is using VLAN interfaces. In that case, all the traf-
fic goes over the regular Ethernet interface. However, the VLAN interface does not
get the regular traffic, because only the VLAN packets that are tagged for the correct
VLAN are forwarded.
For more information about the setup of a VLAN interface see Section 13.2.3, “Using
VLAN Interfaces”.
Part V Managing Virtual Machines

with QEMU
26 QEMU Overview
233 z 366 2017-05-27 13:21

QEMU is a fast, cross-platform open source machine emulator which can emu-
late a huge number of hardware architectures for you. QEMU lets you run a
complete unmodified operating system (VM Guest) on top of your existing sys-
tem (VM Host Server).

This section documents how to set up and use SUSE Linux Enterprise Server 12
SP2 as a QEMU-KVM based virtual machine host.
The libvirt -based tools such as virt-manager and virt-install offer
convenient interfaces to set up and manage virtual machines. They act as a kind
of wrapper for the qemu-system-ARCH command. However, it is also possible
to use qemu-system-ARCH directly without using libvirt -based tools.
29 Running Virtual Machines with qemu-system-ARCH

Once you have a virtual disk image ready (for more information on disk images,
see Section 28.2, “Managing Disk Images with qemu-img”), it is time to start the
related virtual machine. Section 28.1, “Basic Installation with qemu-sys-
tem-ARCH” introduced simple commands to install and run a VM Guest. …
30 Virtual Machine Administration Using QEMU Monitor

When QEMU is running, a monitor console is provided for performing interac-
tion with the user. Using the commands available in the monitor console, it is
possible to inspect the running operating system, change removable media, take
screenshots or audio grabs and control other aspects of the virtual …
26 QEMU Overview
QEMU is a fast, cross-platform open source machine emulator which can emulate a
huge number of hardware architectures for you. QEMU lets you run a complete un-
modified operating system (VM Guest) on top of your existing system (VM Host
Server).
234 z 366 2017-05-27 13:21

You can also use QEMU for debugging purposes—you can easily stop your running
virtual machine, inspect its state and save and restore it later.
QEMU consists of the following parts:
processor emulator (x86, s390x, PowerPC, Sparc)
emulated devices (graphic card, network card, hard disks, mice)
generic devices used to connect the emulated devices to the related host de-
vices
descriptions of the emulated machines (PC, Power Mac)
debugger
user interface used to interact with the emulator
QEMU is central to KVM and Xen Virtualization, where it provides the general ma-
chine emulation. Xen's usage of QEMU is somewhat hidden from the user, while
KVM's usage exposes most QEMU features transparently. If the VM Guest hardware
architecture is the same as the VM Host Server's architecture, QEMU can take advan-
tage of the KVM acceleration (SUSE only supports QEMU with the KVM acceleration
loaded).
Apart from providing a core virtualization infrastructure and processor-specific driv-

ers, QEMU also provides an architecture-specific user space program for managing
VM Guests. Depending on the architecture this program is one of:
qemu-system-i386
qemu-system-s390x
qemu-system-x86_64
In the following this command is called qemu-system- ARCH ; in examples the

qemu-system-x86_64 command is used.

235 z 366 2017-05-27 13:21

This section documents how to set up and use SUSE Linux Enterprise Server 12 SP2
as a QEMU-KVM based virtual machine host.
Tip: Resources
In general, the virtual guest system needs the same hardware resources as SUSE
Linux Enterprise Server installed on a physical machine. The more guests you
plan to run on the host system, the more hardware resources—CPU, disk, mem-
ory, and network—you need to add.

To run KVM, your CPU must support virtualization, and virtualization needs to be
enabled in BIOS. The file /proc/cpuinfo includes information about your CPU
features.
To find out whether your system supports virtualization, see Section 7.3, “KVM Hard-
ware Requirements”.

The KVM host requires several packages to be installed. To install all necessary pack-
ages, do the following:
1. Run YaST › Virtualization › Install Hypervisor and Tools.
236 z 366 2017-05-27 13:21

FIGURE 27.1: INSTALLING THE KVM HYPERVISOR AND TOOLS
2. Select KVM server and preferably also KVM tools, and confirm with Accept.
3. During the installation process, you can optionally let YaST create a Network
Bridge for you automatically. If you do not plan to dedicate an additional
physical network card to your virtual guests, network bridge is a standard
way to connect the guest machines to the network.
FIGURE 27.2: NETWORK BRIDGE
4. After all the required packages are installed (and new network setup acti-
vated), try to load the KVM kernel module relevant for your CPU type—
kvm-intel or kvm-amd :
root # modprobe kvm-intel
Check if the module is loaded into memory:
237 z 366 2017-05-27 13:21

tux > lsmod | grep kvm

kvm_intel 64835 6
kvm 411041 1 kvm_intel
Now the KVM host is ready to serve KVM VM Guests. For more information,
see Chapter 29, Running Virtual Machines with qemu-system-ARCH.

You can improve the performance of KVM-based VM Guests by letting them fully use
specific features of the VM Host Server's hardware (paravirtualization). This section
introduces techniques to make the guests access the physical host's hardware di-
rectly—without the emulation layer—to make the most use of it.
Tip
Examples included in this section assume basic knowledge of the

qemu-system- ARCH command line options. For more information, see Chap-
ter 29, Running Virtual Machines with qemu-system-ARCH.
27.3.1 Using the Host Storage with virtio-scsi

virtio-scsi is an advanced storage stack for KVM. It replaces the former
virtio-blk stack for SCSI devices pass-through. It has several advantages over
virtio-blk :
Improved scalability
KVM guests have a limited number of PCI controllers, which results in a limited
number of possibly attached devices. virtio-scsi solves this limitation by
grouping multiple storage devices on a single controller. Each device on a
virtio-scsi controller is represented as a logical unit, or LUN.
Standard command set

virtio-blk uses a small set of commands that need to be known to both the
virtio-blk driver and the virtual machine monitor, and so introducing a new
command requires updating both the driver and the monitor.
238 z 366 2017-05-27 13:21

By comparison, virtio-scsi does not define commands, but rather a trans-

port protocol for these commands following the industry-standard SCSI specifi-
cation. This approach is shared with other technologies, such as Fibre Channel,
ATAPI, and USB devices.
Device naming
virtio-blk devices are presented inside the guest as /dev/vd X , which is
different from device names in physical systems and may cause migration prob-
lems.
virtio-scsi keeps the device names identical to those on physical systems,
making the virtual machines easily relocatable.
SCSI device pass-through

For virtual disks backed by a whole LUN on the host, it is preferable for the guest
to send SCSI commands directly to the LUN (pass-through). This is limited in
virtio-blk , as guests need to use the virtio-blk protocol instead of SCSI com-
mand pass-through, and, moreover, it is not available for Windows guests.
virtio-scsi natively removes these limitations.
27.3.1.1 virtio-scsi Usage

KVM supports the SCSI pass-through feature with the virtio-scsi-pci device:
qemu-system-x86_64 [...] \
-device virtio-scsi-pci,id=scsi
27.3.2 Accelerated Networking with vhost-net

The vhost-net module is used to accelerate KVM's paravirtualized network driv-
ers. It provides better latency and greater network throughput. Use the vhost-net
driver by starting the guest with the following example command line:
-netdev tap,id=guest0,vhost=on,script=no \
-net nic,model=virtio,netdev=guest0,macaddr=00:16:35:AF:94:4B
Note that guest0 is an identification string of the vhost-driven device.
27.3.3 Scaling Network Performance with Multiqueue

virtio-net
239 z 366 2017-05-27 13:21

As the number of virtual CPUs increases in VM Guests, QEMU offers a way of im-
proving the network performance using multiqueue. Multiqueue virtio-net scales the
network performance by allowing VM Guest virtual CPUs to transfer packets in par-
allel. Multiqueue support is required on both the VM Host Server and VM Guest
sides.
Tip: Performance Benefit
The multiqueue virtio-net solution is most beneficial in the following cases:
Network traffic packets are large.
VM Guest has many connections active at the same time, mainly between
the guest systems, or between the guest and the host, or between the
guest and an external system.
The number of active queues is equal to the number of virtual CPUs in the
VM Guest.
Note
While multiqueue virtio-net increases the total network throughput, it increases

CPU consumption as it uses of the virtual CPU's power.
PROCEDURE 27.1: HOW TO ENABLE MULTIQUEUE VIRTIO-NET
The following procedure lists important steps to enable the multiqueue feature
with qemu-system-ARCH . It assumes that a tap network device with multi-
queue capability (supported since kernel version 3.8) is set up on the VM Host
Server.
1. In qemu-system-ARCH , enable multiqueue for the tap device:
-netdev tap,vhost=on,queues=N
where N stands for the number of queue pairs.
2. In qemu-system-ARCH , enable multiqueue and specify MSI-X (Message Sig-

naled Interrupt) vectors for the virtio-net-pci device:
-device virtio-net-pci,mq=on,vectors=2*N+2
240 z 366 2017-05-27 13:21

where the formula for the number of MSI-X vectors results from: N vectors
for TX (transmit) queues, N for RX (receive) queues, one for configuration
purposes, and one for possible VQ (vector quantization) control.
3. In VM Guest, enable multiqueue on the relevant network interface ( eth0 in

this example):
ethtool -L eth0 combined 2*N
The resulting qemu-system-ARCH command line will look similar to the following
example:
qemu-system-x86_64 [...] -netdev tap,id=guest0,queues=4,vhost=on \

-device virtio-net-pci,netdev=guest0,mq=on,vectors=10
Note that the id of the network device ( guest0 ) needs to be identical for both
options.
Inside the running VM Guest, specify the following command as root :
ethtool -L eth0 combined 8
Now the guest system networking uses the multiqueue support from the
qemu-system-ARCH hypervisor.
27.3.4 VFIO: Secure Direct Access to Devices

Directly assigning a PCI device to a VM Guest (PCI pass-through) avoids performance
issues caused by avoiding any emulation in performance-critical paths. VFIO re-
places the traditional KVM PCI Pass-Through device assignment. A prerequisite for
this feature is a VM Host Server configuration described in Important: Requirements
for VFIO and SR-IOV.
To be able to assign a PCI device via VFIO to a VM Guest, you need to find out which
IOMMU Group it belongs to. The IOMMU (input/output memory management unit
that connects a direct memory access-capable I/O bus to the main memory) API sup-
ports the notion of groups. A group is a set of devices that can be isolated from all
other devices in the system. Groups are therefore the unit of ownership used by
VFIO.
PROCEDURE 27.2: ASSIGNING A PCI DEVICE TO A VM GUEST VIA VFIO
1. Identify the host PCI device to assign to the guest.
tux > sudo lspci -nn

[...]
241 z 366 2017-05-27 13:21

00:10.0 Ethernet controller [0200]: Intel Corporation 82576 \

Virtual Function [8086:10ca] (rev 01)
[...]
Note down the device ID ( 00:10.0 in this case) and the vendor ID (
8086:10ca ).
2. Find the IOMMU group of this device:
tux > sudo readlink /sys/bus/pci/devices/0000\:00\:10.0/iommu_group

../../../kernel/iommu_groups/20
The IOMMU group for this device is 20 . Now you can check the devices be-
longing to the same IOMMU group:
ls -l /sys/bus/pci/devices/0000:01:10.0/iommu_group/devices/0000:01:10.0
[...] 0000:00:1e.0 -> ../../../../devices/pci0000:00/0000:00:1e.0
[...] 0000:01:10.0 -> ../../../../devices/pci0000:00/0000:00:1e.0/0000:01:10.0
[...] 0000:01:10.1 -> ../../../../devices/pci0000:00/0000:00:1e.0/0000:01:10.1
3. Unbind the device from the device driver:
sudo echo "0000:01:10.0" > /sys/bus/pci/devices/0000\:01\:10.0/driver/unbind
4. Bind the device to the vfio-pci driver using the vendor ID from step 1:
sudo echo "8086 153a" > /sys/bus/pci/drivers/vfio-pci/new_id
A new device /dev/vfio/ IOMMU_GROUP will be created as a result,

/dev/vfio/20 in this case.
5. Change the ownership of the newly created device:
chown qemu.qemu /dev/vfio/DEVICE
6. Now run the VM Guest with the PCI device assigned.
qemu-system-ARCH [...] -device

vfio-pci,host=00:10.0,id=ID
Important: No Hotplugging
As of SUSE Linux Enterprise Server 12 SP2 hotplugging of PCI devices passed to a

VM Guest via VFIO is not supported.
242 z 366 2017-05-27 13:21

You can find more detailed information on the VFIO driver in the
/usr/src/linux/Documentation/vfio.txt file (package kernel-source
needs to be installed).
27.3.5 VirtFS: Sharing Directories between Host and Guests

VM Guests usually run in a separate computing space—they are provided their own
memory range, dedicated CPUs, and file system space. The ability to share parts of
the VM Host Server's file system makes the virtualization environment more flexible
by simplifying mutual data exchange. Network file systems, such as CIFS and NFS,
have been the traditional way of sharing directories. But as they are not specifically
designed for virtualization purposes, they suffer from major performance and fea-
ture issues.
KVM introduces a new optimized method called VirtFS (sometimes called “file system
pass-through”). VirtFS uses a paravirtual file system driver, which avoids converting
the guest application file system operations into block device operations, and then
again into host file system operations.
You typically use VirtFS for the following situations:
To access a shared directory from several guests, or to provide guest-to-guest

file system access.
To replace the virtual disk as the root file system to which the guest's RAM
disk connects during the guest boot process.
To provide storage services to different customers from a single host file sys-
tem in a cloud environment.
27.3.5.1 Implementation
In QEMU, the implementation of VirtFS is simplified by defining two types of devices:
virtio-9p-pci device which transports protocol messages and data be-

tween the host and the guest.
fsdev device which defines the export file system properties, such as file
system type and security model.
EXAMPLE 27.1: EXPORTING HOST'S FILE SYSTEM WITH VIRTFS
-fsdev local,id=exp1 1 ,path=/tmp/ 2 ,security_model=mapped 3 \
-device virtio-9p-pci,fsdev=exp1 4 ,mount_tag=v_tmp 5
243 z 366 2017-05-27 13:21

1 Identification of the file system to be exported.

2 File system path on the host to be exported.
3 Security model to be used— mapped keeps the guest file system modes
and permissions isolated from the host, while none invokes a “pass-
through” security model in which permission changes on the guest's files
are reflected on the host as well.
4 The exported file system ID defined before with -fsdev id= .
5 Mount tag used later on the guest to mount the exported file system.
Such an exported file system can be mounted on the guest as follows:
sudo mount -t 9p -o trans=virtio v_tmp /mnt
where v_tmp is the mount tag defined earlier with -device mount_tag= and
/mnt is the mount point where you want to mount the exported file system.
27.3.6 KSM: Sharing Memory Pages between Guests

Kernel Same Page Merging (KSM) is a Linux Kernel feature that merges identical
memory pages from multiple running processes into one memory region. Because
KVM guests run as processes under Linux, KSM provides the memory overcommit
feature to hypervisors for more efficient use of memory. Therefore, if you need to
run multiple virtual machines on a host with limited memory, KSM may be helpful to
you.
KSM stores its status information in the files under the /sys/kernel/mm/ksm di-
rectory:
tux > ls -1 /sys/kernel/mm/ksm

full_scans
merge_across_nodes
pages_shared
pages_sharing
pages_to_scan
pages_unshared
pages_volatile
run
sleep_millisecs
For more information on the meaning of the /sys/kernel/mm/ksm/* files, see

/usr/src/linux/Documentation/vm/ksm.txt (package kernel-source ).
To use KSM, do the following.
1. Although SLES includes KSM support in the kernel, it is disabled by default.

To enable it, run the following command:
244 z 366 2017-05-27 13:21

root # echo 1 > /sys/kernel/mm/ksm/run
2. Now run several VM Guests under KVM and inspect the content of files
pages_sharing and pages_shared , for example:
while [ 1 ]; do cat /sys/kernel/mm/ksm/pages_shared; sleep 1; done

13522
13523
13519
13518
13520
13520
13528
28.1 Basic Installation with qemu-system-ARCH
The libvirt -based tools such as virt-manager and virt-install offer con-
venient interfaces to set up and manage virtual machines. They act as a kind of
wrapper for the qemu-system-ARCH command. However, it is also possible to use
qemu-system-ARCH directly without using libvirt -based tools.
Warning: qemu-system-ARCH and libvirt
Virtual Machines created with qemu-system-ARCH are not "visible" for the
libvirt -based tools.
28.1 Basic Installation with
245 z 366 2017-05-27 13:21

qemu-system-ARCH
In the following example, a virtual machine for a SUSE Linux Enterprise Server 11 in-
stallation is created. For detailed information on the commands, refer to the respec-
tive man pages.
If you do not already have an image of a system that you want to run in a virtualized
environment, you need to create one from the installation media. In such case, you
need to prepare a hard disk image, and obtain an image of the installation media or
the media itself.
Create a hard disk with qemu-img .
qemu-img create 1 -f raw 2 /images/sles/hda 3 8G 4
1 The subcommand create tells qemu-img to create a new image.

2 Specify the disk's format with the -f parameter.
3 The full path to the image file.
4 The size of the image—8 GB in this case. The image is created as a Sparse im-
age file file that grows when the disk is filled with data. The specified size de-
fines the maximum size to which the image file can grow.
After at least one hard disk image is created, you can set up a virtual machine with
qemu-system-ARCH that will boot into the installation system:
root # qemu-system-x86_64 -name "sles" 1 -machine accel=kvm -M pc 2 -m 768

-smp 2 4 -boot d 5 \
-drive file=/images/sles/hda,if=virtio,index=0,media=disk,format=raw 6 \
-drive file=/isos/SLES-11-SP3-DVD-x86_64-GM-DVD1.iso,index=1,media=cdrom 7
-net nic,model=virtio,macaddr=52:54:00:05:11:11 8 -net user \
-vga cirrus 9 -balloon virtio 10
246 z 366 2017-05-27 13:21

1 Name of the virtual machine that will be displayed in the window caption and
be used for the VNC server. This name must be unique.
2 Specifies the machine type. Use qemu-system-ARCH -M ? to display a list of
valid parameters. pc is the default Standard PC.
3 Maximum amount of memory for the virtual machine.
4 Defines an SMP system with two processors.
5 Specifies the boot order. Valid values are a , b (floppy 1 and 2), c (first hard
disk), d (first CD-ROM), or n to p (Ether-boot from network adapter 1-3).
Defaults to c .
6 Defines the first ( index=0 ) hard disk. It will be accessed as a paravirtualized (
if=virtio ) drive in raw format.
7 The second ( index=1 ) image drive will act as a CD-ROM.
8 Defines a paravirtualized ( model=virtio ) network adapter with the MAC ad-
dress 52:54:00:05:11:11 . Be sure to specify a unique MAC address, other-
wise a network conflict may occur.
9 Specifies the graphic card. If you specify none , the graphic card will be dis-
abled.
10 Defines the paravirtualized balloon device that allows to dynamically change
the amount of memory (up to the maximum value specified with the parame-
ter -m ).
After the installation of the guest operating system finishes, you can start the related
virtual machine without the need to specify the CD-ROM device:
root # qemu-system-x86_64 -name "sles" -machine type=pc,accel=kvm -m 768 \

-smp 2 -boot c \
-drive file=/images/sles/hda,if=virtio,index=0,media=disk,format=raw \
-net nic,model=virtio,macaddr=52:54:00:05:11:11 \
-vga cirrus -balloon virtio

In the previous section (see Section 28.1, “Basic Installation with
qemu-system-ARCH ”), we used the qemu-img command to create an image of a
hard disk. You can, however, use qemu-img for general disk image manipulation.
This section introduces qemu-img subcommands to help manage the disk images
flexibly.
28.2.1 General Information on qemu-img Invocation
247 z 366 2017-05-27 13:21

qemu-img uses subcommands (like zypper does) to do specific tasks. Each sub-
command understands a different set of options. Some options are general and
used by more of these subcommands, while some are unique to the related sub-
command. See the qemu-img manual page ( man 1 qemu-img ) for a list of all sup-
ported options. qemu-img uses the following general syntax:
tux > qemu-img subcommand [options]
and supports the following subcommands:
create
Creates a new disk image on the file system.
check
Checks an existing disk image for errors.
compare
Check if two images have the same content.
map
Dumps the metadata of the image file name and its backing file chain.
amend
Amends the image format specific options for the image file name.
convert
Converts an existing disk image to a new one in a different format.
info
Displays information about the relevant disk image.
snapshot
Manages snapshots of existing disk images.
commit
Applies changes made to an existing disk image.
rebase
Creates a new base image based on an existing image.
resize
Increases or decreases the size of an existing image.
248 z 366 2017-05-27 13:21

28.2.2 Creating, Converting and Checking Disk Images

This section describes how to create disk images, check their condition, convert a
disk image from one format to another, and get detailed information about a partic-
ular disk image.
28.2.2.1 qemu-img create

Use qemu-img create to create a new disk image for your VM Guest operating
system. The command uses the following syntax:
tux > qemu-img create -f fmt 1 -o options 2 fname 3 size 4
1 The format of the target image. Supported formats are qed , qcow2 , and
raw .
2 Some image formats support additional options to be passed on the com-
mand line. You can specify them here with the -o option. The raw image for-
mat supports only the size option, so it is possible to insert -o size=8G
instead of adding the size option at the end of the command.
3 Path to the target disk image to be created.
4 Size of the target disk image (if not already specified with the
-o size=<image_size> option. Optional suffixes for the image size are K
(kilobyte), M (megabyte), G (gigabyte), or T (terabyte).
To create a new disk image sles.raw in the directory /images growing up to a

maximum size of 4 GB, run the following command:
tux > qemu-img create -f raw -o size=4G /images/sles.raw

Formatting '/images/sles.raw', fmt=raw size=4294967296
tux > ls -l /images/sles.raw

-rw-r--r-- 1 tux users 4294967296 Nov 15 15:56 /images/sles.raw
tux > qemu-img info /images/sles.raw

image: /images/sles11.raw
file format: raw
virtual size: 4.0G (4294967296 bytes)
disk size: 0
As you can see, the virtual size of the newly created image is 4 GB, but the actual re-
ported disk size is 0 as no data has been written to the image yet.
249 z 366 2017-05-27 13:21

Tip: VM Guest Images on the Btrfs File System
If you need to create a disk image on the Btrfs file system, you can use
nocow=on to reduce the performance overhead created by the copy-on-write
feature of Btrfs:
tux > qemu-img create -o nocow=on test.img 8G
If you, however, want to use copy-on-write (for example for creating snapshots
or sharing them across virtual machines), then leave the command line without
the nocow option.
28.2.2.2 qemu-img convert

Use qemu-img convert to convert disk images to another format. To get a com-
plete list of image formats supported by QEMU, run qemu-img -h and look at the
last line of the output. The command uses the following syntax:
tux > qemu-img convert -c 1 -f fmt 2 -O out_fmt 3 -o options 4 fname
1 Applies compression on the target disk image. Only qcow and qcow2 for-
mats support compression.
2 The format of the source disk image. It is usually autodetected and can there-
fore be omitted.
3 The format of the target disk image.
4 Specify additional options relevant for the target image format. Use -o ? to
view the list of options supported by the target image format.
5 Path to the source disk image to be converted.
6 Path to the converted target disk image.
tux > qemu-img convert -O vmdk /images/sles.raw \

/images/sles.vmdk
tux > ls -l /images/

-rw-r--r-- 1 tux users 4294967296 16. lis 10.50 sles.raw
-rw-r--r-- 1 tux users 2574450688 16. lis 14.18 sles.vmdk
To see a list of options relevant for the selected target image format, run the follow-
ing command (replace vmdk with your image format):
tux > qemu-img convert -O vmdk /images/sles.raw \

/images/sles.vmdk -o ?
Supported options:
size Virtual disk size
250 z 366 2017-05-27 13:21

backing_file File name of a base image

compat6 VMDK version 6 image
subformat VMDK flat extent format, can be one of {monolithicSparse \
(default) | monolithicFlat | twoGbMaxExtentSparse | twoGbMaxExtentFlat}
scsi SCSI image
28.2.2.3 qemu-img check

Use qemu-img check to check the existing disk image for errors. Not all disk image
formats support this feature. The command uses the following syntax:
tux > qemu-img check -f fmt 1 fname 2
1 The format of the source disk image. It is usually autodetected and can there-
fore be omitted.
2 Path to the source disk image to be checked.
If no error is found, the command returns no output. Otherwise, the type and num-
ber of errors found is shown.
tux > qemu-img check -f qcow2 /images/sles.qcow2

ERROR: invalid cluster offset=0x2af0000
[...]
ERROR: invalid cluster offset=0x34ab0000
378 errors were found on the image.
28.2.2.4 Increasing the Size of an Existing Disk Image

When creating a new image, you must specify its maximum size before the image is
created (see Section 28.2.2.1, “qemu-img create”). After you have installed the VM
Guest and have been using it for some time, the initial size of the image may no
longer be sufficient and you may need to add more space to it.
To increase the size of an existing disk image by 2 gigabytes, use:
tux > qemu-img resize /images/sles.raw +2GB
Note
You can resize the disk image using the formats raw , qcow2 and qed . To resize
an image in another format, convert it to a supported format with
qemu-img convert first.
251 z 366 2017-05-27 13:21

The image now contains an empty space of 2 GB after the final partition. You can re-
size the existing partitions or add new ones.
FIGURE 28.1: NEW 2 GB PARTITION IN GUEST YAST PARTITIONER
28.2.2.5 Advanced Options for the qcow2 File Format

qcow2 is the main disk image format used by QEMU. Its size grows on demand, and
the disk space is only allocated when it is actually needed by the virtual machine.
A qcow2 formatted file is organized in units of constant size. These units are called
clusters. Viewed from the guest side, the virtual disk is also divided into clusters of
the same size. QEMU defaults to 64 kB clusters, but you can specify a different value
when creating a new image:
tux > qemu-img create -f qcow2 -o cluster_size=128K virt_disk.qcow2 4G
252 z 366 2017-05-27 13:21

A qcow2 image contains a set of tables organized in two levels that are called the L1
and L2 tables. There is just one L1 table per disk image, while there can be many L2
tables depending on how big the image is.
To read or write data to the virtual disk, QEMU needs to read its corresponding L2
table to find out the relevant data location. Because reading the table for each I/O
operation consumes system resources, QEMU keeps a cache of L2 tables in memory
to speed up disk access.
28.2.2.5.1 Choosing the Right Cache Size

The cache size relates to the amount of allocated space. L2 cache can map the fol-
lowing amount of virtual disk:
disk_size = l2_cache_size * cluster_size / 8
With the default 64 kB of cluster size, that is
disk_size = l2_cache_size * 8192
Therefore, to have a cache that maps n gigabytes of disk space with the default
cluster size, you need
l2_cache_size = disk_size_GB * 131072
QEMU uses 1 MB (1048576 bytes) of L2 cache by default. Following the above for-
mulas, 1 MB of L2 cache covers 8 GB (1048576 / 131072) of virtual disk. This means
that the performance is fine with the default L2 cache size if your virtual disk size is
up to 8 GB. For larger disks, you can speed up the disk access by increasing the L2
cache size.
28.2.2.5.2 Configuring the Cache Size

You can use the -drive option on the QEMU command line to specify the cache
sizes. Alternatively when communicating via QMP, use the blockdev-add com-
mand. For more information on QMP, see Section 30.11, “QMP - QEMU Machine
Protocol”.
The following options configure the cache size for the virtual guest:
l2-cache-size
The maximum size of the L2 table cache.
refcount-cache-size
The maximum size of the refcount block cache. For more information on ref-
count, see https://github.com/qemu/qemu/blob/master/docs/specs/qcow2.txt↗
.
253 z 366 2017-05-27 13:21

cache-size
The maximum size of both caches combined.
When specifying values for the options above, be aware of the following:
The size of both the L2 and refcount block caches needs to be a multiple of
the cluster size.
If you only set one of the options, QEMU will automatically adjust the other
options so that the L2 cache is 4 times bigger than the refcount cache.
The refcount cache is used much less often than the L2 cache, therefore you can
keep it relatively small:
root # qemu-system-ARCH [...] \

-drive file=disk_image.qcow2,l2-cache-size=4194304,refcount-cache-size=262144
28.2.2.5.3 Reducing the Memory Usage

The larger the cache, the more memory it consumes. There is a separate L2 cache for
each qcow2 file. When using a lot of big disk images, you will probably need a con-
siderably large amount of memory. Memory consumption is even worse if you add
backing files (Section 28.2.4, “Manipulate Disk Images Effectively”) and snapshots
(see Section 28.2.3, “Managing Snapshots of Virtual Machines with qemu-img”) to
the guest's setup chain.
That is why QEMU introduced the cache-clean-interval setting. It defines an in-

terval in seconds after which all cache entries that have not been accessed are re-
moved from memory.
The following example removes all unused cache entries every 10 minutes:
root # qemu-system-ARCH [...] -drive file=hd.qcow2,cache-clean-interval=600
If this option is not set, the default value is 0 and it disables this feature.
28.2.3 Managing Snapshots of Virtual Machines with

qemu-img
Virtual Machine snapshots are snapshots of the complete environment in which a
VM Guest is running. The snapshot includes the state of the processor (CPU), mem-
ory (RAM), devices, and all writable disks.
254 z 366 2017-05-27 13:21

Snapshots are helpful when you need to save your virtual machine in a particular
state. For example, after you configured network services on a virtualized server and
want to quickly start the virtual machine in the same state you last saved it. Or you
can create a snapshot after the virtual machine has been powered off to create a
backup state before you try something experimental and possibly make VM Guest
unstable. This section introduces the latter case, while the former is described in
Chapter 30, Virtual Machine Administration Using QEMU Monitor.
To use snapshots, your VM Guest must contain at least one writable hard disk image
in qcow2 format. This device is usually the first virtual hard disk.
Virtual Machine snapshots are created with the savevm command in the interactive
QEMU monitor. To make identifying a particular snapshot easier, you can assign it a
tag. For more information on QEMU monitor, see Chapter 30, Virtual Machine Ad-
ministration Using QEMU Monitor.
Once your qcow2 disk image contains saved snapshots, you can inspect them with
the qemu-img snapshot command.
Warning: Shut Down the VM Guest
Do not create or delete virtual machine snapshots with the

qemu-img snapshot command while the virtual machine is running. Other-
wise, you may damage the disk image with the state of the virtual machine saved.
28.2.3.1 Listing Existing Snapshots

Use qemu-img snapshot -l disk_image to view a list of all existing snapshots
saved in the disk_image image. You can get the list even while the VM Guest is
running.
tux > qemu-img snapshot -l /images/sles.qcow2

Snapshot list:
ID 1 TAG 2 VM SIZE 3 DATE 4 VM CLOCK
1 booting 4.4M 2013-11-22 10:51:10 00:00:20.476
2 booted 184M 2013-11-22 10:53:03 00:02:05.394
3 logged_in 273M 2013-11-22 11:00:25 00:04:34.843
4 ff_and_term_running 372M 2013-11-22 11:12:27 00:08:44.965
255 z 366 2017-05-27 13:21

1 Unique identification number of the snapshot. Usually auto-incremented.

2 Unique description string of the snapshot. It is meant as a human-readable
version of the ID.
3 The disk space occupied by the snapshot. Note that the more memory is con-
sumed by running applications, the bigger the snapshot is.
4 Time and date the snapshot was created.
5 The current state of the virtual machine's clock.
28.2.3.2 Creating Snapshots of a Powered-Off Virtual Machine

Use qemu-img snapshot -c snapshot_title disk_image to create a snap-
shot of the current state of a virtual machine that was previously powered off.
tux > qemu-img snapshot -c backup_snapshot /images/sles.qcow2
tux > qemu-img snapshot -l /images/sles.qcow2

Snapshot list:
ID TAG VM SIZE DATE VM CLOCK
1 booting 4.4M 2013-11-22 10:51:10 00:00:20.476
2 booted 184M 2013-11-22 10:53:03 00:02:05.394
3 logged_in 273M 2013-11-22 11:00:25 00:04:34.843
5 backup_snapshot 0 2013-11-22 14:14:00 00:00:00.000
If something breaks in your VM Guest and you need to restore the state of the saved
snapshot (ID 5 in our example), power off your VM Guest and execute the following
command:
tux > qemu-img snapshot -a 5 /images/sles.qcow2
The next time you run the virtual machine with qemu-system-ARCH , it will be in the
state of snapshot number 5.
Note
The qemu-img snapshot -c command is not related to the savevm command

of QEMU monitor (see Chapter 30, Virtual Machine Administration Using QEMU
Monitor). For example, you cannot apply a snapshot with
qemu-img snapshot -a on a snapshot created with savevm in QEMU's moni-
tor.
28.2.3.3 Deleting Snapshots
256 z 366 2017-05-27 13:21

Use qemu-img snapshot -d snapshot_id disk_image to delete old or un-

needed snapshots of a virtual machine. This saves some disk space inside the
qcow2 disk image as the space occupied by the snapshot data is restored:
tux > qemu-img snapshot -d 2 /images/sles.qcow2
28.2.4 Manipulate Disk Images Effectively

Imagine the following real-life situation: you are a server administrator who runs
and manages several virtualized operating systems. One group of these systems is
based on one specific distribution, while another group (or groups) is based on dif-
ferent versions of the distribution or even on a different (and maybe non-Unix) plat-
form. To make the case even more complex, individual virtual guest systems based
on the same distribution usually differ according to the department and deploy-
ment. A file server typically uses a different setup and services than a Web server
does, while both may still be based on SUSE® Linux Enterprise Server.
With QEMU it is possible to create “base” disk images. You can use them as template
virtual machines. These base images will save you plenty of time because you will
never need to install the same operating system more than once.
28.2.4.1 Base and Derived Images

First, build a disk image as usual and install the target system on it. For more infor-
mation, see Section 28.1, “Basic Installation with qemu-system-ARCH ” and Sec-
tion 28.2.2, “Creating, Converting and Checking Disk Images”. Then build a new im-
age while using the first one as a base image. The base image is also called a backing
file. After your new derived image is built, never boot the base image again, but boot
the derived image instead. Several derived images may depend on one base image
at the same time. Therefore, changing the base image can damage the dependen-
cies. While using your derived image, QEMU writes changes to it and uses the base
image only for reading.
It is a good practice to create a base image from a freshly installed (and, if needed,
registered) operating system with no patches applied and no additional applications
installed or removed. Later on, you can create another base image with the latest
patches applied and based on the original base image.
28.2.4.2 Creating Derived Images
257 z 366 2017-05-27 13:21

Note
While you can use the raw format for base images, you cannot use it for derived
images because the raw format does not support the backing_file option.
Use for example the qcow2 format for the derived images.
For example, /images/sles_base.raw is the base image holding a freshly in-

stalled system.
tux > qemu-img info /images/sles_base.raw

image: /images/sles_base.raw
file format: raw
disk size: 2.4G
The image's reserved size is 4 GB, the actual size is 2.4 GB, and its format is raw .
Create an image derived from the /images/sles_base.raw base image with:
tux > qemu-img create -f qcow2 /images/sles_derived.qcow2 \

-o backing_file=/images/sles_base.raw
Formatting '/images/sles_derived.qcow2', fmt=qcow2 size=4294967296 \
backing_file='/images/sles_base.raw' encryption=off cluster_size=0
Look at the derived image details:
tux > qemu-img info /images/sles_derived.qcow2

image: /images/sles_derived.qcow2
file format: qcow2
disk size: 140K
cluster_size: 65536
backing file: /images/sles_base.raw \
(actual path: /images/sles_base.raw)
Although the reserved size of the derived image is the same as the size of the base
image (4 GB), the actual size is 140 KB only. The reason is that only changes made to
the system inside the derived image are saved. Run the derived virtual machine, reg-
ister it, if needed, and apply the latest patches. Do any other changes in the system
such as removing unneeded or installing new software packages. Then shut the VM
Guest down and examine its details once more:
tux > qemu-img info /images/sles_derived.qcow2

image: /images/sles_derived.qcow2
file format: qcow2
disk size: 1.1G
cluster_size: 65536
backing file: /images/sles_base.raw \
(actual path: /images/sles_base.raw)
258 z 366 2017-05-27 13:21

The disk size value has grown to 1.1 GB, which is the disk space occupied by the
changes on the file system compared to the base image.
28.2.4.3 Rebasing Derived Images

After you have modified the derived image (applied patches, installed specific appli-
cations, changed environment settings, etc.), it reaches the desired state. At that
point, you can merge the original base image and the derived image to create a new
base image.
Your original base image ( /images/sles_base.raw ) holds a freshly installed sys-

tem and can be a template for new modified base images, while the new one can
contain the same system as the first one plus all security and update patches ap-
plied, for example. After you have created this new base image, you can use it as a
template for more specialized derived images as well. The new base image becomes
independent of the original one. The process of creating base images from derived
ones is called rebasing:
tux > qemu-img convert /images/sles_derived.qcow2 \

-O raw /images/sles_base2.raw
This command created the new base image /images/sles_base2.raw using the
raw format.
tux > qemu-img info /images/sles_base2.raw

image: /images/sles11_base2.raw
file format: raw
disk size: 2.8G
The new image is 0.4 gigabytes bigger than the original base image. It uses no back-
ing file, and you can easily create new derived images based upon it. This lets you
create a sophisticated hierarchy of virtual disk images for your organization, saving a
lot of time and work.
28.2.4.4 Mounting an Image on a VM Host Server

It can be useful to mount a virtual disk image under the host system. It is strongly
recommended to read Chapter 17, libguestfs and use dedicated tools to access a vir-
tual machine image. However, if you need to do this manually, follow this guide.
Linux systems can mount an internal partition of a raw disk image using a loop-
back device. The first example procedure is more complex but more illustrative,
while the second one is straightforward:
259 z 366 2017-05-27 13:21

PROCEDURE 28.1: MOUNTING DISK IMAGE BY CALCULATING PARTITION OFFSET
1. Set a loop device on the disk image whose partition you want to mount.
tux > losetup /dev/loop0 /images/sles_base.raw
2. Find the sector size and the starting sector number of the partition you want
to mount.
tux > fdisk -lu /dev/loop0
Disk /dev/loop0: 4294 MB, 4294967296 bytes

255 heads, 63 sectors/track, 522 cylinders, total 8388608 sectors
Units = sectors of 1 * 512 = 512 1 bytes
Disk identifier: 0x000ceca8
Device Boot Start End Blocks Id System

/dev/loop0p1 63 1542239 771088+ 82 Linux swap
/dev/loop0p2 * 1542240 2 8385929 3421845 83 Linux
1 The disk sector size.

2 The starting sector of the partition.
3. Calculate the partition start offset:

sector_size * sector_start = 512 * 1542240 = 789626880
4. Delete the loop and mount the partition inside the disk image with the cal-
culated offset on a prepared directory.
tux > losetup -d /dev/loop0

tux > mount -o loop,offset=789626880 \
/images/sles_base.raw /mnt/sles/
tux > ls -l /mnt/sles/
total 112
drwxr-xr-x 2 root root 4096 Nov 16 10:02 bin
drwxr-xr-x 3 root root 4096 Nov 16 10:27 boot
drwxr-xr-x 5 root root 4096 Nov 16 09:11 dev
[...]
drwxrwxrwt 14 root root 4096 Nov 24 09:50 tmp
drwxr-xr-x 12 root root 4096 Nov 16 09:16 usr
drwxr-xr-x 15 root root 4096 Nov 16 09:22 var
5. Copy one or more files onto the mounted partition and unmount it when
finished.
tux > cp /etc/X11/xorg.conf /mnt/sles/root/tmp

tux > ls -l /mnt/sles/root/tmp
tux > umount /mnt/sles/
260 z 366 2017-05-27 13:21

Warning: Do not Write to Images Currently in Use
Never mount a partition of an image of a running virtual machine in a

read-write mode. This could corrupt the partition and break the whole VM
Guest.
29 Running Virtual Machines with

qemu-system-ARCH
Once you have a virtual disk image ready (for more information on disk images, see
Section 28.2, “Managing Disk Images with qemu-img ”), it is time to start the related
virtual machine. Section 28.1, “Basic Installation with qemu-system-ARCH ” intro-
duced simple commands to install and run a VM Guest. This chapter focuses on a
more detailed explanation of qemu-system-ARCH usage, and shows solutions for
more specific tasks. For a complete list of qemu-system-ARCH 's options, see its
manual page ( man 1 qemu ).

The qemu-system-ARCH command uses the following syntax:
qemu-system-ARCH options 1 disk_img 2
261 z 366 2017-05-27 13:21

1 qemu-system-ARCH understands many options. Most of them define param-

eters of the emulated hardware, while others affect more general emulator
behavior. If you do not supply any options, default values are used, and you
need to supply the path to a disk image to be run.
2 Path to the disk image holding the guest system you want to virtualize.
qemu-system-ARCH supports many image formats. Use qemu-img --help
to list them. If you do not supply the path to a disk image as a separate argu-
ment, you need to use the -drive file= option.

This section introduces general qemu-system-ARCH options and options related to
the basic emulated hardware, such as the virtual machine's processor, memory,
model type, or time processing methods.
-name name_of_guest
Specifies the name of the running guest system. The name is displayed in the
window caption and used for the VNC server.
-boot options
Specifies the order in which the defined drives will be booted. Drives are repre-
sented by letters, where a and b stand for the floppy drives 1 and 2, c stands
for the first hard disk, d stands for the first CD-ROM drive, and n to p stand
for Ether-boot network adapters.
For example, qemu-system-ARCH [...] -boot order=ndc first tries to boot
from network, then from the first CD-ROM drive, and finally from the first hard
disk.
-pidfile fname
Stores the QEMU's process identification number (PID) in a file. This is useful if
you run QEMU from a script.
-nodefaults
By default QEMU creates basic virtual devices even if you do not specify them on
the command line. This option turns this feature off, and you must specify every
single device manually, including graphical and network cards, parallel or serial
ports, or virtual consoles. Even QEMU monitor is not attached by default.
-daemonize
262 z 366 2017-05-27 13:21

“Daemonizes” the QEMU process after it is started. QEMU will detach from the
standard input and standard output after it is ready to receive connections on
any of its devices.
Note: SeaBIOS BIOS Implementation
SeaBIOS is the default BIOS used. You can boot USB devices, any drive (CD-ROM,
Floppy, or a hard disk). It has USB mouse and keyboard support and supports
multiple VGA cards. For more information about SeaBIOS, refer to the SeaBIOS
Website↗.
29.2.1 Basic Virtual Hardware
29.2.1.1 Machine Type

You can specifies the type of the emulated machine. Run
qemu-system-ARCH -M help to view a list of supported machine types.
tux > qemu-system-x86_64 -M help

Supported machines are:
pc Standard PC (i440FX + PIIX, 1996) (alias of pc-i440fx-2.3)
pc-i440fx-2.3 Standard PC (i440FX + PIIX, 1996) (default)
pc-i440fx-2.2 Standard PC (i440FX + PIIX, 1996)
pc-1.3 Standard PC (i440FX + PIIX, 1996)
q35 Standard PC (Q35 + ICH9, 2009) (alias of pc-q35-2.3)
pc-q35-2.3 Standard PC (Q35 + ICH9, 2009)
263 z 366 2017-05-27 13:21

isapc ISA-only PC
none empty machine
xenfv Xen Fully-virtualized PC
xenpv Xen Para-virtualized PC
Note: ISA-PC
The machine type isapc: ISA-only-PC is unsupported.
29.2.1.2 CPU Model

To specify the type of the processor (CPU) model, run qemu-system-ARCH -cpu
MODEL . Use qemu-system-ARCH -cpu help to view a list of supported CPU mod-
els.
x86 qemu64 QEMU Virtual CPU version 2.3.1

x86 phenom AMD Phenom(tm) 9550 Quad-Core Processor
x86 core2duo Intel(R) Core(TM)2 Duo CPU T7700 @ 2.40GHz
x86 kvm64 Common KVM processor
x86 qemu32 QEMU Virtual CPU version 2.3.1
x86 kvm32 Common 32-bit KVM processor
x86 coreduo Genuine Intel(R) CPU T2600 @ 2.16GHz
x86 486
x86 pentium
x86 pentium2
x86 pentium3
x86 athlon QEMU Virtual CPU version 2.3.1
x86 n270 Intel(R) Atom(TM) CPU N270 @ 1.60GHz
x86 Conroe Intel Celeron_4x0 (Conroe/Merom Class Core 2)
x86 Penryn Intel Core 2 Duo P9xxx (Penryn Class Core 2)
x86 Nehalem Intel Core i7 9xx (Nehalem Class Core i7)
x86 Westmere Westmere E56xx/L56xx/X56xx (Nehalem-C)
x86 SandyBridge Intel Xeon E312xx (Sandy Bridge)
x86 IvyBridge Intel Xeon E3-12xx v2 (Ivy Bridge)
x86 Haswell-noTSX Intel Core Processor (Haswell, no TSX)
x86 Haswell Intel Core Processor (Haswell)
x86 Broadwell-noTSX Intel Core Processor (Broadwell, no TSX)
x86 Broadwell Intel Core Processor (Broadwell)
x86 Opteron_G1 AMD Opteron 240 (Gen 1 Class Opteron)
x86 Opteron_G2 AMD Opteron 22xx (Gen 2 Class Opteron)
x86 Opteron_G3 AMD Opteron 23xx (Gen 3 Class Opteron)
x86 Opteron_G4 AMD Opteron 62xx class CPU
x86 Opteron_G5 AMD Opteron 63xx class CPU
x86 host KVM processor with all supported host features (only available in
Recognized CPUID flags:

fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 pn
clflush ds acpi mmx fxsr sse sse2 ss ht tm ia64 pbe
pni|sse3 pclmulqdq|pclmuldq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cid
fma cx16 xtpr pdcm pcid dca sse4.1|sse4_1 sse4.2|sse4_2 x2apic movbe popcnt
tsc-deadline aes xsave osxsave avx f16c rdrand hypervisor
fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f
264 z 366 2017-05-27 13:21

rdseed adx smap avx512pf avx512er avx512cd

syscall nx|xd mmxext fxsr_opt|ffxsr pdpe1gb rdtscp lm|i64 3dnowext 3dnow
lahf_lm cmp_legacy svm extapic cr8legacy abm sse4a misalignsse 3dnowprefetch
osvw ibs xop skinit wdt lwp fma4 tce nodeid_msr tbm topoext perfctr_core
perfctr_nb
invtsc
xstore xstore-en xcrypt xcrypt-en ace2 ace2-en phe phe-en pmm pmm-en
kvmclock kvm_nopiodelay kvm_mmu kvmclock kvm_asyncpf kvm_steal_time
kvm_pv_eoi kvm_pv_unhalt kvmclock-stable-bit
npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists
pause_filter pfthreshold
xsaveopt xsavec xgetbv1 xsaves
CPU flags information can be found at CPUID Wikipedia↗.
29.2.1.3 Other Basics Options

The following is a list of most commonly used options while launching qemu from
command line. To see all options available refer to qemu-doc man page.
-m megabytes
Specifies how many megabytes are used for the virtual RAM size.
-balloon virtio
Specifies a paravirtualized device to dynamically change the amount of virtual
RAM memory assigned to VM Guest. The top limit is the amount of memory
specified with -m .
-smp number_of_cpus
Specifies how many CPUs will be emulated. QEMU supports up to 255 CPUs on
the PC platform (up to 64 with KVM acceleration used). This option also takes
other CPU-related parameters, such as number of sockets, number of cores per
socket, or number of threads per core.
The following is an example of a working qemu-system-ARCH command line:
tux > qemu-system-x86_64 -name "SLES 11 SP3" -M pc-i440fx-2.3 -m 512 \

-machine accel=kvm -cpu kvm64 -smp 2 /images/sles.raw
265 z 366 2017-05-27 13:21

FIGURE 29.1: QEMU WINDOW WITH SLES 11 SP3 AS VM GUEST
-no-acpi
Disables ACPI support.
-S
QEMU starts with CPU stopped. To start CPU, enter c in QEMU monitor. For
more information, see Chapter 30, Virtual Machine Administration Using QEMU
Monitor.
29.2.2 Storing and Reading Configuration of Virtual Devices
-readconfig cfg_file
Instead of entering the devices configuration options on the command line each
time you want to run VM Guest, qemu-system-ARCH can read it from a file that
was either previously saved with -writeconfig or edited manually.
266 z 366 2017-05-27 13:21

-writeconfig cfg_file
Dumps the current virtual machine's devices configuration to a text file. It can be
consequently re-used with the -readconfig option.
tux > qemu-system-x86_64 -name "SLES 11 SP3" -machine accel=kvm -M pc-i440fx-2.3 -m

-smp 2 /images/sles.raw -writeconfig /images/sles.cfg
(exited)
tux > cat /images/sles.cfg
# qemu config file
[drive]
index = "0"
media = "disk"
file = "/images/sles_base.raw"
This way you can effectively manage the configuration of your virtual machines'
devices in a well-arranged way.
29.2.3 Guest Real-Time Clock
-rtc options
Specifies the way the RTC is handled inside a VM Guest. By default, the clock of
the guest is derived from that of the host system. Therefore, it is recommended
that the host system clock is synchronized with an accurate external clock (for
example, via NTP service).
If you need to isolate the VM Guest clock from the host one, specify clock=vm
instead of the default clock=host .
You can also specify the initial time of the VM Guest's clock with the base op-
tion:
qemu-system-x86_64 [...] -rtc clock=vm,base=2010-12-03T01:02:00
Instead of a time stamp, you can specify utc or localtime . The former in-
structs VM Guest to start at the current UTC value (Coordinated Universal Time,
see http://en.wikipedia.org/wiki/UTC↗), while the latter applies the local time
setting.
267 z 366 2017-05-27 13:21

QEMU virtual machines emulate all devices needed to run a VM Guest. QEMU sup-
ports, for example, several types of network cards, block devices (hard and remov-
able drives), USB devices, character devices (serial and parallel ports), or multimedia
devices (graphic and sound cards). This section introduces options to configure vari-
ous types of supported devices.
Tip
If your device, such as -drive , needs a special driver and driver properties to be
set, specify them with the -device option, and identify with drive= subop-
tion. For example:
qemu-system-x86_64 [...] -drive if=none,id=drive0,format=raw \

-device virtio-blk-pci,drive=drive0,scsi=off ...
To get help on available drivers and their properties, use -device ? and
-device driver ,? .
29.3.1 Block Devices

Block devices are vital for virtual machines. In general, these are fixed or removable
storage media usually called drives. One of the connected hard disks typically holds
the guest operating system to be virtualized.
Virtual Machine drives are defined with -drive . This option has many sub-options,
some of which are described in this section. For the complete list, see the manual
page ( man 1 qemu ).
SUB-OPTIONS FOR THE -drive OPTION
file=image_fname
Specifies the path to the disk image that will be used with this drive. If not speci-
fied, an empty (removable) drive is assumed.
if=drive_interface
Specifies the type of interface to which the drive is connected. Currently only
floppy , scsi , ide , or virtio are supported by SUSE. virtio defines a
paravirtualized disk driver. Default is ide .
index=index_of_connector
Specifies the index number of a connector on the disk interface (see the if op-
tion) where the drive is connected. If not specified, the index is automatically in-
cremented.
268 z 366 2017-05-27 13:21

media=type
Specifies the type of media. Can be disk for hard disks, or cdrom for remov-
able CD-ROM drives.
format=img_fmt
Specifies the format of the connected disk image. If not specified, the format is
autodetected. Currently, SUSE supports qcow2 , qed and raw formats.
cache=method
Specifies the caching method for the drive. Possible values are unsafe ,
writethrough , writeback , directsync , or none . To improve perfor-
mance when using the qcow2 image format, choose writeback . none dis-
ables the host page cache and, therefore, is the safest option. Default for image
files is writeback . For more information, see Chapter 15, Disk Cache Modes.
Tip
To simplify defining block devices, QEMU understands several shortcuts which

you may find handy when entering the qemu-system-ARCH command line.
You can use
qemu-system-x86_64 -cdrom /images/cdrom.iso
instead of
qemu-system-x86_64 -drive file=/images/cdrom.iso,index=2,media=cdrom
and
qemu-system-x86_64 -hda /images/imagei1.raw -hdb /images/image2.raw -hdc \

/images/image3.raw -hdd /images/image4.raw
instead of
qemu-system-x86_64 -drive file=/images/image1.raw,index=0,media=disk \

-drive file=/images/image2.raw,index=1,media=disk \
-drive file=/images/image3.raw,index=2,media=disk \
-drive file=/images/image4.raw,index=3,media=disk
269 z 366 2017-05-27 13:21

Tip: Using Host Drives Instead of Images
As an alternative to using disk images (see Section 28.2, “Managing Disk Images
with qemu-img ”) you can also use existing VM Host Server disks, connect them
as drives, and access them from VM Guest. Use the host disk device directly in-
stead of disk image file names.
To access the host CD-ROM drive, use
qemu-system-x86_64 [...] -drive file=/dev/cdrom,media=cdrom
To access the host hard disk, use
qemu-system-x86_64 [...] -drive file=/dev/hdb,media=disk
A host drive used by a VM Guest must not be accessed concurrently by the VM

Host Server or another VM Guest.
29.3.1.1 Freeing Unused Guest Disk Space

A Sparse image file is a type of disk image file that grows in size as the user adds
data to it, taking up only as much disk space as is stored in it. For example, if you
copy 1 GB of data inside the sparse disk image, its size grows by 1 GB. If you then
delete for example 500 MB of the data, the image size does not by default decrease
as expected.
That is why the discard=on option is introduced on the KVM command line. It tells
the hypervisor to automatically free the “holes” after deleting data from the sparse
guest image. Note that this option is valid only for the if=scsi drive interface:
qemu-system-x86_64 [...] -drive file=/path/to/file.img,if=scsi,discard=on
Important: Support Status
if=scsi is not supported. This interface does not map to virtio-scsi, but rather
to the lsi SCSI adapter.
29.3.1.2 virtio-blk-data-plane
270 z 366 2017-05-27 13:21

The virtio-blk-data-plane is a new feature for KVM. It enables a high-performance

code path for I/O requests coming from VM Guests. More specifically, this feature in-
troduces dedicated threads (one per virtual block device) to process I/O requests go-
ing through the virtio-blk driver. It uses the Linux AIO (asynchronous I/O) interface of
the VM Host Server kernel directly—without the need to go through the QEMU block
layer. Therefore, it can sustain very high I/O rates on storage setups.
The virtio-blk-data-plane feature can be enabled or disabled by the

x-data-plane=on|off option on the qemu command line when starting the VM
Guest:
tux > qemu-system-x86_64 [...] -drive if=none,id=drive0,cache=none,aio=native,\

format=raw,file=filename -device virtio-blk-pci,drive=drive0,scsi=off,\
config-wce=off,x-data-plane=on [...]
Currently, virtio-blk-data-plane has the following limitations:
Only the raw image format is supported.
No support for live migration.
Block jobs and hot unplug operations fail with -EBUSY .
I/O throttling limits are ignored.
Only Linux VM Host Servers are supported because of the Linux AIO usage,
but non-Linux VM Guests are supported.
The virtio-blk-data-plane feature is not yet supported in SUSE Linux Enterprise

Server. It is released as a technology preview only.
29.3.1.3 Bio-Based I/O Path for virtio-blk

For better performance of I/O-intensive applications, a new I/O path was introduced
for the virtio-blk interface in kernel version 3.7. This bio-based block device driver
skips the I/O scheduler, and thus shortens the I/O path in guest and has lower la-
tency. It is especially useful for high-speed storage devices, such as SSD disks.
The driver is disabled by default. To use it, do the following:
1. Append virtio_blk.use_bio=1 to the kernel command line on the

guest. You can do so via YaST › System › Boot Loader.
271 z 366 2017-05-27 13:21

You can do it also by editing /etc/default/grub , searching for the line

that contains GRUB_CMDLINE_LINUX_DEFAULT= , and adding the kernel pa-
rameter at the end. Then run grub2-mkconfig >/boot/grub2/grub.cfg
to update the grub2 boot menu.
2. Reboot the guest with the new kernel command line active.
Tip: Bio-Based Driver on Slow Devices
The bio-based virtio-blk driver does not help on slow devices such as spin hard
disks. The reason is that the benefit of scheduling is larger than what the short-
ened bio path offers. Do not use the bio-based driver on slow devices.
29.3.1.4 Accessing iSCSI Resources Directly

QEMU now integrates with libiscsi . This allows QEMU to access iSCSI resources
directly and use them as virtual machine block devices. This feature does not require
any host iSCSI initiator configuration, as is needed for a libvirt iSCSI target based
storage pool setup. Instead it directly connects guest storage interfaces to an iSCSI
target LUN by means of the user space library libiscsi. iSCSI-based disk devices can
also be specified in the libvirt XML configuration.
Note: RAW Image Format
This feature is only available using the RAW image format, as the iSCSI protocol
has some technical limitations.
The following is the QEMU command line interface for iSCSI connectivity.
Note: virt-manager Limitation
The use of libiscsi based storage provisioning is not yet exposed by the virt-man-
ager interface, but instead it would be configured by directly editing the guest
xml. This new way of accessing iSCSI based storage is to be done at the command
line.
272 z 366 2017-05-27 13:21

qemu-system-x86_64 -machine accel=kvm \

-drive file=iscsi://192.168.100.1:3260/iqn.2016-08.com.example:314605ab-a88e-49af-b4eb
format=raw,if=none,id=mydrive,cache=none \
-device ide-hd,bus=ide.0,unit=0,drive=mydrive ...
Here is an example snippet of guest domain xml which uses the protocol based
iSCSI:
<devices>
...
<disk type='network' device='disk'>
<driver name='qemu' type='raw'/>
<source protocol='iscsi' name='iqn.2013-07.com.example:iscsi-nopool/2'>
<host name='example.com' port='3260'/>
</source>
<auth username='myuser'>
<secret type='iscsi' usage='libvirtiscsi'/>
</auth>
<target dev='vda' bus='virtio'/>
</disk>
</devices>
Contrast that with an example which uses the host based iSCSI initiator which
virt-manager sets up:
<devices>
...
<disk type='block' device='disk'>
<driver name='qemu' type='raw' cache='none' io='native'/>
<source dev='/dev/disk/by-path/scsi-0:0:0:0'/>
<target dev='hda' bus='ide'/>
<address type='drive' controller='0' bus='0' target='0' unit='0'/>
</disk>
<controller type='ide' index='0'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01'
function='0x1'/>
</controller>
</devices>
29.3.2 Graphic Devices and Display Options

This section describes QEMU options affecting the type of the emulated video card
and the way VM Guest graphical output is displayed.
29.3.2.1 Defining Video Cards

QEMU uses -vga to define a video card used to display VM Guest graphical output.
The -vga option understands the following values:
none
273 z 366 2017-05-27 13:21

Disables video cards on VM Guest (no video card is emulated). You can still ac-
cess the running VM Guest via the serial console.
std
Emulates a standard VESA 2.0 VBE video card. Use it if you intend to use high
display resolution on VM Guest.
cirrus
Emulates Cirrus Logic GD5446 video card. Good choice if you insist on high
compatibility of the emulated video hardware. Most operating systems (even
Windows 95) recognize this type of card.
Tip
For best video performance with the cirrus type, use 16-bit color depth
both on VM Guest and VM Host Server.
29.3.2.2 Display Options

The following options affect the way VM Guest graphical output is displayed.
-display gtk
Display video output in a GTK window. This interface provides UI elements to
configure and control the VM during runtime.
-display sdl
Display video output via SDL, usually in a separate graphics window. For more
information, see the SDL documentation.
-spice option[,option[,...]]
Enables the spice remote desktop protocol.
-display vnc
Refer to Section 29.5, “Viewing a VM Guest with VNC” for more information.
-nographic
Disables QEMU's graphical output. The emulated serial port is redirected to the
console.
After starting the virtual machine with -nographic , press Ctrl – A H in the
virtual console to view the list of other useful shortcuts, for example, to toggle
between the console and the QEMU monitor.
tux > qemu-system-x86_64 -hda /images/sles_base.raw -nographic
274 z 366 2017-05-27 13:21

C-a h print this help

C-a x exit emulator
C-a s save disk data back to file (if -snapshot)
C-a t toggle console timestamps
C-a b send break (magic sysrq)
C-a c switch between console and monitor
C-a C-a sends C-a
(pressed C-a c)
QEMU 2.3.1 monitor - type 'help' for more information

(qemu)
-no-frame
Disables decorations for the QEMU window. Convenient for dedicated desktop
work space.
-full-screen
Starts QEMU graphical output in full screen mode.
-no-quit
Disables the close button of the QEMU window and prevents it from being
closed by force.
-alt-grab , -ctrl-grab
By default, the QEMU window releases the “captured” mouse after pressing
Ctrl – Alt . You can change the key combination to either Ctrl – Alt – Shift (
-alt-grab ), or the right Ctrl key ( -ctrl-grab ).
29.3.3 USB Devices

There are two ways to create USB devices usable by the VM Guest in KVM: you can
either emulate new USB devices inside a VM Guest, or assign an existing host USB
device to a VM Guest. To use USB devices in QEMU you first need to enable the
generic USB driver with the -usb option. Then you can specify individual devices
with the -usbdevice option.
29.3.3.1 Emulating USB Devices in VM Guest

SUSE currently supports the following types of USB devices: disk , host , serial ,
braille , net , mouse , and tablet .
TYPES OF USB DEVICES FOR THE -usbdevice OPTION
disk
275 z 366 2017-05-27 13:21

Emulates a mass storage device based on file. The optional format option is
used rather than detecting the format.
qemu-system-x86_64 [...] -usbdevice

disk:format=raw:/virt/usb_disk.raw
host
Pass through the host device (identified by bus.addr).
serial
Serial converter to a host character device.
braille
Emulates a braille device using BrlAPI to display the braille output.
net
Emulates a network adapter that supports CDC Ethernet and RNDIS protocols.
mouse
Emulates a virtual USB mouse. This option overrides the default PS/2 mouse
emulation. The following example shows the hardware status of a mouse on VM
Guest started with qemu-system-ARCH [...] -usbdevice mouse :
tux > sudo hwinfo --mouse

20: USB 00.0: 10503 USB Mouse
[Created at usb.122]
UDI: /org/freedesktop/Hal/devices/usb_device_627_1_1_if0
[...]
Hardware Class: mouse
Model: "Adomax QEMU USB Mouse"
Hotplug: USB
Vendor: usb 0x0627 "Adomax Technology Co., Ltd"
Device: usb 0x0001 "QEMU USB Mouse"
[...]
tablet
Emulates a pointer device that uses absolute coordinates (such as touchscreen).
This option overrides the default PS/2 mouse emulation. The tablet device is
useful if you are viewing VM Guest via the VNC protocol. See Section 29.5, “View-
ing a VM Guest with VNC” for more information.
29.3.4 Character Devices

Use -chardev to create a new character device. The option uses the following gen-
eral syntax:
276 z 366 2017-05-27 13:21

qemu-system-x86_64 [...] -chardev backend_type,id=id_string
where backend_type can be one of null , socket , udp , msmouse , vc , file ,

pipe , console , serial , pty , stdio , braille , tty , or parport . All charac-
ter devices must have a unique identification string up to 127 characters long. It is
used to identify the device in other related directives. For the complete description
of all back-end's sub-options, see the manual page ( man 1 qemu ). A brief descrip-
tion of the available back-ends follows:
null
Creates an empty device that outputs no data and drops any data it receives.
stdio
Connects to QEMU's process standard input and standard output.
socket
Creates a two-way stream socket. If path is specified, a Unix socket is created:
qemu-system-x86_64 [...] -chardev \

socket,id=unix_socket1,path=/tmp/unix_socket1,server
The server suboption specifies that the socket is a listening socket.

If port is specified, a TCP socket is created:
qemu-system-x86_64 [...] -chardev \

socket,id=tcp_socket1,host=localhost,port=7777,server,nowait
The command creates a local listening ( server ) TCP socket on port 7777.
QEMU will not block waiting for a client to connect to the listening port ( nowait
).
udp
Sends all network traffic from VM Guest to a remote host over the UDP protocol.
-chardev udp,id=udp_fwd,host=mercury.example.com,port=7777
The command binds port 7777 on the remote host mercury.example.com and
sends VM Guest network traffic there.
vc
Creates a new QEMU text console. You can optionally specify the dimensions of
the virtual console:
qemu-system-x86_64 [...] -chardev vc,id=vc1,width=640,height=480 \

-mon chardev=vc1
277 z 366 2017-05-27 13:21

The command creates a new virtual console called vc1 of the specified size,
and connects the QEMU monitor to it.
file
Logs all traffic from VM Guest to a file on VM Host Server. The path is required
and will be created if it does not exist.
-chardev file,id=qemu_log1,path=/var/log/qemu/guest1.log
By default QEMU creates a set of character devices for serial and parallel ports, and
a special console for QEMU monitor. However, you can create your own character
devices and use them for the mentioned purposes. The following options will help
you:
-serial char_dev
Redirects the VM Guest's virtual serial port to a character device char_dev on
VM Host Server. By default, it is a virtual console ( vc ) in graphical mode, and
stdio in non-graphical mode. The -serial understands many sub-options.
See the manual page man 1 qemu for a complete list of them.
You can emulate up to 4 serial ports. Use -serial none to disable all serial
ports.
-parallel device
Redirects the VM Guest's parallel port to a device . This option supports the
same devices as -serial .
Tip
With SUSE Linux Enterprise Server as a VM Host Server, you can directly use
the hardware parallel port devices /dev/parportN where N is the number
of the port.
You can emulate up to 3 parallel ports. Use -parallel none to disable all par-
allel ports.
-monitor char_dev
Redirects the QEMU monitor to a character device char_dev on VM Host
Server. This option supports the same devices as -serial . By default, it is a vir-
tual console ( vc ) in a graphical mode, and stdio in non-graphical mode.
278 z 366 2017-05-27 13:21

For a complete list of available character devices back-ends, see the man page (
man 1 qemu ).

Use the -netdev option in combination with -device to define a specific type of
networking and a network interface card for your VM Guest. The syntax for the
-netdev option is
-netdev type[,prop[=value][,...]]
Currently, SUSE supports the following network types: user , bridge , and tap .
For a complete list of -netdev sub-options, see the manual page ( man 1 qemu ).
SUPPORTED -netdev SUB-OPTIONS
bridge
Uses a specified network helper to configure the TAP interface and attach it to a
specified bridge. For more information, see Section 29.4.3, “Bridged Network-
ing”.
user
Specifies user-mode networking. For more information, see Section 29.4.2,
“User-Mode Networking”.
tap
Specifies bridged or routed networking. For more information, see Sec-
tion 29.4.3, “Bridged Networking”.
29.4.1 Defining a Network Interface Card

Use -netdev together with the related -device option to add a new emulated
network card:
-netdev tap 1 ,id=hostnet0 \
-device virtio-net-pci 2 ,netdev=hostnet0,vlan=1 3 ,\
macaddr=00:16:35:AF:94:4B 4 ,name=ncard1
279 z 366 2017-05-27 13:21

1 Specifies the network device type.

2 Specifies the model of the network card. Use -device help and search for
the Network devices: section to get the list of all network card models sup-
ported by QEMU on your platform:
qemu-system-x86_64 -device help

[...]
Network devices:
name "e1000", bus PCI, desc "Intel Gigabit Ethernet"
name "e1000-82540em", bus PCI, desc "Intel Gigabit Ethernet"
name "e1000-82544gc", bus PCI, desc "Intel Gigabit Ethernet"
name "e1000-82545em", bus PCI, desc "Intel Gigabit Ethernet"
name "i82550", bus PCI, desc "Intel i82550 Ethernet"
name "i82551", bus PCI, desc "Intel i82551 Ethernet"
name "i82557a", bus PCI, desc "Intel i82557A Ethernet"
[...]
Currently, SUSE supports the models rtl8139 , e1000 and its variants
e1000-82540em , e1000-82544gc and e1000-82545em , and
virtio-net-pci . To view a list of options for a specific driver, add help as
a driver option:
qemu-system-x86_64 -device e1000,help

e1000.mac=macaddr
e1000.vlan=vlan
e1000.netdev=netdev
e1000.bootindex=int32
e1000.autonegotiation=on/off
e1000.mitigation=on/off
e1000.addr=pci-devfn
e1000.romfile=str
e1000.rombar=uint32
e1000.multifunction=on/off
e1000.command_serr_enable=on/off
3 Connects the network interface to VLAN number 1. You can specify your own
number—it is mainly useful for identification purpose. If you omit this subop-
tion, QEMU uses the default 0.
4 Specifies the Media Access Control (MAC) address for the network card. It is a
unique identifier and you are advised to always specify it. If not, QEMU sup-
plies its own default MAC address and creates a possible MAC address conflict
within the related VLAN.
29.4.2 User-Mode Networking

The -netdev user option instructs QEMU to use user-mode networking. This is
the default if no networking mode is selected. Therefore, these command lines are
equivalent:
280 z 366 2017-05-27 13:21

qemu-system-x86_64 -hda /images/sles_base.raw
qemu-system-x86_64 -hda /images/sles_base.raw -netdev user,id=hostnet0
This mode is useful if you want to allow the VM Guest to access the external network
resources, such as the Internet. By default, no incoming traffic is permitted and
therefore, the VM Guest is not visible to other machines on the network. No admin-
istrator privileges are required in this networking mode. The user-mode is also use-
ful for doing a network boot on your VM Guest from a local directory on VM Host
Server.
The VM Guest allocates an IP address from a virtual DHCP server. VM Host Server
(the DHCP server) is reachable at 10.0.2.2, while the IP address range for allocation
starts from 10.0.2.15. You can use ssh to connect to VM Host Server at 10.0.2.2,
and scp to copy files back and forth.
29.4.2.1 Command Line Examples

This section shows several examples on how to set up user-mode networking with
QEMU.
EXAMPLE 29.1: RESTRICTED USER-MODE NETWORKING
-netdev user 1 ,id=hostnet0 \
-device virtio-net-pci,netdev=hostnet0,vlan=1 2 ,name=user_net1 3 ,restrict=yes
1 Specifies user-mode networking.

2 Connects to VLAN number 1. If omitted, defaults to 0.
3 Specifies a human-readable name of the network stack. Useful when identify-
ing it in the QEMU monitor.
4 Isolates VM Guest. It then cannot communicate with VM Host Server and no
network packets will be routed to the external network.
EXAMPLE 29.2: USER-MODE NETWORKING WITH CUSTOM IP RANGE
-netdev user,id=hostnet0 \
-device virtio-net-pci,netdev=hostnet0,net=10.2.0.0/8 1 ,host=10.2.0.6 2 ,\
dhcpstart=10.2.0.20 3 ,hostname=tux_kvm_guest 4
281 z 366 2017-05-27 13:21

1 Specifies the IP address of the network that VM Guest sees and optionally the
netmask. Default is 10.0.2.0/8.
2 Specifies the VM Host Server IP address that VM Guest sees. Default is
10.0.2.2.
3 Specifies the first of the 16 IP addresses that the built-in DHCP server can as-
sign to VM Guest. Default is 10.0.2.15.
4 Specifies the host name that the built-in DHCP server will assign to VM Guest.
EXAMPLE 29.3: USER-MODE NETWORKING WITH NETWORK-BOOT AND TFTP
-device virtio-net-pci,netdev=hostnet0,tftp=/images/tftp_dir 1 ,\
bootfile=/images/boot/pxelinux.0 2
1 Activates a built-in TFTP (a file transfer protocol with the functionality of a very
basic FTP) server. The files in the specified directory will be visible to a VM
Guest as the root of a TFTP server.
2 Broadcasts the specified file as a BOOTP (a network protocol that offers an IP
address and a network location of a boot image, often used in diskless work-
stations) file. When used together with tftp , the VM Guest can boot from
network from the local directory on the host.
EXAMPLE 29.4: USER-MODE NETWORKING WITH HOST PORT FORWARDING
-device virtio-net-pci,netdev=hostnet0,hostfwd=tcp::2222-:22
Forwards incoming TCP connections to the port 2222 on the host to the port 22
( SSH ) on VM Guest. If sshd is running on VM Guest, enter
ssh qemu_host -p 2222
where qemu_host is the host name or IP address of the host system, to get a
SSH prompt from VM Guest.
29.4.3 Bridged Networking
282 z 366 2017-05-27 13:21

With the -netdev tap option, QEMU creates a network bridge by connecting the
host TAP network device to a specified VLAN of VM Guest. Its network interface is
then visible to the rest of the network. This method does not work by default and
needs to be explicitly specified.
First, create a network bridge and add a VM Host Server physical network interface
(usually eth0 ) to it:
1. Start YaST Control Center and select System › Network Settings.
2. Click Add and select Bridge from the Device Type drop-down box in the Hard-
ware Dialog window. Click Next.
3. Choose whether you need a dynamically or statically assigned IP address,

and fill the related network settings if applicable.
4. In the Bridged Devices pane, select the Ethernet device to add to the bridge.
FIGURE 29.2: CONFIGURING NETWORK BRIDGE WITH YAST
283 z 366 2017-05-27 13:21

Click Next. When asked about adapting an already configured device, click
Continue.
5. Click OK to apply the changes. Check if the bridge is created:
tux > brctl show

br0 8000.001676d670e4 no eth0
29.4.3.1 Connecting to a Bridge Manually

Use the following example script to connect VM Guest to the newly created bridge
interface br0 . Several commands in the script are run via the sudo mechanism be-
cause they require root privileges.
Note: Required Packages
Make sure the tunctl and bridge-utils packages are installed on the VM
Host Server. If not, install them with zypper in tunctl bridge-utils .
#!/bin/bash
bridge=br0 1
tap=$(sudo tunctl -u $(whoami) -b) 2
sudo ip link set $tap up 3
sleep 1s 4
sudo brctl addif $bridge $tap 5
qemu-system-x86_64 -machine accel=kvm -m 512 -hda /images/sles_base.raw \
-netdev tap,id=hostnet0 \
-device virtio-net-pci,netdev=hostnet0,vlan=0,macaddr=00:16:35:AF:94:4B,\
ifname=$tap 6 ,script=no 7 ,downscript=no
sudo brctl delif $bridge $tap 8
sudo ip link set $tap down 9
sudo tunctl -d $tap 10
284 z 366 2017-05-27 13:21

1 Name of the bridge device.

2 Prepare a new TAP device and assign it to the user who runs the script. TAP de-
vices are virtual network devices often used for virtualization and emulation
setups.
3 Bring up the newly created TAP network interface.
4 Make a 1-second pause to make sure the new TAP network interface is really
up.
5 Add the new TAP device to the network bridge br0 .
6 The ifname= suboption specifies the name of the TAP network interface
used for bridging.
7 Before qemu-system-ARCH connects to a network bridge, it checks the
script and downscript values. If it finds the specified scripts on the VM
Host Server file system, it runs the script before it connects to the network
bridge and downscript after it exits the network environment. You can use
these scripts to first set up and bring up the bridged network devices, and
then to deconfigure them. By default, /etc/qemu-ifup and
/etc/qemu-ifdown are examined. If script=no and downscript=no are
specified, the script execution is disabled and you need to take care of it man-
ually.
8 Deletes the TAP interface from a network bridge br0 .
9 Sets the state of the TAP device to down .
10 Deconfigures the TAP device.
29.4.3.2 Connecting to a Bridge with qemu-bridge-helper

Another way to connect VM Guest to a network through a network bridge is by
means of the qemu-bridge-helper helper program. It configures the TAP inter-
face for you, and attaches it to the specified bridge. The default helper executable is
/usr/lib/qemu-bridge-helper . The helper executable is setuid root, which is
only executable by the members of the virtualization group ( kvm ). Therefore the
qemu-system-ARCH command itself does not need to be run under root privi-
leges.
The helper is automatically called when you specify a network bridge:
-netdev bridge,id=hostnet0,vlan=0,br=br0 \
-device virtio-net-pci,netdev=hostnet0
You can specify your own custom helper script that will take care of the TAP device
(de)configuration, with the helper=/path/to/your/helper option:
-netdev bridge,id=hostnet0,vlan=0,br=br0,helper=/path/to/bridge-helper \
285 z 366 2017-05-27 13:21

-device virtio-net-pci,netdev=hostnet0
Tip
To define access privileges to qemu-bridge-helper , inspect the

/etc/qemu/bridge.conf file. For example the following directive
allow br0
allows the qemu-system-ARCH command to connect its VM Guest to the net-

work bridge br0 .

By default QEMU uses a GTK (a cross-platform toolkit library) window to display the
graphical output of a VM Guest. With the -vnc option specified, you can make
QEMU listen on a specified VNC display and redirect its graphical output to the VNC
session.
Tip
When working with QEMU's virtual machine via VNC session, it is useful to work
with the -usbdevice tablet option.
Moreover, if you need to use another keyboard layout than the default en-us ,
specify it with the -k option.
The first suboption of -vnc must be a display value. The -vnc option understands
the following display specifications:
host:display
Only connections from host on the display number display will be accepted.
The TCP port on which the VNC session is then running is normally a 5900 +
display number. If you do not specify host , connections will be accepted
from any host.
unix:path
The VNC server listens for connections on Unix domain sockets. The path op-
tion specifies the location of the related Unix socket.
286 z 366 2017-05-27 13:21

none
The VNC server functionality is initialized, but the server itself is not started. You
can start the VNC server later with the QEMU monitor. For more information,
see Chapter 30, Virtual Machine Administration Using QEMU Monitor.
Following the display value there may be one or more option flags separated by
commas. Valid options are:
reverse
Connect to a listening VNC client via a reverse connection.
websocket
Opens an additional TCP listening port dedicated to VNC Websocket connec-
tions. By definition the Websocket port is 5700+display.
password
Require that password-based authentication is used for client connections.
tls
Require that clients use TLS when communicating with the VNC server.
x509=/path/to/certificate/dir
Valid if TLS is specified. Require that x509 credentials are used for negotiating
the TLS session.
x509verify=/path/to/certificate/dir
Valid if TLS is specified. Require that x509 credentials are used for negotiating
the TLS session.
sasl
Require that the client uses SASL to authenticate with the VNC server.
acl
Turn on access control lists for checking of the x509 client certificate and SASL
party.
lossy
Enable lossy compression methods (gradient, JPEG, ...).
non-adaptive
Disable adaptive encodings. Adaptive encodings are enabled by default.
share=[allow-exclusive|force-shared|ignore]
Set display sharing policy.
287 z 366 2017-05-27 13:21

Note
For more details about the display options, see the qemu-doc man page.
An example VNC usage:
tux > qemu-system-x86_64 [...] -vnc :5

(on the client:)
wilber > :~>vinagre venus:5905 &
FIGURE 29.3: QEMU VNC SESSION
29.5.1 Secure VNC Connections
288 z 366 2017-05-27 13:21

The default VNC server setup does not use any form of authentication. In the previ-
ous example, any user can connect and view the QEMU VNC session from any host
on the network.
There are several levels of security that you can apply to your VNC client/server con-
nection. You can either protect your connection with a password, use x509 certifi-
cates, use SASL authentication, or even combine some authentication methods in
one QEMU command.
See Section B.2, “Generating x509 Client/Server Certificates” for more information
about the x509 certificates generation. For more information about configuring
x509 certificates on a VM Host Server and the client, see Section 11.3.2, “Remote
TLS/SSL Connection with x509 Certificate ( qemu+tls or xen+tls )” and Sec-
tion 11.3.2.3, “Configuring the Client and Testing the Setup”.
The Vinagre VNC viewer supports advanced authentication mechanisms. Therefore,

it will be used to view the graphical output of VM Guest in the following examples.
For this example, let us assume that the server x509 certificates ca-cert.pem ,
server-cert.pem , and server-key.pem are located in the /etc/pki/qemu di-
rectory on the host, while the client's certificates are distributed in the following lo-
cations on the client:
EXAMPLE 29.5: PASSWORD AUTHENTICATION
qemu-system-x86_64 [...] -vnc :5,password -monitor stdio
Starts the VM Guest graphical output on VNC display number 5 (usually port
5905). The password suboption initializes a simple password-based authenti-
cation method. There is no password set by default and you need to set one
with the change vnc password command in QEMU monitor:
QEMU 2.3.1 monitor - type 'help' for more information

(qemu) change vnc password
Password: ****
You need the -monitor stdio option here, because you would not be able to
manage the QEMU monitor without redirecting its input/output.
289 z 366 2017-05-27 13:21

FIGURE 29.4: AUTHENTICATION DIALOG IN VINAGRE
EXAMPLE 29.6: X509 CERTIFICATE AUTHENTICATION
The QEMU VNC server can use TLS encryption for the session and x509 certifi-
cates for authentication. The server asks the client for a certificate and validates
it against the CA certificate. Use this authentication type if your company pro-
vides an internal certificate authority.
qemu-system-x86_64 [...] -vnc :5,tls,x509verify=/etc/pki/qemu
EXAMPLE 29.7: X509 CERTIFICATE AND PASSWORD AUTHENTICATION
290 z 366 2017-05-27 13:21

You can combine the password authentication with TLS encryption and x509
certificate authentication to create a two-layer authentication model for clients.
Remember to set the password in the QEMU monitor after you run the follow-
ing command:
qemu-system-x86_64 [...] -vnc :5,password,tls,x509verify=/etc/pki/qemu \

-monitor stdio
EXAMPLE 29.8: SASL AUTHENTICATION
Simple Authentication and Security Layer (SASL) is a framework for authentica-

tion and data security in Internet protocols. It integrates several authentication
mechanisms, like PAM, Kerberos, LDAP and more. SASL keeps its own user data-
base, so the connecting user accounts do not need to exist on VM Host Server.
For security reasons, you are advised to combine SASL authentication with TLS
encryption and x509 certificates:
qemu-system-x86_64 [...] -vnc :5,tls,x509,sasl -monitor stdio
30 Virtual Machine Administration

Using QEMU Monitor
30.2 Getting Information about the Guest System
30.9 Suspending and Resuming Virtual Machine Execution
291 z 366 2017-05-27 13:21


When QEMU is running, a monitor console is provided for performing interaction

with the user. Using the commands available in the monitor console, it is possible to
inspect the running operating system, change removable media, take screenshots or
audio grabs and control other aspects of the virtual machine.
Note
The following sections list selected useful QEMU monitor commands and their
purpose. To get the full list, enter help in the QEMU monitor command line.

You can access the monitor console from QEMU window either by a keyboard short-
cut—press Ctrl – Alt – 2 (to return to QEMU, press Ctrl – Alt – 1 )—or alterna-
tively by clicking View in the QEMU GUI window, then compatmonitor0. The most con-
venient way is to show the QEMU window tabs with View › Show Tabs. Then you can
easily switch between the guest screen, monitor screen, and the output of the serial
and parallel console.
To get help while using the console, use help or ? . To get help for a specific com-
mand, use help command .
30.2 Getting Information about the Guest

System
To get information about the guest system, use info . If used without any option,
the list of possible options is printed. Options determine which part of the system
will be analyzed:
info version
Shows the version of QEMU.
292 z 366 2017-05-27 13:21

info commands
Lists available QMP commands.
info network
Shows the network state.
info chardev
Shows the character devices.
info block
Information about block devices, such as hard disks, floppy drives, or CD-ROMs.
info blockstats
Read and write statistics on block devices.
info registers
Shows the CPU registers.
info cpus
Shows information about available CPUs.
info history
Shows the command line history.
info irq
Shows the interrupt statistics.
info pic
Shows the i8259 (PIC) state.
info pci
Shows the PCI information.
info tlb
Shows virtual to physical memory mappings.
info mem
Shows the active virtual memory mappings.
info jit
Shows dynamic compiler information.
info kvm
Shows the KVM information.
293 z 366 2017-05-27 13:21

info numa
Shows the NUMA information.
info usb
Shows the guest USB devices.
info usbhost
Shows the host USB devices.
info profile
Shows the profiling information.
info capture
Shows the capture (audio grab) information.
info snapshots
Shows the currently saved virtual machine snapshots.
info status
Shows the current virtual machine status.
info pcmcia
Shows the guest PCMCIA status.
info mice
Shows which guest mice are receiving events.
info vnc
Shows the VNC server status.
info name
Shows the current virtual machine name.
info uuid
Shows the current virtual machine UUID.
info usernet
Shows the user network stack connection states.
info migrate
Shows the migration status.
info balloon
Shows the balloon device information.
294 z 366 2017-05-27 13:21

info qtree
Shows the device tree.
info qdm
Shows the qdev device model list.
info roms
Shows the ROMs.
info migrate_cache_size
Shows the current migration xbzrle (“Xor Based Zero Run Length Encoding”)
cache size.
info migrate_capabilities
Shows the status of the various migration capabilities, such as xbzrle compres-
sion.
info mtree
Shows the VM Guest memory hierarchy.
info trace-events
Shows available trace-events and their status.

To change the VNC password, use the change vnc password command and enter
the new password:
(qemu) change vnc password

Password: ********
(qemu)

To add a new disk while the guest is running (hotplug), use the drive_add and
device_add commands. First define a new drive to be added as a device to bus 0:
(qemu) drive_add 0 if=none,file=/tmp/test.img,format=raw,if=disk1

OK
You can confirm your new device by querying the block subsystem:
295 z 366 2017-05-27 13:21

(qemu) info block

[...]
disk1: removable=1 locked=0 tray-open=0 file=/tmp/test.img ro=0 drv=raw \
encrypted=0 bps=0 bps_rd=0 bps_wr=0 iops=0 iops_rd=0 iops_wr=0
After the new drive is defined, it needs to be connected to a device so that the guest
can see it. The typical device would be a virtio-blk-pci or scsi-disk . To get
the full list of available driver values, run:
(qemu) device_add ?
name "VGA", bus PCI
name "usb-storage", bus usb-bus
[...]
name "virtio-blk-pci", bus virtio-bus
Now add the device
(qemu) device_add virtio-blk-pci,drive=disk1,id=myvirtio1
and confirm with
(qemu) info pci

[...]
Bus 0, device 4, function 0:
SCSI controller: PCI device 1af4:1001
IRQ 0.
BAR0: I/O at 0xffffffffffffffff [0x003e].
BAR1: 32 bit memory at 0xffffffffffffffff [0x00000ffe].
id "myvirtio1"
Tip
Devices added with the device_add command can be removed from the guest
with device_del . Enter help device_del on the QEMU monitor command
line for more information.
To release the device or file connected to the removable media device, use the
eject device command. Use the optional -f to force ejection.
To change removable media (like CD-ROMs), use the change device command.
The name of the removable media can be determined using the info block com-
mand:
(qemu) info block

ide1-cd0: type=cdrom removable=1 locked=0 file=/dev/sr0 ro=1 drv=host_device
(qemu) change ide1-cd0 /path/to/image
296 z 366 2017-05-27 13:21


It is possible to use the monitor console to emulate keyboard and mouse input if
necessary. For example, if your graphical user interface intercepts some key combi-
nations at low level (such as Ctrl – Alt – F1 in X Window), you can still enter them
using the sendkey keys :
sendkey ctrl-alt-f1
To list the key names used in the keys option, enter sendkey and press →| .
To control the mouse, the following commands can be used:
mouse_move dx dy [ dz ]
Move the active mouse pointer to the specified coordinates dx, dy with the op-
tional scroll axis dz.
mouse_button val
Change the state of the mouse buttons (1=left, 2=middle, 4=right).
mouse_set index
Set which mouse device receives events. Device index numbers can be obtained
with the info mice command.

If the virtual machine was started with the -balloon virtio option and the par-
avirtualized balloon device that allows to dynamically change the amount of mem-
ory available is therefore enabled, it is possible to change the available memory dy-
namically. For more information about enabling the balloon device, see Sec-
tion 28.1, “Basic Installation with qemu-system-ARCH ”.
To get information about the balloon device in the monitor console and to deter-
mine whether the device is enabled, use the info balloon command:
(qemu) info balloon
If the balloon device is enabled, use the balloon memory_in_MB command to set
the requested amount of memory:
297 z 366 2017-05-27 13:21

(qemu) balloon 400

To save the content of the virtual machine memory to a disk or console output, use
the following commands:
memsave addr size filename

Saves virtual memory dump starting at addr of size size to file filename
pmemsave addr size filename

Saves physical memory dump starting at addr of size size to file filename -
x / fmt addr
Makes a virtual memory dump starting at address addr and formatted accord-
ing to the fmt string. The fmt string consists of three parameters
countformatsize :
The count parameter is the number of items to be dumped.
The format can be x (hex), d (signed decimal), u (unsigned decimal), o (oc-
tal), c (char) or i (assembly instruction).
The size parameter can be b (8 bits), h (16 bits), w (32 bits) or g (64 bits).
On x86, h or w can be specified with the i format to respectively select 16 or
32-bit code instruction size.
xp / fmt addr
Makes a physical memory dump starting at address addr and formatted ac-
cording to the fmt string. The fmt string consists of three parameters
countformatsize :
The count parameter is the number of the items to be dumped.
The format can be x (hex), d (signed decimal), u (unsigned decimal), o (oc-
tal), c (char) or i (asm instruction).
The size parameter can be b (8 bits), h (16 bits), w (32 bits) or g (64 bits).
On x86, h or w can be specified with the i format to respectively select 16 or
32-bit code instruction size.
298 z 366 2017-05-27 13:21

Managing snapshots in QEMU monitor is not officially supported by SUSE yet. The
information found in this section may be helpful in specific cases.
Virtual Machine snapshots are snapshots of the complete virtual machine including
the state of CPU, RAM, and the content of all writable disks. To use virtual machine
snapshots, you must have at least one non-removable and writable block device us-
ing the qcow2 disk image format.
Snapshots are helpful when you need to save your virtual machine in a particular
state. For example, after you have configured network services on a virtualized
server and want to quickly start the virtual machine in the same state that was saved
last. You can also create a snapshot after the virtual machine has been powered off
to create a backup state before you try something experimental and possibly make
VM Guest unstable. This section introduces the former case, while the latter is de-
scribed in Section 28.2.3, “Managing Snapshots of Virtual Machines with qemu-img”.
The following commands are available for managing snapshots in QEMU monitor:
savevm name
Creates a new virtual machine snapshot under the tag name or replaces an ex-
isting snapshot.
loadvm name
Loads a virtual machine snapshot tagged name .
delvm
Deletes a virtual machine snapshot.
info snapshots
Prints information about available snapshots.
(qemu) info snapshots

Snapshot list:
ID 1 TAG 2 VM SIZE 3 DATE 4 VM CLOCK
1 booting 4.4M 2013-11-22 10:51:10 00:00:20.476
2 booted 184M 2013-11-22 10:53:03 00:02:05.394
3 logged_in 273M 2013-11-22 11:00:25 00:04:34.843
1 Unique identification number of the snapshot. Usually auto-incremented.

2 Unique description string of the snapshot. It is meant as a human read-
able version of the ID.
3 The disk space occupied by the snapshot. Note that the more memory is
consumed by running applications, the bigger the snapshot is.
4 Time and date the snapshot was created.
5 The current state of the virtual machine's clock.
299 z 366 2017-05-27 13:21

30.9 Suspending and Resuming Virtual

Machine Execution
The following commands are available for suspending and resuming virtual ma-
chines:
stop
Suspends the execution of the virtual machine.
cont
Resumes the execution of the virtual machine.
system_reset
Resets the virtual machine. The effect is similar to the reset button on a physical
machine. This may leave the file system in an unclean state.
system_powerdown
Sends an ACPI shutdown request to the machine. The effect is similar to the
power button on a physical machine.
q or quit
Terminates QEMU immediately.

The live migration process allows to transmit any virtual machine from one host sys-
tem to another host system without any interruption in availability. It is possible to
change hosts permanently or only during maintenance.
The requirements for live migration:
All requirements from Section 10.7.1, “Migration Requirements” are applica-

ble.
Live migration is only possible between VM Host Servers with the same CPU
features.
AHCI interface, VirtFS feature, and the -mem-path command line option are
not compatible with migration.
300 z 366 2017-05-27 13:21

The guest on the source and destination hosts must be started in the same
way.
-snapshot qemu command line option should not be used for migration
(and this qemu command line option is not supported).
The postcopy mode is not yet supported in SUSE Linux Enterprise Server. It is
released as a technology preview only. For more information about postcopy ,
see http://wiki.qemu.org/Features/PostCopyLiveMigration↗.
More recommendations can be found at the following Web site: http://www.linux-

kvm.org/page/Migration↗
The live migration process has the following steps:
1. The virtual machine instance is running on the source host.
2. The virtual machine is started on the destination host in the frozen listening
mode. The parameters used are the same as on the source host plus the
-incoming tcp: ip : port parameter, where ip specifies the IP address
and port specifies the port for listening to the incoming migration. If 0 is
set as IP address, the virtual machine listens on all interfaces.
3. On the source host, switch to the monitor console and use the
migrate -d tcp: destination_ip : port command to initiate the mi-
gration.
4. To determine the state of the migration, use the info migrate command
in the monitor console on the source host.
5. To cancel the migration, use the migrate_cancel command in the moni-

tor console on the source host.
6. To set the maximum tolerable downtime for migration in seconds, use the
migrate_set_downtime number_of_seconds command.
7. To set the maximum speed for migration in bytes per second, use the
migrate_set_speed bytes_per_second command.
301 z 366 2017-05-27 13:21

QMP is a JSON-based protocol that allows applications—such as libvirt —to com-

municate with a running QEMU instance. There are several ways you can access the
QEMU monitor using QMP commands.
30.11.1 Access QMP via Standard Input/Output

The most flexible way to use QMP is by specifying the -mon option. The following
example creates a QMP instance using standard input/output. Note that in the fol-
lowing examples, -> marks lines with commands sent from client to the running
QEMU instance, while <- marks lines with the output returned from QEMU.
# qemu-system-x86_64 [...] \
-chardev stdio,id=mon0 \
-mon chardev=mon0,mode=control,pretty=on
<- {
"QMP": {
"version": {
"qemu": {
"micro": 0,
"minor": 0,
"major": 2
},
"package": ""
},
"capabilities": [
]
}
}
When a new QMP connection is established, QMP sends its greeting message and
enters capabilities negotiation mode. In this mode, only the qmp_capabilities
command works. To exit capabilities negotiation mode and enter command mode,
the qmp_capabilities command must be issued first:
-> { "execute": "qmp_capabilities" }

<- {
"return": {
}
}
Note that "return": {} is a QMP's success response.
QMP's commands can have arguments. For example to eject a CD-ROM drive, enter
the following:
->{ "execute": "eject", "arguments": { "device": "ide1-cd0" } }

<- {
"timestamp": {
"seconds": 1410353381,
"microseconds": 763480
},
302 z 366 2017-05-27 13:21

"event": "DEVICE_TRAY_MOVED",
"data": {
"device": "ide1-cd0",
"tray-open": true
}
}
{
"return": {
}
}
30.11.2 Access QMP via Telnet

Instead of the standard input/output, you can connect the QMP interface to a net-
work socket and communicate with it via a specified port:
# qemu-system-x86_64 [...] \
-chardev socket,id=mon0,host=localhost,port=4444,server,nowait \
And then run telnet to connect to port 4444:
# telnet localhost 4444

Trying ::1...
Connected to localhost.
Escape character is '^]'.
<- {
"QMP": {
"version": {
"qemu": {
"micro": 0,
"minor": 0,
"major": 2
},
"package": ""
},
"capabilities": [
]
}
}
You can create several monitor interfaces at the same time. The following example
creates one HMP instance—human monitor which understands 'normal' QEMU
monitor's commands—on the standard input/output, and one QMP instance on lo-
calhost port 4444:
# qemu-system-x86_64 [...] \
-chardev stdio,id=mon0 -mon chardev=mon0,mode=readline \
-chardev socket,id=mon1,host=localhost,port=4444,server,nowait \
303 z 366 2017-05-27 13:21

30.11.3 Access QMP via Unix Socket

Invoke QEMU using the -qmp option, and create a unix socket:
# qemu-system-x86_64 [...] \
-qmp unix:/tmp/qmp-sock,server --monitor stdio
QEMU waiting for connection on: unix:./qmp-sock,server
To communicate with the QEMU instance via the /tmp/qmp-sock socket, use nc
(see man 1 nc for more information) from another terminal on the same host:
# nc -U /tmp/qmp-sock
<- {"QMP": {"version": {"qemu": {"micro": 0, "minor": 0, "major": 2} [...]
30.11.4 Access QMP via libvirt 's virsh Command

If you run your virtual machines under libvirt (see Part II, “Managing Virtual Ma-
chines with libvirt ”), you can communicate with its running guests by running
the virsh qemu-monitor-command :
# virsh qemu-monitor-command vm_guest1 \

--pretty '{"execute":"query-kvm"}'
<- {
"return": {
"enabled": true,
"present": true
},
"id": "libvirt-8"
}
In the above example, we ran the simple command query-kvm which checks if the
host is capable of running KVM and if KVM is enabled.
Tip: Generating Human-Readable Output
To use the standard human-readable output format of QEMU instead of the JSON
format, use the --hmp option:
# virsh qemu-monitor-command vm_guest1 --hmp "query-kvm"
304 z 366 2017-05-27 13:21

Part VI Managing Virtual Machines

with LXC
31 Linux Containers
32 Migration from LXC to libvirt-lxc

Since SUSE Linux Enterprise Server 12, LXC is integrated into libvirt library. This
decision has several advantages over using LXC as a separate solution—such as
a unified approach with other virtualization solutions or independence on the
kernel used. This chapter describes steps needed to migrate …
31 Linux Containers
31.4 Differences Between the libvirt LXC Driver and LXC
305 z 366 2017-05-27 13:21

A container is a kind of “virtual machine” that can be started, stopped, frozen, or

cloned (to name but a few tasks). To set up an LXC container, you first need to create
a root file system containing the guest distribution:
PROCEDURE 31.1: CREATING A ROOT FILE SYSTEM
There is currently no GUI to create a root file system. You will thus need to
open a terminal and use virt-create-rootfs as root to populate the new
root file system. In the following steps, the new root file system will be cre-
ated in /path/to/rootfs .
Important: Registration Code Needed
virt-create-rootfs needs a registration code to set up a SUSE Linux

Enterprise Server root file system.
1. Run the virt-create-rootfs command:
virt-create-rootfs --root /path/to/rootfs --distro SLES-12.0 -c registration code
2. Change the root path to the root file system with the chroot command:
chroot /path/to/rootfs
3. Change the password for user root with passwd .
4. Create an operator user without root privileges:
useradd -m operator
5. Change the operator's password:
passwd operator
6. Leave the chroot environment with exit .
PROCEDURE 31.2: DEFINING THE CONTAINER
1. Start Virtual Machine Manager.
2. (Optional) If not already present, add a local LXC connection by clicking

File › Add Connection.
Select LXC (Linux Containers) as the hypervisor and click Connect.
3. Select the localhost (LXC) connection and click File New Virtual Machine menu.
4. Activate Operating system container and click Forward.
306 z 366 2017-05-27 13:21

5. Type the path to the root file system from Procedure 31.1, “Creating a Root
File System” and click the Forward button.
6. Choose the maximum amount of memory and CPUs to allocate to the con-
tainer. Then click the Forward button.
7. Type in a name for the container. This name will be used for all virsh com-
mands on the container.
Click Advanced options. Select the network to connect the container to and
click the Finish button: the container will then be created and started. A con-
sole will also be automatically opened.
Warning: Container Network
To configure the container network, edit the

/etc/sysconfig/network/ifcfg-* files. Make sure not to change the IPv6
setting: this would lead to errors while starting the network.

Libvirt also allows to run single applications instead of full blown Linux distributions
in containers. In this example, bash will be started in its own container.
PROCEDURE 31.3: DEFINING AN APPLICATION CONTAINER USING YAST
1. Start Virtual Machine Manager.
2. (Optional) If not already present, add a local LXC connection by clicking

File › Add Connection.
Select LXC (Linux Containers) as the hypervisor and click Connect.
3. Select the localhost (LXC) connection and click File New Virtual Machine menu.
4. Activate Application container and click Forward.
Set the path to the application to be launched. As an example, the field is

filled with /bin/sh , which is fine to create a first container. Click Forward.
5. Choose the maximum amount of memory and CPUs to allocate to the con-
tainer. Click Forward.
6. Type in a name for the container. This name will be used for all virsh com-
mands on the container.
307 z 366 2017-05-27 13:21

Click Advanced options. Select the network to connect the container to and
click Finish. The container will be created and started. A console will be
opened automatically.
Note that the container will be destroyed after the application has finished
running.

By default, containers are not secured using AppArmor or SELinux. There is no
graphical user interface to change the security model for a libvirt domain, but
virsh will help.
1. Edit the container XML configuration using virsh:
virsh -c lxc:/// edit mycontainer
2. Add the following to the XML configuration, save it and exit the editor.
<domain>
...
<seclabel type="dynamic" model="apparmor"/>
...
</domain>
3. With this configuration, an AppArmor profile for the container will be cre-
ated in the /etc/apparmor.d/libvirt directory. The default profile only
allows the minimum applications to run in the container. This can be
changed by modifying the libvirt- container-uuid file: this file is not
overwritten by libvirt.
31.4 Differences Between the libvirt LXC Driver

and LXC
SUSE Linux Enterprise Server 11 SP3 was shipping LXC, while SUSE Linux Enterprise
Server 12 comes with the libvirt LXC driver, sometimes named libvirt-lxc to avoid
confusion. The containers are not managed or configured in the same way in these
tools. Here is a non-exhaustive list of differences.
308 z 366 2017-05-27 13:21

The main difference is that domain configuration in libvirt is an XML file, while LXC
configuration is a properties file. Most of the LXC properties can be mapped to the
domain XML. The properties that cannot be migrated are:
lxc.network.script.up: this script can be implemented using the

/etc/libvirt/hooks/network libvirt hook, though the script will need to
be adapted.
lxc.network.ipv*: libvirt cannot set the container network configuration from

the domain configuration.
lxc.network.name: libvirt cannot set the container network card name.
lxc.devttydir: libvirt does not allow changing the location of the console de-
vices.
lxc.console: there is currently no way to log the output of the console into a
file on the host for libvirt LXC containers.
lxc.pivotdir: libvirt does not allow to fine-tune the directory used for the
pivot_root . /.olroot is used.
lxc.rootfs.mount: libvirt does not allow to fine-tune this.
LXC VLAN networks automatically create the VLAN interface on the host and then
move it into the guest namespace. libvirt-lxc configuration can mention a VLAN tag
ID only for Open vSwitch tap devices or PCI pass-through of SR-IOV VF. The conver-
sion tool actually needs the user to manually create the VLAN interface on the host
side.
LXC rootfs can also be an image file, but LXC brute-forces the mount to try to detect
the proper file system format. libvirt-lxc can mount image files of several formats,
but the 'auto' value for the format parameter is explicitly not supported. This means
that the generated configuration will need to be tweaked by the user to get a proper
match in that case.
LXC can support any cgroup configuration, even future ones, while libvirt domain
configuration, needs to map each of them.
LXC can mount block devices in the rootfs, but it cannot mount raw partition files:
the file needs to be manually attached to a loop device. On the other hand libvirt-lxc
can mount block devices, but also partition files of any format.
309 z 366 2017-05-27 13:21

LXC Container Driver

http://libvirt.org/drvlxc.html↗
32 Migration from LXC to

libvirt-lxc
32.1 Host Migration
Since SUSE Linux Enterprise Server 12, LXC is integrated into libvirt library. This
decision has several advantages over using LXC as a separate solution—such as a
unified approach with other virtualization solutions or independence on the kernel
used. This chapter describes steps needed to migrate an existing LXC environment
for use with the libvirt library.
32.1 Host Migration

The migration itself has two phases. You first need to migrate the host, then the LXC
containers. After that, you can run the original containers as VM Guests in the
libvirt environment.
PROCEDURE 32.1: HOST MIGRATION
1. Upgrade the host to SUSE Linux Enterprise Server 12 using the official DVD
media.
2. After the upgrade, install the libvirt-daemon-lxc and

libvirt-daemon-config-network packages.
3. Create a libvirt XML configuration lxc_container.xml from the exist-

ing container lxc_container :
310 z 366 2017-05-27 13:21

# virt-lxc-convert /etc/lxc/lxc_container/config > lxc_container.xml
4. Check if the network configuration on the host is the same as in the con-
tainer configuration file, and fix it if needed.
5. Check the lxc_container.xml file for any weird or missing configuration.

Note that some LXC configuration options cannot be mapped to libvirt
configuration. Although the conversion should usually be fine, check Sec-
tion 31.4, “Differences Between the libvirt LXC Driver and LXC” for more de-
tails.
6. Define the container in libvirt based on the created XML definition:
# virsh -c lxc:/// define lxc_container.xml

After the host is migrated, the LXC container in libvirt will not boot. It needs to
be migrated to SUSE Linux Enterprise Server 12 as well to get everything working.
PROCEDURE 32.2: CONTAINER MIGRATION
1. The baseproduct file is missing (and zypper keeps complaining about it).
Create the relevant symbolic link:
# ROOTFS=/var/lib/lxc/lxc_container/rootfs
# ln -s $ROOTFS/etc/products.d/SUSE_SLES.prod $ROOTFS/etc/products.d/baseproduct
2. Add the DVD repository. Note that you need to replace the DVD device with
the one attached to your container:
# zypper --root $ROOTFS ar \

cd:///?devices=/dev/dvd SLES12-12
3. Disable or remove previous repositories:
# zypper --root $ROOTFS lr

| Alias | Name | Enabled | Refresh
--+-----------------------------+------------------------------+---------+--------
1 | SLES12-12 | SLES12-12 | Yes | No
2 | SUSE-[...]-Server-11-SP3 38 | SUSE-[...]-Server-11-SP3 138 | Yes | No
# zypper --root $ROOTFS rr 2
4. Upgrade the container:
311 z 366 2017-05-27 13:21

# zypper --root $ROOTFS dup
5. Install the Minimal pattern to make sure everything required is installed:
# zypper --root $ROOTFS in -t pattern Minimal

After the host and container migration is complete, the container can be started:
# virsh -c lxc:/// start lxc_container
If you need to get a console to view the logging messages produced by the con-
tainer, run:
# virsh -c lxc:/// console lxc_container
Glossary
General
Create Virtual Machine Wizard
A software program available in YaST and Virtual Machine Manager that pro-
vides a graphical interface to guide you through the steps to create virtual ma-
chines. It can also be run in text mode by entering virt-install at a com-
mand prompt in the host environment.
312 z 366 2017-05-27 13:21

Dom0
The term is used in Xen environments, and refers to a virtual machine. The host
operating system is actually a virtual machine running in a privileged domain
and can be called Dom0. All other virtual machines on the host run in unprivi-
leged domains and can be called domain U's.
hardware-assisted
Intel* and AMD* provide virtualization hardware-assisted technology. This re-
duces frequency of VM IN/OUT (fewer VM traps), because software is a major
source of overhead, and increases the efficiency (the execution is done by the
hardware). Moreover this reduces the memory footprint, provides better re-
source control, and allows secure assignment of specific I/O devices.
Host Environment
The desktop or command line environment that allows interaction with the host
computer's environment. It provides a command line environment and can also
include a graphical desktop, such as GNOME or IceWM. The host environment
runs as a special type of virtual machine that has privileges to control and man-
age other virtual machines. Other commonly used terms include Dom0, privi-
leged domain, and host operating system.
Hypervisor
The software that coordinates the low-level interaction between virtual ma-
chines and the underlying physical computer hardware.
KVM
See Chapter 3, Introduction to KVM Virtualization
Paravirtualized Frame Buffer

The video output device that drives a video display from a memory buffer con-
taining a complete frame of data for virtual machine displays running in paravir-
tual mode.
VHS
Virtualization Host Server
The physical computer running a SUSE virtualization platform software. The vir-
tualization environment consists of the hypervisor, the host environment, vir-
tual machines, and associated tools, commands, and configuration files. Other
commonly used terms include host, Host Computer, Host Machine (HM), Virtual
Server (VS), Virtual Machine Host (VMH), and VM Host Server (VHS).
VirtFS
313 z 366 2017-05-27 13:21

VirtFS is a new paravirtualized file system interface designed for improving

pass-through technologies in the KVM environment. It is based on the VirtIO
framework.
Virtual Machine
A virtualized PC environment (VM) capable of hosting a guest operating system
and associated applications. Could be also called a VM Guest.
Virtual Machine Manager

A software program that provides a graphical user interface for creating and
managing virtual machines.
Virtualized
A guest operating system or application running on a virtual machine.
Xen
See Chapter 2, Introduction to Xen Virtualization
xl
A set of commands for Xen that lets administrators manage virtual machines
from a command prompt on the host computer. It replaced the deprecated xm
tool stack.
CPU
CPU capping
Virtual CPU capping allows you to set vCPU capacity to 1–100 percent of the
physical CPU capacity.
CPU hotplugging
CPU hotplugging is used to describe the functions of replacing/adding/removing
a CPU without shutting down the system.
CPU over-commitment
Virtual CPU over-commitment is the ability to assign more virtual CPUs to VMs
than the actual number of physical CPUs present in the physical system. This
procedure does not increase the overall performance of the system, but might
be useful for testing purposes.
314 z 366 2017-05-27 13:21

CPU pinning
Processor affinity, or CPU pinning enables the binding and unbinding of a
process or a thread to a central processing unit (CPU) or a range of CPUs.
Network
Bridged Networking
A type of network connection that lets a virtual machine be identified on an ex-
ternal network as a unique identity that is separate from and unrelated to its
host computer.
Empty Bridge
A type of network bridge that has no physical network device or virtual network
device provided by the host. This lets virtual machines communicate with other
virtual machines on the same host but not with the host or on an external net-
work.
External Network
The network outside a host's internal network environment.
Internal Network
A type of network configuration that restricts virtual machines to their host envi-
ronment.
Local Bridge
A type of network bridge that has a virtual network device but no physical net-
work device provided by the host. This lets virtual machines communicate with
the host and other virtual machines on the host. Virtual machines can communi-
cate on an external network through the host.
Network Address Translation (NAT)

A type of network connection that lets a virtual machine use the IP address and
MAC address of the host.
No Host Bridge
A type of network bridge that has a physical network device but no virtual net-
work device provided by the host. This lets virtual machines communicate on an
external network but not with the host. This lets you separate virtual machine
network communications from the host environment.
315 z 366 2017-05-27 13:21

Traditional Bridge
A type of network bridge that has both a physical network device and a virtual
network device provided by the host.
Storage
AHCI
The Advanced Host Controller Interface (AHCI) is a technical standard defined by
Intel* that specifies the operation of Serial ATA (SATA) host bus adapters in a
non-implementation-specific manner.
Block Device
Data storage devices, such as CD-ROM drives or disk drives, that move data in
the form of blocks. Partitions and volumes are also considered block devices.
File-Backed Virtual Disk

A virtual disk based on a file, also called a disk image file.
Raw Disk
A method of accessing data on a disk at the individual byte level instead of
through its file system.
Sparse image file

A disk image file that does not reserve its entire amount of disk space but ex-
pands as data is written to it.
xvda
The drive designation given to the first virtual disk on a paravirtual machine.
Linux Containers
cgroups
Kernel Control Groups (commonly called “cgroups”) are a Kernel feature that al-
lows aggregating or partitioning tasks (processes) and all their children into hier-
archical organized groups to isolate resources.
316 z 366 2017-05-27 13:21

See also Book “System Analysis and Tuning Guide”, Chapter 9 “Kernel Control
Groups”.
chroot
A change root (chroot, or change root jail) is a section in the file system that is
isolated from the rest of the file system. For this purpose, the chroot or
pivot_root command is used to change the root of the file system. A program
that is executed in such a “chroot jail” cannot access files outside the designated
directory tree.
container
Can be seen as a kind of “virtual machine” on the host server that can run any
Linux system, for example openSUSE, SUSE Linux Enterprise Desktop, or SUSE
Linux Enterprise Server. The main difference with a normal virtual machine is
that the container shares its kernel with the host it runs on.
Kernel namespaces
A Kernel feature to isolate some resources like network, users, and others for a
group of processes.
Acronyms
ACPI
Advanced Configuration and Power Interface (ACPI) specification provides an
open standard for device configuration and power management by the operat-
ing system.
AER
Advanced Error Reporting
AER is a capability provided by the PCI Express specification which allows for re-
porting of PCI errors and recovery from some of them.
APIC
Advanced Programmable Interrupt Controller (APIC) is a family of interrupt con-
trollers.
BDF
Bus:Device:Function
Notation used to succinctly describe PCI and PCIe devices.
317 z 366 2017-05-27 13:21

CG
Control Groups
Feature to limit, account and isolate resource usage (CPU, memory, disk I/O,
etc.).
EDF
Earliest Deadline First
This scheduler provides weighted CPU sharing in an intuitive way and uses
real-time algorithms to ensure time guarantees.
EPT
Extended Page Tables
Performance in a virtualized environment is close to that in a native environ-
ment. Virtualization does create some overheads, however. These come from
the virtualization of the CPU, the MMU, and the I/O devices. In some recent x86
processors AMD and Intel have begun to provide hardware extensions to help
bridge this performance gap. In 2006, both vendors introduced their first gener-
ation hardware support for x86 virtualization with AMD-Virtualization (AMD-V)
and Intel® VT-x technologies. Recently Intel introduced its second generation of
hardware support that incorporates MMU-virtualization, called Extended Page
Tables (EPT). EPT-enabled systems can improve performance compared to using
shadow paging for MMU virtualization. EPT increases memory access latencies
for a few workloads. This cost can be reduced by effectively using large pages in
the guest and the hypervisor.
FLASK
Flux Advanced Security Kernel
Xen implements a type of mandatory access control via a security architecture
called FLASK using a module of the same name.
HAP
High Assurance Platform
HAP combines hardware and software technologies to improve workstation and
network security.
HVM
Hardware Virtual Machine (commonly called like this by Xen).
IOMMU
Input/Output Memory Management Unit
IOMMU (AMD* technology) is a memory management unit (MMU) that connects
a direct memory access-capable (DMA-capable) I/O bus to the main memory.
318 z 366 2017-05-27 13:21

KSM
Kernel Same Page Merging
KSM allows for automatic sharing of identical memory pages between guests to
save host memory. KVM is optimized to use KSM if enabled on the VM Host
Server.
MMU
Memory Management Unit
is a computer hardware component responsible for handling accesses to mem-
ory requested by the CPU. Its functions include translation of virtual addresses
to physical addresses (that is, virtual memory management), memory protec-
tion, cache control, bus arbitration and in simpler computer architectures (espe-
cially 8-bit systems) bank switching.
PAE
Physical Address Extension
32-bit x86 operating systems use Physical Address Extension (PAE) mode to en-
able addressing of more than 4 GB of physical memory. In PAE mode, page table
entries (PTEs) are 64 bits in size.
PCID
Process-context identifiers
These are a facility by which a logical processor may cache information for multi-
ple linear-address spaces so that the processor may retain cached information
when software switches to a different linear address space. INVPCID instruction
is used for fine-grained TLB flush, which is benefit for kernel.
PCIe
Peripheral Component Interconnect Express
PCIe was designed to replace older PCI, PCI-X and AGP bus standards. PCIe has
numerous improvements including a higher maximum system bus throughput,
a lower I/O pin count and smaller physical footprint. Moreover it also has a
more detailed error detection and reporting mechanism (AER), and a native hot-
plug functionality. It is also backward compatible with PCI.
PSE and PSE36

Page Size Extended
PSE refers to a feature of x86 processors that allows for pages larger than the
traditional 4 KiB size. PSE-36 capability offers 4 more bits, in addition to the nor-
mal 10 bits, which are used inside a page directory entry pointing to a large
page. This allows a large page to be located in 36-bit address space.
319 z 366 2017-05-27 13:21

PT
Page Table
A page table is the data structure used by a virtual memory system in a com-
puter operating system to store the mapping between virtual addresses and
physical addresses. Virtual addresses are those unique to the accessing process.
Physical addresses are those unique to the hardware (RAM).
QXL
QXL is a cirrus VGA framebuffer (8M) driver for virtualized environment.
RVI or NPT
Rapid Virtualization Indexing, Nested Page Tables
An AMD second generation hardware-assisted virtualization technology for the
processor memory management unit (MMU).
SATA
Serial ATA
SATA is a computer bus interface that connects host bus adapters to mass stor-
age devices such as hard disks and optical drives.
Seccomp2-based sandboxing
Sandboxed environment where only predetermined system calls are permitted
for added protection against malicious behavior.
SMEP
Supervisor Mode Execution Protection
This prevents the execution of user-mode pages by the Xen hypervisor, making
many application-to-hypervisor exploits much harder.
SPICE
Simple Protocol for Independent Computing Environments
SXP
An SXP file is a Xen Configuration File.
TCG
Tiny Code Generator
Instructions are emulated rather than executed by the CPU.
THP
Transparent Huge Pages
This allows CPUs to address memory using pages larger than the default 4 KB.
This helps reduce memory consumption and CPU cache usage. KVM is opti-
mized to use THP (via madvise and opportunistic methods) if enabled on the VM
Host Server.
320 z 366 2017-05-27 13:21

TLB
Translation Lookaside Buffer
TLB is a cache that memory management hardware uses to improve virtual ad-
dress translation speed. All current desktop, notebook, and server processors
use a TLB to map virtual and physical address spaces, and it is nearly always
present in any hardware that uses virtual memory.
VCPU
A scheduling entity, containing each state for virtualized CPU.
VDI
Virtual Desktop Infrastructure
VFIO
Since kernel v3.6; a new method of accessing PCI devices from user space called
VFIO.
VHS
Virtualization Host Server
VM root
VMM will run in VMX root operation and guest software will run in VMX
non-root operation. Transitions between VMX root operation and VMX non-root
operation are called VMX transitions.
VMCS
Virtual Machine Control Structure
VMX non-root operation and VMX transitions are controlled by a data structure
called a virtual-machine control structure (VMCS). Access to the VMCS is man-
aged through a component of processor state called the VMCS pointer (one per
logical processor). The value of the VMCS pointer is the 64-bit address of the
VMCS. The VMCS pointer is read and written using the instructions VMPTRST
and VMPTRLD. The VMM configures a VMCS using the VMREAD, VMWRITE, and
VMCLEAR instructions. A VMM could use a different VMCS for each virtual ma-
chine that it supports. For a virtual machine with multiple logical processors (vir-
tual processors), the VMM could use a different VMCS for each virtual processor.
VMDq
Virtual Machine Device Queue
Multi-queue network adapters exist which support multiple VMs at the hard-
ware level, having separate packet queues associated to the different hosted
VMs (by means of the IP addresses of the VMs).
321 z 366 2017-05-27 13:21

VMM
Virtual Machine Monitor (Hypervisor)
When the processor encounters an instruction or event of interest to the Hyper-
visor (VMM), it exits from guest mode back to the VMM. The VMM emulates the
instruction or other event, at a fraction of native speed, and then returns to
guest mode. The transitions from guest mode to the VMM and back again are
high-latency operations, during which guest execution is completely stalled.
VMX
Virtual Machine eXtensions
VPID
New support for software control of TLB (VPID improves TLB performance with
small VMM development effort).
VT-d
Virtualization Technology for Directed I/O
Like IOMMU for Intel*↗.
vTPM
Component to establish end-to-end integrity for guests via Trusted Computing.
A Virtual Machine Drivers
Virtualization allows the consolidation of workloads on newer, more powerful, en-

ergy-efficient hardware. Paravirtualized operating systems such as SUSE® Linux
Enterprise Server and other Linux distributions are aware of the underlying virtual-
ization platform, and can therefore interact efficiently with it. Unmodified operating
systems such as Microsoft Windows* are unaware of the virtualization platform and
expect to interact directly with the hardware. Because this is not possible when con-
solidating servers, the hardware must be emulated for the operating system. Emula-
tion can be slow, but it is especially troubling for high-throughput disk and network
subsystems. Most performance loss occurs in this area.
322 z 366 2017-05-27 13:21

The SUSE Linux Enterprise Virtual Machine Driver Pack (VMDP) contains 32-bit and
64-bit paravirtualized network, bus and block drivers for several Microsoft Windows
operating systems. These drivers bring many of the performance advantages of par-
avirtualized operating systems to unmodified operating systems because only the
paravirtualized device driver (not the rest of the operating system) is aware of the
virtualization platform. For example, a paravirtualized disk device driver appears as
a normal, physical disk to the operating system. However, the device driver interacts
directly with the virtualization platform (with no emulation) to efficiently deliver disk
access, allowing the disk and network subsystems to operate at near native speeds
in a virtualized environment, without requiring changes to existing operating sys-
tems.
The SUSE® Linux Enterprise Virtual Machine Driver Pack is available as an add-on
product for SUSE Linux Enterprise Server. For detailed information refer to
http://www.suse.com/products/vmdriverpack/↗.
Refer to the Official VMDP Installation Guide at https://www.suse.com

/documentation/sle-vmdp-22/↗ for more information.
B Appendix
B.1.1 Installing virtio Drivers for Microsoft Windows*

SUSE has developed virtio-based drivers for Windows, which are available in the Vir-
tual Machine Driver Pack (VMDP). See http://www.suse.com/products/vmdriverpack/
↗ for more information on the VMDP. Installation instructions are now available in a
dedicated official documentation.
323 z 366 2017-05-27 13:21


To be able to create x509 client and server certificates you need to issue them by a
Certificate Authority (CA). It is recommended to set up an independent CA that only
issues certificates for libvirt .
1. Set up a CA as described in Book “Security Guide”, Chapter 17 “Managing X.509

Certification”, Section 17.2.1 “Creating a Root CA”.
2. Create a server and a client certificate as described in Book “Security Guide”,

Chapter 17 “Managing X.509 Certification”, Section 17.2.4 “Creating or Revoking
User Certificates”. The Common Name (CN) for the server certificate must be
the fully qualified host name, while the Common Name for the client certifi-
cate can be freely chosen. For all other fields stick with the defaults sug-
gested by YaST.
Export the client and server certificates to a temporary location (for exam-
ple, /tmp/x509/ ) by performing the following steps:
a. Select the certificate on the certificates tab.
b. Choose Export › Export to File › Certificate and the Key Unencrypted in

PEM Format, provide the Certificate Password and the full path and the
file name under File Name, for example, /tmp/x509/server.pem or
/tmp/x509/client.pem .
c. Open a terminal and change to the directory where you have saved
the certificate and issue the following commands to split it into certifi-
cate and key (this example splits the server key):
csplit -z -f s_ server.pem '/-----BEGIN/' '{1}'

mv s_00 servercert.pem
mv s_01 serverkey.pem
d. Repeat the procedure for each client and server certificate you want
to export.
3. Finally export the CA certificate by performing the following steps:
a. Switch to the Description tab.
b. Choose Advanced › Export to File › Only the Certificate in PEM Format

and enter the full path and the file name under File Name, for exam-
ple, /tmp/x509/cacert.pem .
324 z 366 2017-05-27 13:21

C XM, XL Toolstacks and Libvirt

framework
C.1 Xen Toolstacks
C.2 Import Xen Domain Configuration into libvirt
C.3 Differences Between the xm and xl Applications
C.4 External links
C.5 Saving a Xen Guest Configuration in an xm Compatible Format
C.1 Xen Toolstacks

Since the early Xen 2.x releases, xend has been the de facto toolstack for managing
Xen installations. In Xen 4.1, a new toolstack called libxenlight (also known as libxl)
was introduced with technology preview status. libxl is a small, low-level library writ-
ten in C. It has been designed to provide a simple API for all client toolstacks (XAPI↗,
libvirt , xl). In Xen 4.2, libxl was promoted to officially supported status and
xend was marked deprecated. xend has been included in the Xen 4.3 and 4.4 se-
ries to give users ample time to convert their tooling to libxl, but it has been re-
moved from the upstream Xen project and will no longer be provided starting with
the Xen 4.5 series and SUSE Linux Enterprise Server 12 SP1.
Although SLES 11 SP3 contains Xen 4.2, SUSE retained the xend toolstack since
making such an invasive change in a service pack would be too disruptive for SUSE
Linux Enterprise customers. However, SLES 12 provides a suitable opportunity to
move to the new libxl toolstack and remove the deprecated, unmaintained xend
stack. Starting with SUSE Linux Enterprise Server 12 SP1, xend is no longer sup-
ported.
325 z 366 2017-05-27 13:21

One of the major differences between xend and libxl is that the former is stateful,
while the latter is stateless. With xend , all client applications such as xm and
libvirt see the same system state. xend is responsible for maintaining state for
the entire Xen host. In libxl, client applications such as xl or libvirt must main-
tain state. Thus domains created with xl or not visible or known to other libxl ap-
plications such as libvirt . Generally, it is discouraged to mix and match libxl ap-
plications and is preferred that a single libxl application be used to manage a Xen
host. In SUSE Linux Enterprise 12 , it is recommended to use libvirt to manage
Xen hosts, allowing management of the Xen system through libvirt applications
such as virt-manager , virt-install , virt-viewer , libguestfs, etc. If xl is
used to manage the Xen host, any virtual machines under its management will not
be accessible to libvirt , and hence not accessible to any of the libvirt applica-
tions.
C.1.1 Upgrading from xend/xm to xl/libxl

The xl application, along with its configuration format (see man xl.cfg ), was de-
signed to be backward-compatible with the xm application and its configuration
format (see man xm.cfg ). Existing xm configuration should be usable with xl .
Since libxl is stateless, and xl does not support the notion of managed domains,
SUSE recommends using libvirt to manage SLES 12 Xen hosts. SUSE has pro-
vided a tool called xen2libvirt , which provides a simple mechanism to import
domains previously managed by xend into libvirt . See Section C.2, “Import Xen
Domain Configuration into libvirt ” for more information on xen2libvirt .
C.1.2 XL design
The basic structure of every xl command is:
xl subcommand OPTIONS DOMAIN
DOMAIN is the numeric domain id, or the domain name (which will be internally
translated to domain id), and OPTIONS are subcommand specific options.
Although xl/libxl was designed to be backward-compatible with xm/xend, there are

a few differences that should be noted:
Managed or persistent domains. libvirt now provides this functionality.
xl/libxl does not support Python code in the domain configuration files.
xl/libxl does not support creating domains from SXP format configuration
files ( xm create -F).
326 z 366 2017-05-27 13:21

xl/libxl does not support sharing storage across DomU's via w! in domain
configuration files.
xl/libxl is relatively new and under heavy development, hence a few features are still
missing with regard to the xm/xend toolstack:
SCSI LUN/Host pass-through (PVSCSI)
USB pass-through (PVUSB)
Support Direct Kernel Boot for fully virtualized Linux guests for Xen does not
work anymore.
C.1.3 Checklist before Upgrade

Before upgrading a SLES 11 SP3 Xen host to SLES 12:
You must remove any Python code from your xm domain configuration files.
It is recommended to capture the libvirt domain XML from all existing virtual
machines using virsh dumpxml DOMAIN_NAME DOMAIN_NAME.xml .
It is recommended to do a backup of /etc/xen/xend-config.sxp and

/boot/grub/menu.lst files to keep references of previous parameters
used for Xen.
Note
Currently, live migrating virtual machines running on a SLES 11 SP3 Xen host to a
SLES 12 Xen host is not supported. The xend and libxl toolstacks are not run-
time-compatible. Virtual machine downtime will be required to move the virtual
machines from SLES 11 SP3 to a SLES 12 host.
C.2 Import Xen Domain Configuration into

libvirt
327 z 366 2017-05-27 13:21

xen2libvirt is a command line tool to import legacy Xen domain configuration

into the libvirt virtualization library (see The Virtualization book for more infor-
mation on libvirt ). xen2libvirt provides an easy way to import domains managed
by the deprecated xm /xend tool stack into the new libvirt /libxl tool stack. Sev-
eral domains can be imported at once using its --recursive mode
xen2libvirt is included in the xen-tools package. If needed, install it with
zypper install xen-tools
xen2libvirt general syntax is
xen2libvirt <options> /path/to/domain/config
where options can be:
-h , --help
Prints short information about xen2libvirt usage.
-c , --convert-only
Converts the domain configuration to the libvirt XML format, but does not
do the import to libvirt .
-r , --recursive
Converts and/or imports all domains configuration recursively, starting at the
specified path.
-f , --format
Specifies the format of the source domain configuration. Can be either xm , or
sexpr (S-expression format).
-v , --verbose
Prints more detailed information about the import process.
EXAMPLE C.1: CONVERTING XEN DOMAIN CONFIGURATION TO libvirt
Suppose you have a Xen domain managed with xm with the following configu-
ration saved in /etc/xen/sle12.xm :
kernel = "/boot/vmlinuz-2.6-xenU"
memory = 128
name = "SLE12"
root = "/dev/hda1 ro"
disk = [ "file:/var/xen/sle12.img,hda1,w" ]
Convert it to libvirt XML without importing it, and look at its content:
328 z 366 2017-05-27 13:21

# xen2libvirt -f xm -c /etc/xen/sle12.xm > /etc/libvirt/qemu/sles12.xml

# cat /etc/libvirt/qemu/sles12.xml
<domain type='xen'>
<name>SLE12</name>
<uuid>43e1863c-8116-469c-a253-83d8be09aa1d</uuid>
<memory unit='KiB'>131072</memory>
<currentMemory unit='KiB'>131072</currentMemory>
<vcpu placement='static'>1</vcpu>
<os>
<type arch='x86_64' machine='xenpv'>linux</type>
<kernel>/boot/vmlinuz-2.6-xenU</kernel>
</os>
<clock offset='utc' adjustment='reset'/>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>restart</on_crash>
<devices>
<disk type='file' device='disk'>
<driver name='file'/>
<source file='/var/xen/sle12.img'/>
<target dev='hda1' bus='xen'/>
</disk>
<console type='pty'>
<target type='xen' port='0'/>
</console>
</devices>
</domain>
To import the domain into libvirt , you can either run the same
xen2libvirt command without the -c option, or use the exported file
/etc/libvirt/qemu/sles12.xml and define a new Xen domain using
virsh :
# sudo virsh define /etc/libvirt/qemu/sles12.xml
C.3 Differences Between the xm and xl

Applications
The purpose of this chapter is to list all differences between xm and xl applica-
tions. Generally, xl is designed to be compatible with xm . Replacing xm with xl
in custom scripts or tools is usually sufficient.
You can also use the libvirt framework using the virsh command. In this docu-
mentation only the first OPTION for virsh will be shown. To get more help on this
option do a:
329 z 366 2017-05-27 13:21

virsh help OPTION
C.3.1 Notation Conventions

To easily understand the difference between xl and xm commands, the following
notation is used in this section:
TABLE C.1: NOTATION CONVENTIONS
Notation Meaning
(-) minus Option exists in xm , but xl does not include it.
(+) plus Option exists in xl , but xm does not include it.
C.3.2 New Global Options

TABLE C.2: NEW GLOBAL OPTIONS
Op-
Task
tions
(+)
Verbose, increase the verbosity of the output
-v
(+)
Dry run, do not actually execute the command
-N
Force execution. xl will refuse to run some commands if it detects that
(+)
xend is also running, this option will force the execution of those com-
-f
mands, even though it is unsafe
C.3.3 Unchanged Options

List of common options of xl and xm , and their libvirt equivalents.
TABLE C.3: COMMON OPTIONS
330 z 366 2017-05-27 13:21

Options Task libvirt equivalent

destroy Immediately terminate the
virsh destroy
DOMAIN domain.
domid
Convert a domain name to
DO‐ virsh domid
a DOMAIN_ID .
MAIN_NAME
domname Convert a DOMAIN_ID to a
virsh domname
DOMAIN_ID DOMAIN_NAME .
Display the short help
help message (that is, common virsh help
commands).
Pause a domain. When in a
paused state, the domain
will still consume allocated
pause
resources such as memory, virsh suspend
DOMAIN_ID
but will not be eligible for
scheduling by the Xen hy-
pervisor.
Move a domain out of the
paused state. This will al-
unpause low a previously paused
virsh resume
DOMAIN_ID domain to be eligible for
scheduling by the Xen hy-
pervisor.
1. virsh dumpxml DOMAINNAME >

DOMXML
rename Change the domain name 2. modify the domain's name in

DOMAIN_ID DOMXML
of DOMAIN_ID to
NEW_DO‐
MAIN_NAME NEW_DOMAIN_NAME . 3. virsh undefine DOMAINNAME
4. virsh define DOMAINNAME
sysrq
virsh send-keys can send Magic
DOMAIN <let-
Sys Req only for KVM
ter>
331 z 366 2017-05-27 13:21


Send a Magic System Re-
quest to the domain, each
type of request is repre-
sented by a different letter.
It can be used to send
SysRq requests to Linux
guests, see sysrq.txt in
your Linux Kernel sources
for more information. It re-
quires PV drivers to be in-
stalled in your guest OS.
vncviewer Attach to domain's VNC
virt-view DOMAIN_ID
OPTIONS server, forking a vncviewer
virsh vncdisplay
DOMAIN process.
Enable the vcpu-count vir-
tual CPUs for the domain
in question. Like mem-set
vcpu-set
, this command can only
DOMAIN_ID virsh setvcpus
allocate up to the maxi-
<vCPUs>
mum virtual CPU count
configured at boot for the
domain.
List VCPU information for a
specific domain. If no do-
vcpu-list
main is specified, VCPU in- virsh vcpuinfo
DOMAIN_ID
formation for all domains
will be provided.
Pin the VCPU to only run
vcpu-pin
on the specific CPUs. The
DOMAIN_ID
keyword all can be used to virsh vcpupin
<VCPU|all>
apply the CPU list to all VC-
<CPUs|all>
PUs in the domain.
Read the Xen message buf-
fer, similar to dmesg on a
Linux system. The buffer
dmesg [-c] contains informational,
warning, and error mes-
sages created during Xen's
boot process.
332 z 366 2017-05-27 13:21


Execute the xentop com-
mand, which provides real
virsh nodecpustats
top time monitoring of do-
virsh nodememstats
mains. xentop is a curses
interface.
Print the current uptime of
the domains running. With
uptime [-s]
the xl command, the
DOMAIN
DOMAIN argument is
mandatory.
Send debug keys to Xen. It
is the same as pressing the
debug-keys
Xen conswitch (Ctrl-A by de-
KEYS
fault) three times and then
pressing "keys".
cpupool-mi-
Move a domain specified
grate
by DOMAIN_ID or
DOMAIN
DOMAIN into a CPU_POOL .
CPU_POOL
Deactivate a cpu pool. This
cpupool-de-
is possible only if no do-
stroy
main is active in the
CPU_POOL
cpu-pool.
Detach a domain's virtual
block device. devid may be
the symbolic name or the
block-detach
numeric device id given to
DOMAIN_ID virsh detach-disk
the device by Dom0. You
DevId
will need to run xl
block-list to determine
that number.
Create a new network de-
network-at- vice in the domain speci-
tach fied by DOMAIN_ID . net-
DOMAIN_ID work-device describes the virsh attach-interface
NET‐ device to attach, using the virsh attach-device
WORK_DE‐ same format as the vif
VICE string in the domain con-
figuration file
333 z 366 2017-05-27 13:21


Hotplug a new
pci-attach pass-through PCI device to
DOMAIN the specified domain. BDF
virsh attach-device
<BDF> [Vir- is the PCI Bus/Device/Func-
tual Slot] tion of the physical device
to be passed through.
pci-list List pass-through PCI de-
DOMAIN_ID vices for a domain
Determine if the FLASK se-
getenforce curity module is loaded
and enforcing its policy.
Enable or disable enforc-
ing of the FLASK access
setenforce controls. The default is
<1|0|En‐ permissive and can be
forc‐
changed using the
ing|Per‐
missive> flask_enforcing option on
the hypervisor's command
line.
C.3.4 Removed Options

List of xm options which are no more available with the XL tool stack and a re-
placement solution if available.
C.3.4.1 Domain Management

The list of Domain management removed command and their replacement.
TABLE C.4: DOMAIN MANAGEMENT REMOVED OPTIONS
334 z 366 2017-05-27 13:21

Domain Management Removed Options

Options Task Equivalent
This log file can
be found in
(-) log Print out the Xend log.
/var/log
/xend.log
Remove a domain from Xend domain manage-
virsh
(-) delete ment. The list option shows the domain
undefine
names
virsh
(-) new Adds a domain to Xend domain management
define
Start a Xend managed domain that was added us-
(-) start virsh start
ing the xm new command
Dry run - prints the resulting configuration in SXP
(-) dryrun xl -N
but does not create the domain
(-) reset Reset a domain virsh reset
virsh
(-) domstate Show domain state
domstate
(-) serve Proxy Xend XMLRPC over stdio
(-) resume
Moves a domain out of the suspended state and virsh
DOMAIN
back into memory resume
OPTIONS
Suspend a domain to a state file so that it can be virsh
(-) suspend later resumed using the resume subcommand. managedsave
DOMAIN Similar to the save subcommand although the virsh
state file may not be specified suspend
C.3.4.2 USB Devices

USB options are not available with xl/libxl tool stack. virsh has the
attach-device and detach-device options but it does not work yet with USB .
TABLE C.5: USB DEVICES MANAGEMENT REMOVED OPTIONS
335 z 366 2017-05-27 13:21

USB Devices Management Removed Options

Options Task
(-) usb-add Add a new USB physical bus to a domain
(-) usb-del Delete a USB physical bus from a domain
Attach a new USB physical bus to domain's vir-
(-) usb-attach
tual port
Detach a USB physical bus from domain's vir-
(-) usb-detach
tual port
List domain's attachment state of all virtual
(-) usb-list
port
(-)
usb-list-assignable-de‐ List all the assignable USB devices
vices
Create a domain's new virtual USB host con-
(-) usb-hc-create
troller
(-) usb-hc-destroy Destroy a domain's virtual USB host controller
C.3.4.3 CPU Management

CPU management options has changed. New options are available, see: Sec-
tion C.3.5.10, “ xl cpupool-* ”
TABLE C.6: CPU MANAGEMENT REMOVED OPTIONS
CPU Management Removed Options

Options Task
(-) cpupool-new Adds a CPU pool to Xend CPU pool management
(-) cpupool-start Starts a Xend CPU pool
(-) cpupool-delete Removes a CPU pool from Xend management
C.3.4.4 Other Options
TABLE C.7: OTHER OPTIONS
336 z 366 2017-05-27 13:21

Other Removed Options

Options Task
(-) shell Launch an interactive shell
(-) change-vnc-passwd Change vnc password
(-) vtpm-list List virtual TPM devices
(-) block-configure Change block device configuration
C.3.5 Changed Options
C.3.5.1 create
xl create CONFIG_FILE OPTIONS VARS
Note: libvirt Equivalent:
virsh create
TABLE C.8: xl create CHANGED OPTIONS
create Changed Options

Options Task
(*) -f= FILE , --defconfig= FILE Use the given configuration file
TABLE C.9: xm create REMOVED OPTIONS
337 z 366 2017-05-27 13:21

create Removed Options

Options Task
Skip DTD checking - skips checks on XML before cre-
(-) -s , --skipdtd
ating
(-) -x , --xmldryrun XML dry run
(-)
-F= FILE , --config= Use the given SXP formatted configuration script
FILE
(-) --path Search path for configuration scripts
Print the available configuration variables (vars) for
(-) --help_config
the configuration script
Dry run — prints the configuration in SXP but does
(-) -n , --dryrun
not create the domain
(-) -c ,
Connect to the console after the domain is created
--console_autoconnect
(-) -q , --quiet Quiet mode
(-) -p , --paused Leave the domain paused after it is created
TABLE C.10: xl create ADDED OPTIONS
create Added Options

Options Task
Attach to domain's VNC server, forking a
(+) -V , --vncviewer
vncviewer process
(+) -A ,
Pass VNC password to vncviewer via stdin
--vncviewer-autopass
C.3.5.2 console
xl console OPTIONS DOMAIN
Note: libvirt Equivalent
virsh console
TABLE C.11: xl console ADDED OPTIONS
338 z 366 2017-05-27 13:21

console Added Option

Option Task
Connect to a PV console or connect to an emulated serial console.
(+) -t [pv|se-
PV consoles are the only consoles available for PV domains while
rial]
HVM domains can have both
C.3.5.3 info
xl info
TABLE C.12: xm info REMOVED OPTIONS
info Removed Options

Options Task
(-) -n , --numa Numa info
(-) -c , --config List Xend configuration parameters
C.3.5.4 dump-core
xl dump-core DOMAIN FILENAME
virsh dump
TABLE C.13: xm dump-core REMOVED OPTIONS
dump-core Removed Options

Options Task
(-) -L , --live Dump core without pausing the domain
(-) -C , --crash Crash domain after dumping core
(-) -R , --reset Reset domain after dumping core
C.3.5.5 list
xl list options DOMAIN
339 z 366 2017-05-27 13:21

virsh list --all
TABLE C.14: xm list REMOVED OPTIONS
list Removed Options

Options Task
(-) -l , --long The output for xm list presents the data in SXP format
(-) --state== STATE Output information for VMs in the specified state
TABLE C.15: xl list ADDED OPTIONS
list Added Options

Options Task
(+) -Z ,
Also prints the security labels
--context
(+) -v , Also prints the domain UUIDs, the shutdown reason and secu-
--verbose rity labels
C.3.5.6 mem-*
virsh setmem
virsh setmaxmem
TABLE C.16: xl mem-* CHANGED OPTIONS
mem-* Changed Options

Options Task
Appending t for terabytes, g for gigabytes, m for
mem-max DOMAIN_ID
megabytes, k for kilobytes and b for bytes. Specify the
MEM
maximum amount of memory the domain can use.
mem-set DOMAIN_ID
Set the domain's used memory using the balloon driver
MEM
340 z 366 2017-05-27 13:21

C.3.5.7 migrate
xl migrate OPTIONS DOMAIN HOST
virsh migrate --live hvm-sles11-qcow2 xen+ CONNECTOR :// USER @

IP_ADDRESS /
TABLE C.17: xm migrate REMOVED OPTIONS
migrate Removed Options

Options Task
Use live migration. This will migrate the domain be-
(-) -l , --live
tween hosts without shutting down the domain
(-) -r , --resource Mbs Set maximum Mbs allowed for migrating the domain
(-) -c ,
Change home server for managed domains
--change_home_server
(-) --max_iters=
Number of iterations before final suspend (default:30)
MAX_ITERS
(-) --max_factor= Max amount of memory to transfer before final sus-
MAX_FACTOR pend (default: 3*RAM).
(-) --min_remaining= Number of dirty pages before final suspend (de-
MIN_REMAINING fault:50)
(-) --abort_if_busy Abort migration instead of doing final suspend
(-) --log_progress Log progress of migration to xend.log
(-) -s , --ssl Use ssl connection for migration
TABLE C.18: xl migrate ADDED OPTIONS
migrate Added Options

Options Task
(+) -s SSHCOMMAND Use <sshcommand> instead of ssh
On the new host, do not wait in the background (on
(+) -e
<host>) for the death of the domain
Send <config> instead of a configuration file from cre-
(+) -C config
ation
341 z 366 2017-05-27 13:21

C.3.5.8 Domain Management

xl reboot OPTIONS DOMAIN
virsh reboot
TABLE C.19: xm reboot REMOVED OPTIONS
reboot Removed Options

Options Task
(-) -a , --all Reboot all domains
(-) -w , Wait for reboot to complete before returning. This may take a
--wait while, as all services in the domain need to be shut down cleanly
TABLE C.20: xl reboot ADDED OPTIONS
reboot Added Options

Option Task
Fallback to ACPI reset event for HVM guests with
(+) -F
no PV drivers
xl save OPTIONS DOMAIN CHECK_POINT_FILE CONFIG_FILE
virsh save
TABLE C.21: xl save ADDED OPTIONS
save Added Options

Option Task
Leave domain running after creating the snap-
(+) -c
shot
xl restore OPTIONS CONFIG_FILE CHECK_POINT_FILE
342 z 366 2017-05-27 13:21

virsh restore
TABLE C.22: xl restore ADDED OPTIONS
restore Added Options

Options Task
(+) -p Do not unpause domain after restoring it
Do not wait in the background for the death of
(+) -e
the domain on the new host
(+) -d Enable debug messages
Attach to domain's VNC server, forking a
(+) -V , --vncviewer
vncviewer process
(+) -A ,
Pass VNC password to vncviewer via stdin
--vncviewer-autopass
xl shutdown OPTIONS DOMAIN
virsh shutdown
TABLE C.23: xm shutdown REMOVED OPTIONS
shutdown Removed Options

Options Task
(-) -w , --wait Wait for the domain to complete shutdown before returning
(-) -a Shutdown all guest domains
(-) -R
(-) -H
TABLE C.24: xl shutdown ADDED OPTIONS
343 z 366 2017-05-27 13:21

shutdown Added Options

Option Task
If the guest does not support PV shutdown con-
(+) -F trol then fallback to sending an ACPI power
event
TABLE C.25: xl trigger CHANGED OPTIONS
trigger Changed Options

Option Task
trigger DOMAIN <nmi|re-
Send a trigger to a domain. Only available for
set|init|power|sleep|s3re-
HVM domains
sume> VCPU
C.3.5.9 xl sched-*
xl sched-credit OPTIONS
virsh schedinfo
TABLE C.26: xm sched-credit REMOVED OPTIONS
sched-credit Removed Options

Options Task
-d DOMAIN ,
Domain
--domain= DOMAIN
A domain with a weight of 512 will get twice as much CPU
-w WEIGHT , as a domain with a weight of 256 on a contended host.
--weight= WEIGHT Legal weights range from 1 to 65535 and the default is
256
The CAP optionally fixes the maximum amount of CPU a
-c CAP , --cap= CAP
domain can consume
TABLE C.27: xl sched-credit ADDED OPTIONS
344 z 366 2017-05-27 13:21

sched-credit Added Options

Options Task
(+) -p CPUPOOL , --cpupool= Restrict output to domains in the specified
CPUPOOL cpupool
Specify to list or set pool-wide scheduler param-
(+) -s , --schedparam
eters
(+) -t TSLICE , Timeslice tells the scheduler how long to allow
--tslice_ms= TSLICE VMs to run before pre-empting
(+) -r RLIMIT , Ratelimit attempts to limit the number of sched-
--ratelimit_us= RLIMIT ules per second
xl sched-credit2 OPTIONS
Note: libvirt Status
virsh only supports credit scheduler, not credit2 scheduler
TABLE C.28: xm sched-credit2 REMOVED OPTIONS
sched-credit2 Removed Options

Options Task
-d DOMAIN , --domain=
Domain
DOMAIN
-w WEIGHT , --weight= Legal weights range from 1 to 65535 and the de-
WEIGHT fault is 256
TABLE C.29: xl sched-credit2 ADDED OPTIONS
sched-credit2 Added Options

Option Task
(+) -p CPUPOOL , --cpupool= Restrict output to domains in the specified
CPUPOOL cpupool
xl sched-sedf OPTIONS
TABLE C.30: xm sched-sedf REMOVED OPTIONS
345 z 366 2017-05-27 13:21

sched-sedf Removed Options

Options Task
The normal EDF scheduling usage in millisec-
-p PERIOD , --period= PERIOD
onds
The normal EDF scheduling usage in millisec-
-s SLICE , --slice= SLICE
onds
-l LATENCY , --latency=
Scaled period if domain is doing heavy I/O
LATENCY
Flag for allowing domain to run in extra time
-e EXTRA , --extra= EXTRA
(0 or 1)
-w WEIGHT , --weight= WEIGHT Another way of setting CPU slice
TABLE C.31: xl sched-sedf ADDED OPTIONS
sched-sedf Added Options

Options Task
(+) -c CPUPOOL , --cpupool= Restrict output to domains in the specified
CPUPOOL cpupool
(+) -d DOMAIN , --domain=
Domain
DOMAIN
C.3.5.10 xl cpupool-*
xl cpupool-cpu-remove CPU_POOL <CPU nr>|node:<node nr>
xl cpupool-list [-c|--cpus] CPU_POOL
TABLE C.32: xm cpupool-list REMOVED OPTIONS
cpupool-* Removed Options

Option Task
(-) -l , --long Output all CPU pool details in SXP format
xl cpupool-cpu-add CPU_POOL cpu-nr|node:node-nr
xl cpupool-create OPTIONS CONFIG_FILE [Variable=Value ...]
TABLE C.33: xm cpupool-create REMOVED OPTIONS
346 z 366 2017-05-27 13:21

cpupool-create Removed Options

Options Task
(-) -f FILE , --def- Use the given Python configuration script. The configuration
config= FILE script is loaded after arguments have been processed
Dry run - prints the resulting configuration in SXP but does
(-) -n , --dryrun
not create the CPU pool
Print the available configuration variables (vars) for the con-
(-) --help_config
figuration script
Search path for configuration scripts. The value of PATH is a
(-) --path= PATH
colon-separated directory list
(-) -F= FILE , --con-
CPU pool configuration to use (SXP)
fig= FILE
C.3.5.11 PCI and Block Devices

xl pci-detach [-f] DOMAIN_ID <BDF>
virsh detach-device
TABLE C.34: xl pci-detach ADDED OPTIONS
pci-detach Added Options

Op-
Task
tion
(+) If -f is specified, xl is going to forcefully remove the device even without
-f guest's collaboration
TABLE C.35: xm block-list REMOVED OPTIONS
block-list Removed Options

Option Task
(-) -l , --long List virtual block devices for a domain
TABLE C.36: OTHER OPTIONS
347 z 366 2017-05-27 13:21

Option libvirt equivalent

xl block-attach DOMAIN <disk-spec-com- virsh
ponent(s)> attach-disk/attach-device
xl block-list DOMAIN_ID virsh domblklist
C.3.5.12 Network
TABLE C.37: NETWORK OPTIONS
Option libvirt equivalent

xl network-list DOMAIN(s) virsh domiflist
xl network-detach DOMAIN_ID de-
virsh detach-interface
vid|mac
virsh
xl network-attach DOMAIN(s)
attach-interface/attach-device
TABLE C.38: xl network-attach REMOVED OPTIONS
Removed Options
Option Task
(-) -l , --long
C.3.6 New Options

TABLE C.39: NEW OPTIONS
348 z 366 2017-05-27 13:21

Options Task
Update the saved configuration for a running do-
main. This has no immediate effect but will be ap-
config-update DOMAIN
plied when the guest is next restarted. This com-
CONFIG_FILE OPTIONS
mand is useful to ensure that runtime modifications
VARS
made to the guest will be preserved when the guest
is restarted
migrate-receive
List count of shared pages.List specifically for that do-
sharing DOMAIN
main. Otherwise, list for all domains
Prints information about guests. This list excludes in-
vm-list formation about service or auxiliary domains such as
Dom0 and stubdoms
cpupool-rename
Renames a cpu-pool to newname
CPU_POOL NEWNAME
Splits up the machine into one cpu-pool per numa
cpupool-numa-split
node
Insert a CD-ROM into a guest domain's existing vir-
cd-insert DOMAIN <Virtu-
tual CD drive. The virtual drive must already exist but
alDevice> <type:path>
can be current empty
cd-eject DOMAIN <Virtu- Eject a CD-ROM from a guest's virtual CD drive. Only
alDevice> works with HVM domains
List all the assignable PCI devices. These are devices
in the system which are configured to be available
pci-assignable-list
for pass-through and are bound to a suitable PCI
back-end driver in Dom0 rather than a real driver
Make the device at PCI Bus/Device/Function BDF as-
pci-assignable-add
signable to guests.This will bind the device to the
<BDF>
pciback driver
Make the device at PCI Bus/Device/Function BDF as-
pci-assignable-remove
signable to guests. This will at least unbind the device
OPTIONS <BDF>
from pciback
Load FLASK policy from the given policy file. The ini-
tial policy is provided to the hypervisor as a multi-
loadpolicy
boot module; this command allows runtime updates
POLICY_FILE
to the policy. Loading new security policy will reset
runtime changes to device labels
349 z 366 2017-05-27 13:21

C.4 External links

For more information on Xen tool stacks refer to the following online resources:
XL in Xen
XL in Xen 4.2↗
xl command
XL↗ command line.
xl.cfg
xl.cfg↗ domain configuration file syntax.
xl disk
xl disk↗ configuration option.
XL vs Xend
XL vs Xend↗ feature comparison.
BDF doc
BDF documentation↗.
libvirt
virsh↗ command.
C.5 Saving a Xen Guest Configuration in an xm

Compatible Format
Although xl is now the current toolkit for managing Xen guests (apart from the
preferred libvirt ), you may need to export the guest configuration to the previ-
ously used xm format. To do this, follow these steps:
1. First export the guest configuration to a file:
virsh dumpxml guest_id > guest_cfg.xml
2. Then convert the configuration to the xm format:
virsh domxml-to-native xen-xm guest_cfg.xml > guest_xm_cfg
350 z 366 2017-05-27 13:21

D Documentation Updates
D.1 November 2016 (Release of SUSE Linux Enterprise Server 12 SP2)
D.2 March 2016 (Maintenance Release of SUSE Linux Enterprise Server 12
SP1)
D.3 December 2015 (Initial Release of SUSE Linux Enterprise Server 12 SP1)
D.4 February 2015 (Documentation Maintenance Update)
D.5 October 2014 (Initial Release of SUSE Linux Enterprise Server 12)
This chapter lists content changes for this document.
This manual was updated on the following dates:
Section D.1, “November 2016 (Release of SUSE Linux Enterprise Server 12

SP2)”
Section D.2, “March 2016 (Maintenance Release of SUSE Linux Enterprise

Server 12 SP1)”
Section D.3, “December 2015 (Initial Release of SUSE Linux Enterprise Server
12 SP1)”
Section D.4, “February 2015 (Documentation Maintenance Update)”
Section D.5, “October 2014 (Initial Release of SUSE Linux Enterprise Server
12)”
D.1 November 2016 (Release of SUSE Linux

Enterprise Server 12 SP2)
351 z 366 2017-05-27 13:21

General
The e-mail address for documentation feedback has changed to

doc-team@suse.com .
The documentation for Docker has been enhanced and renamed to

Docker Guide.
General Changes to this Guide
Added tip about changing the default boot kernel in Section 6.2, “In-
stalling Xen”.
Moved and adjusted two network sections from Chapter 19, Virtual Net-
working to Section 13.2.3, “Using VLAN Interfaces” and Section 13.2.1.1,
“Adding a Network Bridge”.
Removed the deprecated PCI and USB Pass-through related snippets in

favor of VFIO (Fate #320350).
Removed the deprecated tool vm-install across the whole guide (Fate
#319659).
Updated parts of the guide that refer to Virtual Machine Manager and
virt-viewer to work with versions 1.4 and 4.0 of those software pack-
ages, respectively (Fate #319990).
Chapter 5, Virtualization Tools
In Section 5.2, “Virtualization GUI Tools” add note to disable automatic

USB device redirection.
Chapter 7, Supported Guests, Hosts and Features
Added PowerKVM and updated list in Section 7.2, “ Supported VM Host

Servers for SUSE Linux Enterprise Server 12 SP2 VM Guests ”
(https://bugzilla.suse.com/show_bug.cgi?id=984119↗ and Fate
#320424).
Updated list in Section 7.1, “Supported VM Guests” (Fate #320426).
Chapter 10, Basic VM Guest Management
Updated list in Section 10.7.1, “Migration Requirements” (Fate #320425).
Chapter 11, Connecting and Authorizing
352 z 366 2017-05-27 13:21

Fixed usermod options (https://bugzilla.suse.com

/show_bug.cgi?id=983983↗).
Chapter 13, Managing Networks
Extended the <dhcp> element with <host> for assigning the predefined
IP and host name in Example 13.1, “NAT Based Network” (Fate #320265).
Added Section 13.1.1.5, “Obtaining IP Addresses with nsswitch for NAT

Networks (in KVM)” (Fate #316628).
Added more commands to obtain addresses in Section 13.1.2.2, “Listing

Networks” (Fate #316628).
Chapter 14, Configuring Virtual Machines
Fixed loading igb driver Section 14.11.2, “Loading and Configuring the
SR-IOV Host Drivers”.
Chapter 16, VM Guest Clock Settings
Use kvm-clock together with NTP (Section 16.1, “KVM: Using kvm_clock
”, https://bugzilla.suse.com/show_bug.cgi?id=994806↗).
Chapter 17, libguestfs
New section (Section 17.3.7, “Converting a Physical Machine into a KVM

Guest”, Fate #316274).
Chapter 18, Setting Up a Virtual Machine Host
New section (Section 18.6, “USB Pass-Through with PVUSB”, Fate

#316612).
Chapter 20, Managing a Virtualization Environment
Added Section 20.4, “Time Stamp Counter”.
Chapter 21, Block Devices in Xen
Added Section 21.2, “Mapping Network Storage to Virtual Disk” (Fate

#318801).
Added Section 21.5, “Scripts for Managing Advanced Storage Scenarios”

(Fate #319810).
353 z 366 2017-05-27 13:21

Chapter 29, Running Virtual Machines with qemu-system-ARCH
Added Section 28.2.2.5, “Advanced Options for the qcow2 File Format”.
Added Section 29.3.1.4, “Accessing iSCSI Resources Directly” (Fate

#320080).
Bugfixes
Updated the network bridge invocation command lines in Sec-

tion 29.4.3.2, “Connecting to a Bridge with qemu-bridge-helper”
(https://bugzilla.suse.com/show_bug.cgi?id=983917↗).
Added PVHVM virtualization mode in Section 1.5, “I/O Virtualization”

Section 30.10, “Live Migration”

postcopy is a technology preview.
D.2 March 2016 (Maintenance Release of SUSE

Linux Enterprise Server 12 SP1)
Added Windows 10 to the list, added latest RHEL versions

Added missing supported platform KVM for IBM z Systems 1.1.0

D.3 December 2015 (Initial Release of SUSE

Linux Enterprise Server 12 SP1)
General
Book “Subscription Management Tool for SLES 12 SP2” is now part of the
documentation for SUSE Linux Enterprise Server.
354 z 366 2017-05-27 13:21

Add-ons provided by SUSE have been renamed as modules and exten-

sions. The manuals have been updated to reflect this change.
Numerous small fixes and additions to the documentation, based on

technical feedback.
The registration service has been changed from Novell Customer Center
to SUSE Customer Center.
In YaST, you will now reach Network Settings via the System group. Net-
work Devices is gone (https://bugzilla.suse.com/show_bug.cgi?id=867809
↗).
General
Added Chapter 13, Managing Networks.
Rewrote large parts of the PCI Pass-Through documentation since KVM

PCI Pass-Through has been replaced by VFIO.
Updated the lists of supported VM Guests and VM Host Servers (Fate

#319284 and #319285).
Refined the information about supported live migration scenarios in Sec-

tion 10.7.1, “Migration Requirements” (Fate #319283).
Added Section 10.8.2, “Monitoring with virt-top ” (Fate #319422).
Chapter 12, Managing Storage
Added Section 12.5, “ Sharing Directories between Host and Guests (File
System Pass-Through) ”.
Added a note on how to add SR-IOV devices on VM Guest creation to

Section 14.11, “Adding SR-IOV Devices” (Fate #315577).
Chapter 19, Virtual Networking
Fixed unclear wording in Procedure 19.1, “Configuring a routed IPv4 VM

Guest” (doc comment 28137).
355 z 366 2017-05-27 13:21

Chapter 23, Administrative Tasks
Added new list item for hosts in different subnets in Section 23.3, “Mi-
grating Xen VM Guest Systems” (Fate #303927).
Chapter 27, Setting Up a KVM VM Host Server
Added the whole chapter and moved virtio related KVM content to a
subsidiary Section 27.3, “KVM Host-Specific Features”.
Added support for e1000 network card variants in Section 29.4.1, “Defin-
ing a Network Interface Card”.
Updated the NIC command line specification to the preferred -netdev

-device in Section 29.4.1, “Defining a Network Interface Card”.
Bugfixes
VFIO related improvements to Virtualization Guide

Added a snippet recommending only one watchdog device at a time in

Section 18.1, “Best Practices and Suggestions” (https://bugzilla.suse.com
virsh Console Configuration Is Not Documented

Wrong Table Title (https://bugzilla.suse.com/show_bug.cgi?id=938671↗).
Missing Word "Virtualized" (https://bugzilla.suse.com

Added Section 12.5, “ Sharing Directories between Host and Guests (File
System Pass-Through) ” (https://bugzilla.suse.com
D.4 February 2015 (Documentation

Maintenance Update)
356 z 366 2017-05-27 13:21

Bugfixes
FIPS: starting of openvn client on SLES 12 fails...

Misleading Information Related to Supported Virtualization Host (Citrix

XenServer) (https://bugzilla.suse.com/show_bug.cgi?id=912700↗).
qemu-bridge-helper Location (https://bugzilla.suse.com

Virtualization Support Statement of OES 11 SP2

D.5 October 2014 (Initial Release of SUSE Linux

Enterprise Server 12)
General
Removed all KDE documentation and references because KDE is no

longer shipped.
Removed all references to SuSEconfig, which is no longer supported

(Fate #100011).
Move from System V init to systemd (Fate #310421). Updated affected

parts of the documentation.
YaST Runlevel Editor has changed to Services Manager (Fate #312568).

Updated affected parts of the documentation.
Removed all references to ISDN support, as ISDN support has been re-
moved (Fate #314594).
Removed all references to the YaST DSL module as it is no longer

shipped (Fate #316264).
Removed all references to the YaST Modem module as it is no longer

shipped (Fate #316264).
Btrfs has become the default file system for the root partition (Fate
#315901). Updated affected parts of the documentation.
The dmesg now provides human-readable time stamps in ctime() -like

format (Fate #316056). Updated affected parts of the documentation.
357 z 366 2017-05-27 13:21

syslog and syslog-ng have been replaced by rsyslog (Fate #316175). Up-
dated affected parts of the documentation.
MariaDB is now shipped as the relational database instead of MySQL

(Fate #313595). Updated affected parts of the documentation.
SUSE-related products are no longer available from

http://download.novell.com↗ but from http://download.suse.com↗. Ad-
justed links accordingly.
Novell Customer Center has been replaced with SUSE Customer Center.
Updated affected parts of the documentation.
/var/run is mounted as tmpfs (Fate #303793). Updated affected parts

of the documentation.
The following architectures are no longer supported: IA64 and x86. Up-
dated affected parts of the documentation.
The traditional method for setting up the network with ifconfig has
been replaced by wicked . Updated affected parts of the documenta-
tion.
A lot of networking commands are deprecated and have been replaced

by newer commands (usually ip ). Updated affected parts of the docu-
mentation.
arp : ip neighbor
ifconfig : ip addr , ip link
iptunnel : ip tunnel
iwconfig : iw
nameif : ip link , ifrename
netstat : ss , ip route , ip -s link , ip maddr
route : ip route
Numerous small fixes and additions to the documentation, based on

technical feedback.
Part I, “Introduction”
Added Chapter 7, Supported Guests, Hosts and Features.
Updated Section 7.1, “Supported VM Guests”.
Updated Section 7.2, “ Supported VM Host Servers for SUSE Linux

Enterprise Server 12 SP2 VM Guests ”.
Chapter 8, Starting and Stopping libvirtd
358 z 366 2017-05-27 13:21

Added introductory paragraph on virt-install .
Chapter 9, Guest Installation
Updated Section 9.1, “GUI-Based Guest Installation” to match SUSE Linux

Enterprise Server 12 state.
Added Section 9.2, “Installing from the Command Line with

virt-install ”.
Chapter 12, Managing Storage
Added Section 12.4, “Online Resizing of Guest Block Devices”.
Added Section 14.11.4, “Dynamic Allocation of VFs from a Pool”.
Added tip in Section 10.4.2, “Saving and Restoring with virsh ”.
Part IV, “Managing Virtual Machines with Xen”
Migrated from Xend/xm toolkit to xl.
Chapter 22, Virtualization: Configuration Options and Settings
Added Section 22.6.3, “Increasing the Number of PCI-IRQs”.
Chapter 28, Guest Installation
Added a tip on nocow option in Section 28.2.2, “Creating, Converting

and Checking Disk Images”.
Added Section 29.3.1.3, “Bio-Based I/O Path for virtio-blk”.
Added Section 27.3.3, “Scaling Network Performance with Multiqueue

virtio-net”.
Added Section 29.3.1.1, “Freeing Unused Guest Disk Space”.
359 z 366 2017-05-27 13:21

Chapter 30, Virtual Machine Administration Using QEMU Monitor
Added device_add and device_del commands.
Added Section 30.11, “QMP - QEMU Machine Protocol”.
Part VI, “Managing Virtual Machines with LXC”
Added Chapter 32, Migration from LXC to libvirt-lxc .
Bugfixes
E GNU Licenses
E.1 GNU Free Documentation License
This appendix contains the GNU Free Documentation License version 1.2.
GNU Free Documentation License

Copyright (C) 2000, 2001, 2002 Free Software Foundation, Inc. 51 Franklin St, Fifth Floor, Boston, MA 02110-1301
USA. Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is
not allowed.
0. PREAMBLE
The purpose of this License is to make a manual, textbook, or other functional and useful document "free" in
the sense of freedom: to assure everyone the effective freedom to copy and redistribute it, with or without
modifying it, either commercially or non-commercially. Secondarily, this License preserves for the author and
publisher a way to get credit for their work, while not being considered responsible for modifications made by
others.
360 z 366 2017-05-27 13:21

This License is a kind of "copyleft", which means that derivative works of the document must themselves be
free in the same sense. It complements the GNU General Public License, which is a copyleft license designed
for free software.
We have designed this License to use it for manuals for free software, because free software needs free docu-
mentation: a free program should come with manuals providing the same freedoms that the software does.
But this License is not limited to software manuals; it can be used for any textual work, regardless of subject
matter or whether it is published as a printed book. We recommend this License principally for works whose
purpose is instruction or reference.
1. APPLICABILITY AND DEFINITIONS

This License applies to any manual or other work, in any medium, that contains a notice placed by the copy-
right holder saying it can be distributed under the terms of this License. Such a notice grants a world-wide,
royalty-free license, unlimited in duration, to use that work under the conditions stated herein. The "Docu-
ment", below, refers to any such manual or work. Any member of the public is a licensee, and is addressed as
"you". You accept the license if you copy, modify or distribute the work in a way requiring permission under
copyright law.
A "Modified Version" of the Document means any work containing the Document or a portion of it, either
copied verbatim, or with modifications and/or translated into another language.
A "Secondary Section" is a named appendix or a front-matter section of the Document that deals exclusively
with the relationship of the publishers or authors of the Document to the Document's overall subject (or to re-
lated matters) and contains nothing that could fall directly within that overall subject. (Thus, if the Document is
in part a textbook of mathematics, a Secondary Section may not explain any mathematics.) The relationship
could be a matter of historical connection with the subject or with related matters, or of legal, commercial,
philosophical, ethical or political position regarding them.
The "Invariant Sections" are certain Secondary Sections whose titles are designated, as being those of Invariant
Sections, in the notice that says that the Document is released under this License. If a section does not fit the
above definition of Secondary then it is not allowed to be designated as Invariant. The Document may contain
zero Invariant Sections. If the Document does not identify any Invariant Sections then there are none.
The "Cover Texts" are certain short passages of text that are listed, as Front-Cover Texts or Back-Cover Texts, in
the notice that says that the Document is released under this License. A Front-Cover Text may be at most 5
words, and a Back-Cover Text may be at most 25 words.
A "Transparent" copy of the Document means a machine-readable copy, represented in a format whose speci-
fication is available to the general public, that is suitable for revising the document straightforwardly with
generic text editors or (for images composed of pixels) generic paint programs or (for drawings) some widely
available drawing editor, and that is suitable for input to text formatters or for automatic translation to a vari-
ety of formats suitable for input to text formatters. A copy made in an otherwise Transparent file format whose
markup, or absence of markup, has been arranged to thwart or discourage subsequent modification by read-
ers is not Transparent. An image format is not Transparent if used for any substantial amount of text. A copy
that is not "Transparent" is called "Opaque".
361 z 366 2017-05-27 13:21

Examples of suitable formats for Transparent copies include plain ASCII without markup, Texinfo input format,
LaTeX input format, SGML or XML using a publicly available DTD, and standard-conforming simple HTML, Post-
Script or PDF designed for human modification. Examples of transparent image formats include PNG, XCF and
JPG. Opaque formats include proprietary formats that can be read and edited only by proprietary word proces-
sors, SGML or XML for which the DTD and/or processing tools are not generally available, and the machine-
generated HTML, PostScript or PDF produced by some word processors for output purposes only.
The "Title Page" means, for a printed book, the title page itself, plus such following pages as are needed to
hold, legibly, the material this License requires to appear in the title page. For works in formats which do not
have any title page as such, "Title Page" means the text near the most prominent appearance of the work's ti-
tle, preceding the beginning of the body of the text.
A section "Entitled XYZ" means a named subunit of the Document whose title either is precisely XYZ or con-
tains XYZ in parentheses following text that translates XYZ in another language. (Here XYZ stands for a specific
section name mentioned below, such as "Acknowledgements", "Dedications", "Endorsements", or "History".) To
"Preserve the Title" of such a section when you modify the Document means that it remains a section "Entitled
XYZ" according to this definition.
The Document may include Warranty Disclaimers next to the notice which states that this License applies to
the Document. These Warranty Disclaimers are considered to be included by reference in this License, but only
as regards disclaiming warranties: any other implication that these Warranty Disclaimers may have is void and
has no effect on the meaning of this License.
2. VERBATIM COPYING
You may copy and distribute the Document in any medium, either commercially or non-commercially, pro-
vided that this License, the copyright notices, and the license notice saying this License applies to the Docu-
ment are reproduced in all copies, and that you add no other conditions whatsoever to those of this License.
You may not use technical measures to obstruct or control the reading or further copying of the copies you
make or distribute. However, you may accept compensation in exchange for copies. If you distribute a large
enough number of copies you must also follow the conditions in section 3.
You may also lend copies, under the same conditions stated above, and you may publicly display copies.
3. COPYING IN QUANTITY
If you publish printed copies (or copies in media that commonly have printed covers) of the Document, num-
bering more than 100, and the Document's license notice requires Cover Texts, you must enclose the copies in
covers that carry, clearly and legibly, all these Cover Texts: Front-Cover Texts on the front cover, and Back-Cover
Texts on the back cover. Both covers must also clearly and legibly identify you as the publisher of these copies.
The front cover must present the full title with all words of the title equally prominent and visible. You may add
other material on the covers in addition. Copying with changes limited to the covers, as long as they preserve
the title of the Document and satisfy these conditions, can be treated as verbatim copying in other respects.
If the required texts for either cover are too voluminous to fit legibly, you should put the first ones listed (as
many as fit reasonably) on the actual cover, and continue the rest onto adjacent pages.
362 z 366 2017-05-27 13:21

If you publish or distribute Opaque copies of the Document numbering more than 100, you must either in-
clude a machine-readable Transparent copy along with each Opaque copy, or state in or with each Opaque
copy a computer-network location from which the general network-using public has access to download using
public-standard network protocols a complete Transparent copy of the Document, free of added material. If
you use the latter option, you must take reasonably prudent steps, when you begin distribution of Opaque
copies in quantity, to ensure that this Transparent copy will remain thus accessible at the stated location until
at least one year after the last time you distribute an Opaque copy (directly or through your agents or retailers)
of that edition to the public.
It is requested, but not required, that you contact the authors of the Document well before redistributing any
large number of copies, to give them a chance to provide you with an updated version of the Document.
4. MODIFICATIONS
You may copy and distribute a Modified Version of the Document under the conditions of sections 2 and 3
above, provided that you release the Modified Version under precisely this License, with the Modified Version
filling the role of the Document, thus licensing distribution and modification of the Modified Version to who-
ever possesses a copy of it. In addition, you must do these things in the Modified Version:
1. Use in the Title Page (and on the covers, if any) a title distinct from that of the Document, and from
those of previous versions (which should, if there were any, be listed in the History section of the Doc-
ument). You may use the same title as a previous version if the original publisher of that version gives
permission.
2. List on the Title Page, as authors, one or more persons or entities responsible for authorship of the
modifications in the Modified Version, together with at least five of the principal authors of the Docu-
ment (all of its principal authors, if it has fewer than five), unless they release you from this require-
ment.
3. State on the Title page the name of the publisher of the Modified Version, as the publisher.
4. Preserve all the copyright notices of the Document.
5. Add an appropriate copyright notice for your modifications adjacent to the other copyright notices.
6. Include, immediately after the copyright notices, a license notice giving the public permission to use
the Modified Version under the terms of this License, in the form shown in the Addendum below.
7. Preserve in that license notice the full lists of Invariant Sections and required Cover Texts given in the
Document's license notice.
8. Include an unaltered copy of this License.
9. Preserve the section Entitled "History", Preserve its Title, and add to it an item stating at least the title,
year, new authors, and publisher of the Modified Version as given on the Title Page. If there is no sec-
tion Entitled "History" in the Document, create one stating the title, year, authors, and publisher of the
Document as given on its Title Page, then add an item describing the Modified Version as stated in the
previous sentence.
363 z 366 2017-05-27 13:21

10. Preserve the network location, if any, given in the Document for public access to a Transparent copy of
the Document, and likewise the network locations given in the Document for previous versions it was
based on. These may be placed in the "History" section. You may omit a network location for a work
that was published at least four years before the Document itself, or if the original publisher of the ver-
sion it refers to gives permission.
11. For any section Entitled "Acknowledgements" or "Dedications", Preserve the Title of the section, and
preserve in the section all the substance and tone of each of the contributor acknowledgements
and/or dedications given therein.
12. Preserve all the Invariant Sections of the Document, unaltered in their text and in their titles. Section
numbers or the equivalent are not considered part of the section titles.
13. Delete any section Entitled "Endorsements". Such a section may not be included in the Modified Ver-
sion.
14. Do not retitle any existing section to be Entitled "Endorsements" or to conflict in title with any Invariant
Section.
15. Preserve any Warranty Disclaimers.
If the Modified Version includes new front-matter sections or appendices that qualify as Secondary Sections
and contain no material copied from the Document, you may at your option designate some or all of these
sections as invariant. To do this, add their titles to the list of Invariant Sections in the Modified Version's li-
cense notice. These titles must be distinct from any other section titles.
You may add a section Entitled "Endorsements", provided it contains nothing but endorsements of your Modi-
fied Version by various parties--for example, statements of peer review or that the text has been approved by
an organization as the authoritative definition of a standard.
You may add a passage of up to five words as a Front-Cover Text, and a passage of up to 25 words as a
Back-Cover Text, to the end of the list of Cover Texts in the Modified Version. Only one passage of Front-Cover
Text and one of Back-Cover Text may be added by (or through arrangements made by) any one entity. If the
Document already includes a cover text for the same cover, previously added by you or by arrangement made
by the same entity you are acting on behalf of, you may not add another; but you may replace the old one, on
explicit permission from the previous publisher that added the old one.
The author(s) and publisher(s) of the Document do not by this License give permission to use their names for
publicity for or to assert or imply endorsement of any Modified Version.
5. COMBINING DOCUMENTS
You may combine the Document with other documents released under this License, under the terms defined
in section 4 above for modified versions, provided that you include in the combination all of the Invariant Sec-
tions of all of the original documents, unmodified, and list them all as Invariant Sections of your combined
work in its license notice, and that you preserve all their Warranty Disclaimers.
364 z 366 2017-05-27 13:21

The combined work need only contain one copy of this License, and multiple identical Invariant Sections may
be replaced with a single copy. If there are multiple Invariant Sections with the same name but different con-
tents, make the title of each such section unique by adding at the end of it, in parentheses, the name of the
original author or publisher of that section if known, or else a unique number. Make the same adjustment to
the section titles in the list of Invariant Sections in the license notice of the combined work.
In the combination, you must combine any sections Entitled "History" in the various original documents, form-
ing one section Entitled "History"; likewise combine any sections Entitled "Acknowledgements", and any sec-
tions Entitled "Dedications". You must delete all sections Entitled "Endorsements".
6. COLLECTIONS OF DOCUMENTS
You may make a collection consisting of the Document and other documents released under this License, and
replace the individual copies of this License in the various documents with a single copy that is included in the
collection, provided that you follow the rules of this License for verbatim copying of each of the documents in
all other respects.
You may extract a single document from such a collection, and distribute it individually under this License, pro-
vided you insert a copy of this License into the extracted document, and follow this License in all other re-
spects regarding verbatim copying of that document.
7. AGGREGATION WITH INDEPENDENT WORKS

A compilation of the Document or its derivatives with other separate and independent documents or works, in
or on a volume of a storage or distribution medium, is called an "aggregate" if the copyright resulting from the
compilation is not used to limit the legal rights of the compilation's users beyond what the individual works
permit. When the Document is included in an aggregate, this License does not apply to the other works in the
aggregate which are not themselves derivative works of the Document.
If the Cover Text requirement of section 3 is applicable to these copies of the Document, then if the Document
is less than one half of the entire aggregate, the Document's Cover Texts may be placed on covers that bracket
the Document within the aggregate, or the electronic equivalent of covers if the Document is in electronic
form. Otherwise they must appear on printed covers that bracket the whole aggregate.
8. TRANSLATION
Translation is considered a kind of modification, so you may distribute translations of the Document under the
terms of section 4. Replacing Invariant Sections with translations requires special permission from their copy-
right holders, but you may include translations of some or all Invariant Sections in addition to the original ver-
sions of these Invariant Sections. You may include a translation of this License, and all the license notices in
the Document, and any Warranty Disclaimers, provided that you also include the original English version of
this License and the original versions of those notices and disclaimers. In case of a disagreement between the
translation and the original version of this License or a notice or disclaimer, the original version will prevail.
If a section in the Document is Entitled "Acknowledgements", "Dedications", or "History", the requirement (sec-
tion 4) to Preserve its Title (section 1) will typically require changing the actual title.
365 z 366 2017-05-27 13:21

9. TERMINATION
You may not copy, modify, sublicense, or distribute the Document except as expressly provided for under this
License. Any other attempt to copy, modify, sublicense or distribute the Document is void, and will automati-
cally terminate your rights under this License. However, parties who have received copies, or rights, from you
under this License will not have their licenses terminated so long as such parties remain in full compliance.
10. FUTURE REVISIONS OF THIS LICENSE

The Free Software Foundation may publish new, revised versions of the GNU Free Documentation License from
time to time. Such new versions will be similar in spirit to the present version, but may differ in detail to ad-
dress new problems or concerns. See http://www.gnu.org/copyleft/ ↗.
Each version of the License is given a distinguishing version number. If the Document specifies that a particu-
lar numbered version of this License "or any later version" applies to it, you have the option of following the
terms and conditions either of that specified version or of any later version that has been published (not as a
draft) by the Free Software Foundation. If the Document does not specify a version number of this License, you
may choose any version ever published (not as a draft) by the Free Software Foundation.
ADDENDUM: How to use this License for your documents

Copyright (c) YEAR YOUR NAME.
Permission is granted to copy, distribute and/or modify this document
under the terms of the GNU Free Documentation License, Version 1.2
or any later version published by the Free Software Foundation;
with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts.
A copy of the license is included in the section entitled “GNU
Free Documentation License”.
If you have Invariant Sections, Front-Cover Texts and Back-Cover Texts, replace the “with...Texts.” line with this:
with the Invariant Sections being LIST THEIR TITLES, with the
Front-Cover Texts being LIST, and with the Back-Cover Texts being LIST.
If you have Invariant Sections without Cover Texts, or some other combination of the three, merge those two
alternatives to suit the situation.
If your document contains nontrivial examples of program code, we recommend releasing these examples in
parallel under your choice of free software license, such as the GNU General Public License, to permit their use
in free software.
366 z 366 2017-05-27 13:21

Virtualization Guide - SUSE Linux Enterprise Server 12 SP2

Uploaded by

Copyright:

Available Formats

Virtualization Guide - SUSE Linux Enterprise Server 12 SP2

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Virtualization Guide - SUSE Linux Enterprise Server 12 SP2

Uploaded by

Copyright:

Available Formats

Virtualization Guide | SUSE Linux Enterprise Server 12 SP2 https://www.suse.com/documentation/sles-12/singlehtml/book_virt/boo...

SUSE Linux Enterprise Server 12 SP2

Describes virtualization technology in general, and introduces libvirt—the

About This Manual

2 Introduction to Xen Virtualization

3 Introduction to KVM Virtualization

1 z 366 2017-05-27 13:21

4 Introduction to Linux Containers

6 Installation of Virtualization Components

7 Supported Guests, Hosts and Features

II Managing Virtual Machines with libvirt

10 Basic VM Guest Management

11 Connecting and Authorizing

2 z 366 2017-05-27 13:21

11.3 Configuring Remote Connections

14 Configuring Virtual Machines

III Hypervisor-Independent Features

16 VM Guest Clock Settings

3 z 366 2017-05-27 13:21

16.2 Xen Virtual Machine Clock Settings

IV Managing Virtual Machines with Xen

20 Managing a Virtualization Environment

21 Block Devices in Xen

4 z 366 2017-05-27 13:21

21.5 Scripts for Managing Advanced Storage Scenarios

22 Virtualization: Configuration Options and Settings

24 XenStore: Configuration Database Shared between Domains

25 Xen as a High-Availability Virtualization Host

V Managing Virtual Machines with QEMU

29 Running Virtual Machines with qemu-system-ARCH

5 z 366 2017-05-27 13:21

29.2 General qemu-system-ARCH Options

30 Virtual Machine Administration Using QEMU Monitor

VI Managing Virtual Machines with LXC

32 Migration from LXC to libvirt-lxc

C XM, XL Toolstacks and Libvirt framework

6 z 366 2017-05-27 13:21

C.1 Xen Toolstacks

7 z 366 2017-05-27 13:21

29.2 Configuring Network Bridge with YaST

8 z 366 2017-05-27 13:21

C.33 xm cpupool-create Removed Options

9 z 366 2017-05-27 13:21

About This Manual

This manual offers an introduction to setting up and managing virtualization with

Many chapters in this manual contain links to additional documentation resources.

Article “Installation Quick Start”

Book “Deployment Guide”

10 z 366 2017-05-27 13:21

Book “Administration Guide”

Book “Storage Administration Guide”

Book “Security Guide”

Book “Security and Hardening Guide”

Book “System Analysis and Tuning Guide”

11 z 366 2017-05-27 13:21

Book “Subscription Management Tool for SLES 12 SP2”

Book “GNOME User Guide”

Bugs and Enhancement Requests

12 z 366 2017-05-27 13:21