Ovirt Storage and Live Storage Migration 1

oVirt
Live Storage Migration (Under The Hood)

Ayal Baron
Engineering Manager, Red Hat
January 2013
Live Storage Migration Under The Hood
Agenda
Live Storage Migration in a Nutshell
Live Storage Migration Types
Some Prerequisites
Constraints and Limitations
General Overview
Flow Overview
SPM/HSM API
Detailed Flow and Sequence Diagram
Above and Below (oVirt Engine, libvirt and QEMU)
Move Disk
Live Storage Migration in a Nutshell

Definition
Live Storage Migration is the ability to move one or

more VM disks from one storage to another while the
VM is running
Motivation
Facilitate storage hardware upgrades

Move or clone VM disks across different (and
eventually geographically separated) data centers
SLA
Live Storage Migration Types

With Shared Storage
The hypervisor is able to access both the source and

destination storage backends
The virtual machine remains on the same host
Without Shared Storage
The hypervisor is not able to access both the source

and destination storage
The virtual machine is live migrated to a different host
that is able to access the destination storage
Overview With Shared Storage

Host A
Host A
VM
VM
Virtual Disk
Storage A
Storage B
Virtual Disk
Storage A
Storage B
Overview Without Shared Storage

Host A
Host B
VM
VM
Virtual Disk
Storage A
Virtual Disk
Storage B
Some Prerequisites
General understanding of the oVirt architecture and

few VDSM basics
Virtual disks as ordered collection (chain) of volumes,
e.g.:
Volume 1
Volume 2
Volume 3
General understanding of the QCOW format and how

VDSM handles the QCOW watermark on block storage
domains
Constraints and Limitations
All the image manipulations must be done by the Storage

Pool Manager (SPM)
Most of the data transfer is offloaded (now SPM, future
Xcopy)
Initial focus on shared storage between homogeneous
storage domain types
An image (volume chain) should not be spread among
multiple storage domains
It is important to maintain the source image structure

(external snapshots)
Storage Architecture
Centralized storage system
(disk images, templates, etc...)
Storage Domain
A standalone storage entity

(implemented with NFS, FCP, iSCSI,
FCoE, and SAS)
Stores the images and

associated metadata
Only true persistent storage for
VDSM
Storage Pool
Aggregates several Storage

Domains
(it will be deprecated in the future)
10
Supposed to simplify cross

domain operations
File Storage Domains
11
Use file system features for

segmentation
Use file system for
synchronizing access
Sparse files
Better image manipulation
capabilities
Volumes and metadata are
files
1:1 Mapping between
domain and mount /
directory
Block Storage Domains
Use LVM for segmentation
Very specialized use of LVM
Mailbox
12
Thin provisioning
Devices managed by
device-mapper and
multipath
Domain is a VG
Metadata is stored in a
single LV and in LVM tags
Volumes are LVs
Master Domain
Used to store:
Pool metadata
Backup of OVFs
(treated as blobs)
Async tasks
(persistent data)
13
Contains the clustered lock

for the pool
Storage Pool Manager (SPM)
14
The SPM is a role assigned to one host in a data center giving the
host sole authority to make all storage domain structure changes
The role of SPM can be migrated to any host in a data center
Creation, deletion and
manipulation of Virtual Disks,
Snapshots and Templates
Allocation of storage for sparse
block devices (on SAN)
Single meta data writer
SPM lease mechanism
(Chockler and Malkhi 2004,
Light-Weight Leases for
Storage-Centric Coordination)
Storage-centric mailbox
Flow Overview 1/4 Initial State

Volume 1
15
Flow Overview 2/4 Live Snapshot

Volume 1
Volume 1
16
Volume 2
Flow Overview 3/4 Replica & Copy

Volume 1
17
Volume 1
Volume 2
Volume 1
Volume 2
Volume 1'
Volume 2'
Flow Overview 4/4 - Completion

Volume 1
18
Volume 1
Volume 2
Volume 1
Volume 2
Volume 1'
Volume 2'
Volume 1'
Volume 2'
SPM API cloneImageStructure

taskId = cloneImageStructure(spUUID, sdUUID, imgUUID, dstSdUUID)
spUUID
storage pool
sdUUID
source storage domain
imgUUID
image to clone
dstSdUUID destination storage domain
Clone Structure
19
Volume 1
Volume 2
Volume 1'
Volume 2'
SPM API syncImageData

taskId = syncImageData(spUUID, sdUUID, imgUUID, dstSdUUID, syncType)
spUUID
storage pool
sdUUID
source storage domain
imgUUID
image to clone
dstSdUUID destination storage domain
syncType
Synchronize Data
20
synchronization type (ALL, INTERNAL, ...)

Volume 1
Volume 2
Volume 1'
Volume 2'
HSM API diskReplicateStart/Finish

result = diskReplicateStart(vmId, srcDisk, dstDisk)
result = diskReplicateFinish(vmId, srcDisk, dstDisk)
21
vmId
virtual machine id
srcDisk
source disk
dstDisk
destination disk
Volume 1
Volume 2
Volume 1'
Volume 2'
read write
write only
Detailed Flow Initial Live Snapshot
SPM/HSM initial live snapshot to minimize the

amount of data replicated by the qemu process
Volume 1
Volume 1
22
Volume 2
Detailed Flow Clone Image Structure
SPM clone the image structure from the source

storage domain to the destination storage domain
Clone Structure
taskId = cloneImageStructure(spUUID, sdUUID, imgUUID, dstSdUUID)
23
Volume 1
Volume 2
Volume 1'
Volume 2'
Detailed Flow Replicate and Sync
HSM start replicating the virtual machine writes on

the destination storage domain
SPM synchronize the internal volumes data
Synchronize Data
result = diskReplicateStart(vmId, srcDisk, dstDisk)

taskId = syncImageData(spUUID, sdUUID, imgUUID, dstSdUUID, syncType)
24
Volume 1
Volume 2
Volume 1'
Volume 2'
read write
write only
Detailed Flow Finish
HSM complete the switch to the destination storage

domain
result = diskReplicateFinish(vmId, srcDisk, dstDisk)
25
Volume 1
Volume 2
Volume 1'
Volume 2'
Detailed Flow High Watermark 1/3

The watermark limit is monitored on the VM host as for
any other regular virtual disk
Synchronize Data
Volume 1
Volume 2
Volume 1'
Volume 2'
High Watermark Limit
26
read write
write only
When the watermark limit is reached the destination

volume is extended first
Synchronize Data
The source and destination volumes data is written

using the QCOW cluster size granularity (the size is
the same)
Volume 1
Volume 2
Volume 1'
Volume 2'
27
read write
write only
In case any of the extensions fails the VM is paused

(ENOSPC) and the replica is not interrupted
Synchronize Data
After the destination volume has been successfully

extended also the source is extended
Volume 1
Volume 2
Volume 1'
Volume 2'
read write
write only
28
Detailed Flow Error Handling

In case of errors it is possible to interrupt the
replication and fallback to the source storage domain
Synchronize Data
Volume 1
Volume 2
Volume 1'
Volume 2'
read write
write only
result = diskReplicateFinish(vmId, srcDisk, srcDisk)

Volume 1
29
Volume 2
Sequence Diagram
Preliminary Live Snapshot
Engine
HSM
(VM Host)
SPM
cloneImageStructure
syncImageData
extendVolume
extendVolume
diskReplicateFinish
30
syncImageData Task
diskReplicateStart
In Depth Closing the Gap
Live snapshot and disk replication are not one atomic operations
Volume 1
Volume 1
A small amount of data is synchronized by a QEMU block job

Synchronize Data
31
Volume 2
read write
Volume 1
Volume 2
QEMU background synchronization
Volume 1'
Volume 2'
write only
Low Level API
Relevant libvirt API:
blockRebase(disk, base, bandwidth, flags)
VIR_DOMAIN_BLOCK_REBASE_COPY
Starts a copy (replica/mirroring) instead of a regular block pull
VIR_DOMAIN_BLOCK_REBASE_SHALLOW
Consider only the top volume (leaf)
VIR_DOMAIN_BLOCK_REBASE_REUSE_EXT
Reuse an existing pre-initialized volume (prepared by the SPM)
blockJobAbort(disk, flags)
VIR_DOMAIN_BLOCK_JOB_ABORT_PIVOT
Switch to the destination volume (destination storage domain)
32
Relevant QEMU commands:
drive-mirror
block-job-complete
oVirt Engine Changes
Serial Execution of Asynchronous Tasks

Allow an engine command to fire a series of asynchronous SPM tasks in order to
allow complex flows (e.g. Live Storage Migration) to be implemented
CommandBase
SPMAsyncTaskHandler
executeAction()
SPMAsyncTaskHandler
SPM
SPM Action and Task Polling
VDSM Host
Ovirt Engine
executeAction()
http://wiki.ovirt.org/wiki/Features/Serial_Execution_of_Asynchronous_Tasks_Detailed_Design
33
Useful Links and Mailing Lists
34
Useful Links in the oVirt Wiki http://wiki.ovirt.org/wiki/
/Features/Serial_Execution_of_Asynchronous_Tasks_Detailed_Design
/Features/Design/StorageLiveMigration
/Features/Serial_Execution_of_Asynchronous_Tasks
Mailing lists
arch@ovirt.org users@ovirt.org
vdsm-devel@lists.fedorahosted.org
engine-devel@ovirt.org
IRC
#ovirt on OFTC
#vdsm on Freenode

Ovirt Storage and Live Storage Migration 1

Uploaded by

Copyright:

Available Formats

Ovirt Storage and Live Storage Migration 1

Uploaded by

Document Information

Original Description:

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Ovirt Storage and Live Storage Migration 1

Uploaded by

Copyright:

Available Formats

oVirt

Live Storage Migration (Under The Hood)

Live Storage Migration Under The Hood

Live Storage Migration in a Nutshell

Live Storage Migration Types

Constraints and Limitations

Detailed Flow and Sequence Diagram

Above and Below (oVirt Engine, libvirt and QEMU)

Live Storage Migration Under The Hood

Live Storage Migration Under The Hood

Live Storage Migration in a Nutshell

Live Storage Migration is the ability to move one or

Facilitate storage hardware upgrades

Live Storage Migration Types

The hypervisor is able to access both the source and

Without Shared Storage

The hypervisor is not able to access both the source

Live Storage Migration Under The Hood

Overview With Shared Storage

Live Storage Migration Under The Hood

Overview Without Shared Storage

Live Storage Migration Under The Hood

General understanding of the oVirt architecture and

General understanding of the QCOW format and how

Live Storage Migration Under The Hood

Constraints and Limitations

All the image manipulations must be done by the Storage

It is important to maintain the source image structure

Live Storage Migration Under The Hood

A standalone storage entity

Stores the images and

Aggregates several Storage

Supposed to simplify cross

Use file system features for

Live Storage Migration Under The Hood

Use LVM for segmentation

Very specialized use of LVM

Live Storage Migration Under The Hood

Contains the clustered lock

Live Storage Migration Under The Hood

Storage Pool Manager (SPM)

Live Storage Migration Under The Hood

Flow Overview 1/4 Initial State

Live Storage Migration Under The Hood

Flow Overview 2/4 Live Snapshot

Live Storage Migration Under The Hood

Flow Overview 3/4 Replica & Copy

Live Storage Migration Under The Hood

Flow Overview 4/4 - Completion

Live Storage Migration Under The Hood

SPM API cloneImageStructure

source storage domain

dstSdUUID destination storage domain

Live Storage Migration Under The Hood

SPM API syncImageData

source storage domain

dstSdUUID destination storage domain

synchronization type (ALL, INTERNAL, ...)

Live Storage Migration Under The Hood