Top 10 Best Practices Vsphere Backups
Top 10 Best Practices Vsphere Backups
Top 10 Best Practices Vsphere Backups
Hannes Kasparick
Senior Systems Engineer, Veeam
© 2018 Veeam Software. Confidential information. All rights reserved. All trademarks are the property of their respective owners.
Top 10 Best Practices for vSphere Backups
Executive summary
Server virtualization is a widespread practice all around the world. In 2018 VMware is still the market leader and many Veeam®
customers use VMware vSphere as their preferred virtualization platform. This white paper describes best practices that are
specific for backup and Availability of VMware vSphere with Veeam Backup & Replication™ 9.5. It does not include general
best practices for Hyper-V and Veeam Agent specifics.
Introduction
Server virtualization is a widespread practice all around the world. In 2018 VMware is still the market leader and many Veeam
customers use VMware vSphere as their preferred virtualization platform. Talking about the backup of virtual machines on
vSphere is only one part of service Availability. Backup is the foundation for restores, so it is essential to have backups always
available with the required speed. The most important rule as a general best practice in the field of backup is the 3-2-1 rule.
That means having at least three copies of the data (production, first line of backup, second line of backup). It also recommends
storing the backup copies on at least two independent types of media. The “independent” cannot be overemphasized.
Independent means that there is no dependency from a technology perspective. And last but not least, one copy should
be off site and offline, out of reach of natural disasters, malicious software and unauthorized people. For example: Veeam added
an “Insider protection for Veeam Cloud Connect” in Backup & Replication 9.5 Update 3. Of course, tape is still an option for
offsite storage for backups.
Veeam Backup & Replication helps to extend the 3-2-1 rule to the 3-2-1-0 rule. The zero stands for zero restore issues, which
is provided by automated restore tests with Veeam SureBackup.
This document describes several best practices with Veeam Backup & Replication and VMware vSphere. These best practices
are dedicated to Veeam + VMware. Other hypervisors are not covered in this document.
These apply in any case, no matter if it’s VMware, Hyper-V, a cloud provider or physical server backup.
First and most important to know before planning or implementing any solution is to be certain about the requirements.
In an ideal world, the business creates the requirements and tells IT which RPO (recovery point objective, maximum time for
data loss) and RTO (recovery time objective, maximum time to finish a restore) is needed. Is it only about backup, or is disaster
recovery also a requirement?
With this information, it is possible to size the hardware. That includes the number of CPU-cores, amount of memory and
bandwidth requirements for WAN, LAN and SAN. Finally, it needs a source and a backup storage that is fast enough to serve
the required speed.
The next step is the backup itself. Veaam’s “Application Aware Image Processing” uses Microsoft VSS (volume shadow copy
service) to achieve application consistent backup of Windows virtual machines. This mechanism does not use VMware-tools
quiesceing. To make “Application Aware Image Processing” work reliably, it is necessary that the VSS writers of the virtual
machines are working properly.
© 2018 Veeam Software. Confidential information. All rights reserved. All trademarks are the property of their respective owners. 3
Top 10 Best Practices for vSphere Backups
On the other hand, VMware has improved virtual machine snapshot consolidation with ESXi version 6.x. This leads to less
“VM-stuns” on i/o intensive virtual machines during snapshot commit after a backup.
The best practice: Look out for improvements of the latest versions of Veeam Backup & Replication and vSphere.
The properties of each proxy allow the configuration of options above in the “Transport mode” section. Figure 1 shows this:
© 2018 Veeam Software. Confidential information. All rights reserved. All trademarks are the property of their respective owners. 4
Top 10 Best Practices for vSphere Backups
The Network mode or “NBD” mode is the easiest way to do VMware backups. The Veeam proxy server uses the ESXi management
port of each ESXi-host to transfer backup data. That makes the setup very simple as it requires no additional storage- or VM-
configuration. It has very low overhead, which is another advantage. Compared to Hot-Add mode, it does not need any additional
hot-add mount operations that can be time consuming. It also does not create additional storage snapshots like “backup from
storage snapshots” with integrated storage systems. The coordination of VM- and storage-snapshots takes time, so the Network
mode can even be the fastest for incremental backups in environments with many VMs and a low data change rate.
The disadvantage of the network mode is the general performance in certain situations. The ESXi management ports have
a limit of 40% of the available bandwidth for backup traffic. That means for a 1GBit management port, Veeam can only use
400Mbit. This is usually no problem with 10Gbit or faster management ports.
With ESXi 6.5, VMware enforced encryption of backup traffic via NBD-SSL. Encryption was an optional setting before. This reduces
the backup speed significantly. With later updates VMware allowed unencrypted NBD traffic again. Veeam Backup & Replication 9.5
Update 3 supports this “new” unencrypted backup via NBD.
The Direct Storage Access mode backup traffic goes directly from the storage system to the Veeam Backup Proxy. The backup
traffic does not need to go through the ESXi hypervisor. The protocol depends on the storage environment. Usually it is
FibreChannel or iSCSI. Direct Storage Access mode also has the same advantage over Hot-Add as already mentioned for Network
mode: No time-consuming Hot-Add operation. On the other hand, both modes, NBD and Direct Storage Access, use VADP.
VADP is the official API from VMware to back up virtual machines. It has some backup performance implications which is the reason
why Veeam Backup & Replication does not use VADP in three special configurations. These three special configurations are:
• Backup from Storage Snapshots
• Direct NFS (a flavor of Direct Storage Access)
• Virtual Appliance / Hot-Add.
Avoiding VADP leads to significant backup performance improvements, which is the reason why Hot-Add is becoming more
popular since Backup & Replication 9.5. But Hot-Add has one more advantage. In Hot-Add mode, the Veeam Backup Proxy
runs as an additional virtual machine for backups. It mounts the snapshots of the VMs to backup and sends the traffic over the
normal VM network. It does not use the ESXi management interface. This fact makes Hot-Add a performant alternative in 1GBit
networks where Direct Storage Access backup modes are not possible.
The “Hot-Add” backup mode is not recommended in general with NFS datastores. The recommendation with NFS is “Direct
Storage Access” which results in the “Direct NFS” mode. “Direct NFS” has no separate option in the UI. It is just a flavor of Direct
Storage Access. The reason for the recommendation is that “Hot-Add” often results in VM-stuns if the Veeam Proxy does not run
on the same ESXi host as the virtual machine. Veeam KB1681 provides more details in the section “For environments with NFS
Datastores.” If you plan to use “Hot-Add” mode on NFS datastores anyway, apply the following rules and settings:
• One Hot-Add proxy per ESXi host
• Set EnableSameHostHotAddMode = 1 in HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication
As there are diverse options to do backups, you can use the following table to quantify results of each mode and reach
a conclusion as to which one is the best for you.
The best practice: Test which backup mode fits best to your environment.
© 2018 Veeam Software. Confidential information. All rights reserved. All trademarks are the property of their respective owners. 5
Top 10 Best Practices for vSphere Backups
First, it is important to know that file- and object-restore differs from VM- or disk-restore. Veeam restores files or objects
(like Exchange e-mails or Active Directory objects) over the network. Over the network means an RPC (Windows) or SSH (Linux)
connection that transfers the data to restore into the virtual machine. For Windows file level restore there is a restore speed
difference between the Veeam file level recovery Explorer and the Windows Explorer. During a Windows file level recovery
session, Veeam mounts the backup per default to C:\VeeamFLR\... Copy operations of the Windows Explorer are faster because
the Veeam file level recovery Explorer has additional checks.
As backup is VM-snapshot-based as block-level backup, restore of full virtual machines or virtual disks is also block based.
Depending on the restore mode, it makes a difference whether the VM is thick- or thin-provisioned. The restore modes are
the same as for Backup (Direct Storage Access, Virtual Appliance and Network). Additionally, there is Instant-VM-Recovery
combined with Storage-VMotion or Quick-Migration.
Hot-Add- and Network-mode can restore thick- and thin-provisioned VMs. As already mentioned, the “Virtual Appliance”
or “Hot-Add” transport with version 9.5 mode has improved performance for backup. This is also true for full VM or disk restore
with “Hot-Add.” In most scenarios, it makes sense to have at least one Hot-Add proxy available for VM or disk-restores.
Network (NBD) mode is usually the slowest way to restore, as it cannot use the full bandwidth (usually slower than backup).
“Direct Storage Access” mode has no limitations concerning network bandwidth, but it can only restore thick-provisioned disks.
Thin provisioned disks would be converted on-the-fly to thick disks. As “Direct Storage Access” mode uses VADP for restore, it is
usually not the fastest option. The exception here is restore with “Direct NFS” where Veeam Backup & Replication does not use VADP.
To restore a VM or virtual disk, it is not required to fully transfer all data. If the change block tracking information on the production
storage is correct, then a restore based on change block tracking is possible. Setting this option can reduce the restore time.
The “Quick Rollback” option to do this must be enabled during restore manually. Figure 2 shows this:
© 2018 Veeam Software. Confidential information. All rights reserved. All trademarks are the property of their respective owners. 6
Top 10 Best Practices for vSphere Backups
Instant VM recovery is an alternative way for full VM restore. It allows you to instantly boot a virtual machine directly from
the backup repository. The backup repository acts as an NFS datastore that is mounted to an ESXi host. There are two options
to transfer the VM data from the repository NFS datastore back to the production datastore:
• Veeam Quick Migration
• VMware Storage VMotion
As there are diverse options for full-VM restores, you can use the following table to quantify results of each mode and reach
a conclusion as to which one is the best for you.
The best practice: Plan and test the restore options depending on your storage and transport modes. If you do not use NFS
datastores, have at least one “Hot-Add” proxy installed as spare.
This is because Veeam Backup & Replication cannot detect the IP address and without the IP address Veeam cannot connect
to the virtual machine over the network. The fallback mechanism VIX or “vSphere API for guest interaction” does also not work
due to the lack of VMware tools (see #10 for more information on VIX). Figure 3 shows this in an example of a failed guest
credentials test because of missing VMware tools:
© 2018 Veeam Software. Confidential information. All rights reserved. All trademarks are the property of their respective owners. 7
Top 10 Best Practices for vSphere Backups
The second example is SureBackup tests. Heartbeat and ping tests will fail if VMware tools are not present. For VMware tools
the #1 rule applies: Keep them up-to-date.
The best practice: Install VMware tools and keep them up-to-date
The first is that Veeam Backup & Replication can open storage snapshots and restore files and objects directly from the storage
snapshot. This allows you, for example, to schedule storage snapshots every 15 minutes without the requirement to create
virtual machine snapshots. Although the 15 minutes’ snapshot is not a real backup as it does not meet the 3-2-1 rule, it helps
to decrease the RPO times.
Figure 4 shows an example of this concept. It shows the Veeam Explorer™ for Storage Snapshots. The left side shows the storage
snapshots (the LUNs and the snapshots of one LUN). The right side shows the virtual machines of each storage snapshot. From
there it is possible to restore virtual machines with Instant VM Recovery or restore files and application objects. Now imagine the
storage does snapshots of critical LUNs or volumes every 15 minutes and deletes them after four hours. That means it is possible
to restore data from 15 minutes ago, instead of older data from the last night’s backup.
© 2018 Veeam Software. Confidential information. All rights reserved. All trademarks are the property of their respective owners. 8
Top 10 Best Practices for vSphere Backups
The second advantage of a storage integration with Veeam Backup & Replication is the following: Backups of large, highly
transactional virtual machines, for example database servers, are now possible without the risk of “VM stuns” during VMware
snapshot consolidation. Although the situation is much better with current vSphere versions, it is still the main reason
to use storage snapshots.
Finally, “Backup from Storage Snapshot” allows Veeam to use its proprietary data fetcher mechanisms and outperform classic
VADP backups. This is especially relevant for full backups or any backup with high change rates.
The best practice: Use storage integration if you have a storage that has snapshot support for Veeam Backup & Replication
The supported backup modes are “Virtual Appliance / Hot-Add” and “Network mode.” With Hot-Add mode Veeam Backup &
Replication backs up VMs relative to proximity to the virtual machine data. That means the backups occur through the proxy
on the host that has the most VM-specific data. To make that work properly, there must be one Hot-Add proxy per ESXi host.
Host affinity for the proxy-VMs rules prevent VMware DRS (Distributed Resource Scheduler) from moving those virtual
machines to other ESXi hosts.
That means shorter backup windows, as there is less network traffic and latency. If a VM was on one host and the proxy
on a different host then there is more traffic over the network which adds latency and reduces speed.
Veeam participated in the VMware Ready for vSAN program, which was new with the launch of VMware vSAN 6.5. Veeam
Backup & Replication is certified as VMware Ready for vSAN within the Data Protection category. The VMware knowledge
base article 2149874 and the VMware vSAN HCL have further information.
The best practice: Install one Hot-Add proxy per ESXi host if you use “Virtual Appliance” mode with VMware Virtual SAN
The vCenter is one of the most critical parts needed for Veeam Backup & Replication to work. If the vCenter is down backups
will fail. So, the maintenance windows of vCenter should be planned outside the backup windows. Also, keeping an eye
on the vCenter load and number of connections is a good idea. Speaking of connections: Of course, the network between
Veeam Backup-Server and vCenter should be stable!
Depending on your environment, backup can put a significant load on the production storage. Multiple GBytes per second are
not uncommon and can raise the I/O latency on traditional disk arrays. Veeam Backup & Replication “I/O Control” (commonly
known as “Backup I/O control”) throttles backup and restore speed (Figure 5).
© 2018 Veeam Software. Confidential information. All rights reserved. All trademarks are the property of their respective owners. 9
Top 10 Best Practices for vSphere Backups
Storage Latency Control assigns or throttles tasks based on the datastore latency values that Veeam gets from vSphere.
This happens in two stages. First Veeam stops to assign new backup tasks to a datastore. If the latency still increases, then it will
throttle the existing backup tasks. As a result, the backup will take longer, but with less influence on running virtual machines.
With this mechanism, it is possible to do backups during production hours with minimal impact on virtual machines,
applications and users.
Storage Latency Control disables the default setting of a maximum of four virtual machine snapshots per datastore at the same
time. This can also lead to performance improvements.
The best practice: As Veeam Backup & Replication relies heavily on vCenter, make sure it’s running efficiently, monitor the load
during backup windows and tune as required.
#8. Security
Veeam Backup & Replication connects to the vCenter to manage backup and restores of virtual machines. From a security
point of view, it is always a good idea to work with the least privileges required. VMware vCenter offers fine granular
permissions to allow backups.
The “Required permissions” document contains a detailed description of which permissions to configure for which backup
mode. The different backup modes require different permissions. A security relevant permission for the “Virtual Appliance”
backup mode is that it requires the “remove disk” permission.
These security considerations can have influence on the choice of the backup mode. It is also possible to restrict specific backup
servers (if you have multiple) to specific locations or objects in vCenter.
The best practice: Work within the boundaries of the principal of least privilege.
© 2018 Veeam Software. Confidential information. All rights reserved. All trademarks are the property of their respective owners. 10
Top 10 Best Practices for vSphere Backups
The Veeam ONE monitor shows the actual status and current issues of the vSphere environment. Relevant issues around backup
could be, for example, a high storage latency or old/large/many/orphaned virtual machine snapshots.
The Veeam ONE reporter includes the “VM configuration assessment” report that shows potential backup issues. Typical issues
the report shows are:
• VMware tools not installed
• Hardware Version 4 or earlier
• Disks that cannot be backed up (e.g. independent disks)
• Datastores with less than 10% free space
• Raw device mappings in virtual machines
Fixing these issues before running backups prevents further backup issues.
The best practice: Use Veeam ONE to plan the Backup & Replication installation
The preferred way for application aware backup is connecting the application proxy via RPC to the virtual machine. This is the fastest
way. If network segmentation or firewalls prevent a network communication to the virtual machine, Veeam can use the VIX API or
in newer vSphere versions (6.5 and newer) the “vSphere API for guest interaction.” Figure 6 shows the login via VIX marked in orange.
© 2018 Veeam Software. Confidential information. All rights reserved. All trademarks are the property of their respective owners. 11
Top 10 Best Practices for vSphere Backups
VIX or vSphere API for guest interaction does not work out of the box. Veeam KB 1788 describes the requirements in detail.
The summary is that it has two requirements:
• The user account used by Veeam must be a member of the Local Administrators group
• If the account is not named “Administrator,” then UAC (Windows User Account Control) must be disabled
VIX or vSphere API for guest interaction is the fallback mode if RPC does not work. The result for environment, where most
VMs are not reachable via RPC, is that the backup will take longer because Veeam always tries RPC first. For those environments,
it is possible to change the order to “VIX first” with the following registry key on the backup server or guest interaction proxy:
It is important to know that VIX or vSphere API for guest interaction has some limitations on restore operations. It is only
possible to restore files but no application items. That means it is not possible to restore AD, Exchange etc., objects through this
way. It requires network connection for restore. The second thing is that the file is much slower than via network.
Talking about speed: The VeeamLogShipper service that does SQL log-shipping can also use VIX as fallback mechanism if it
cannot reach the repository via network. This can be too slow for most environments. That said, it is recommended that SQL
log-shipping is done via network.
The best practice: Keep in mind the limitations of VIX or vSphere API for guest interaction
Conclusion
The combination of Veeam Backup & Replication with VMware vSphere usually works just right “out of the box.” But there are
several best practices that can make it work even better. Those best practices are not complicated, they can be configured fast
and in quite an easy way.
The best practice: Read the full Veeam Backup & Replication Best Practices guide if you plan a larger or complex deployment
© 2018 Veeam Software. Confidential information. All rights reserved. All trademarks are the property of their respective owners. 12
Top 10 Best Practices for vSphere Backups
Founded in 2006, Veeam currently has 51,000 ProPartners and more than 267,500 customers worldwide. Veeam‘s
global headquarters are located in Baar, Switzerland, and the company has offices throughout the world. To learn more,
visit http://www.veeam.com.
© 2018 Veeam Software. Confidential information. All rights reserved. All trademarks are the property of their respective owners. 13
Top 10 Best Practices for vSphere Backups
© 2018 Veeam Software. Confidential information. All rights reserved. All trademarks are the property of their respective owners. 14