ESXTOP Guide

The esxtop utility provides a detailed look at how ESXi uses resources in real time. It is probably the
most useful utility to troubleshoot performance issues on an ESXi host. The tool is available either on the
ESXi host itself (#esxtop), or from remote systems like the vMA (#resxtop).

In vSphere 6.5, esxtop has 9 panes that can be enable with the following shotcuts:

• c: CPU

• m: Memory

• v: Disk Virtual Machine

• u: Disk Device

• d: Disk Adapter

• n: Network

• i: Interrupt

• p: Power Management

• x: vSAN

CPU Panel

Key Metric Name Description

A ID ID The ID of a running world. The world ID is typically

cloaked and the group ID is displayed, unless the group is
expanded with the "e" command. This also applies to
groups with a single world. The ID is never identical to the
GID.

B GID Group ID The Group ID of a running world. A group is sometimes

also referred as Resource Pool which has nothing in
common with the Resource Pools you can configure in a
DRS Cluster.

Press 'N' to sort by GID column.

Key Metric Name Description

C LWID Leader World The Leader World ID, also known as VMX Cartel ID for
ID Virtual Machines. The LWID typcally the first world that
has been started in a group.

D NAME Name The name of a World's group. The number appended to

the name is the LWID. When the group is expanded with
the "e" command the name of the World is displayed.

E NWLD Number of The number of worlds running in a Group. NWLD is 1

Worlds when either the Group as a single process, or when the
group has been expanded with the "e" command.

F %USED Used Percentage of physical CPU core cycles used by the world
or group. %USED depends on the frequency the CPU core
is running with and can be higher or lower compared to
%RUN when the nominal (rated) frequency differs.
Groups can also be higher than 100% when more vCPUs
are configured or there is a high %SYS usage.
%USED = %RUN + %SYS - %OVRLP

Press 'U' to sort by %USED column.

F %RUN Run Percentage of total scheduled time for the world to run.
%RUN does not account hyperthreading and system time.
On a hyperthreading enabled server, %RUN can be twice
as large as %USED.
The main difference between %RUN and %USED is that
%RUN does not account system time.

F %SYS System Percentage of time spent by the ESXi VMkernel on behalf

of the world to process interrupts and to perform other
system activities. A high %SYS usually means high IO.

F %WAIT Wait Percentage of time a world spent in the blocked or busy

wait state. This percentage also includes %IDLE. The
Key Metric Name Description

theoretical maximum of %WAIT is (NWLD*100). It's

nothing wrong when there is a high value because the
idle time is included in %WAIT.
100%= %WAIT + %RDY + %CSTP + %RUN

F %VMWAIT Virtual The total percentage of time a world spent in a blocked

Machine Wait state waiting for events. The %VMWAIT metric is only
available for vCPUS or combined in a VM group.
For vCPUS when a Virtual Machine is expandede with the
"e" command, %WAIT = %VMWAIT + %IDLE + %SWPWT

F %RDY Ready The percentage of time the world was ready to run but
waiting in a queue to be scheduled. This may happen
even when there is plenty of free CPU cycles when a VM
CPU is administratively limited which is shown by
%MLMTD.
To determine CPU contention from %RDY you have to
take into account %RDY, %MLMTD and the number of
vCPUS. If %RDY - %MLMTD is higher than 10% per vCPU
you are very likely experiencing CPU contention. Typically
you want to see %RDY close to 0.
100% = %RUN + %RDY + %CSTP + %WAIT

Press 'R' to sort by %RDY column.

F %IDLE Idle The percentage of time the vCPU world is in an idle loop.
%IDLE is only available for vCPU worlds. Other worlds do
not have idle loops, so %IDLE is 0.

F %OVRLP Overlap The percentage of time spent by system services on

behalf of other worlds. For example, if VM A is currently
being scheduled and a network packet for VM B is
processed by the ESXi VMkernel, the time spent appears
as %OVRLP for virtual machine A and %SYS for virtual
machine B.
Key Metric Name Description

F %CSTP CoStop The percentage of time that the VM is ready to run but is
waiting for the availability of other vCPUs. The co-
deschedule state applies only for SMP VMs. The CPU
scheduler might put a vCPU in this state, when the VM
workload does not use vCPUs in a balanced fashion. For
example, if you have a VM with 2 vCPUs running a non
SMP aware application, utilizing 1 vCPU at 100% and 1
vCPU at 0%. In that case, the CPU scheduler penalties the
VM to reduce resource shortage for other VMs. This is is
represented as %CSTP.

F %MLMTD Max Limited The percentage of time a world was ready to run but
deliberately wasn't scheduled because doing so would
violate the resource pool or virtual machine CPU limit.
The %MLMTD time is included in %RDY time. If the value
is not 0, the VM has been administratively limited.

F %SWPWT Swap Wait The percentage of time a world spends waiting for the
VMkernel to swap memory. The %SWPWT time is
included in the %WAIT time. If %SWPWT is greater than
0, VMkernel is swapping the VMs memory to the disk.
This will have a large negative impact in overall
performance. Swapping can be caused by high memory
oversubscription or memory limits configured on a
resource pool or VM.

G SWTCH/s Switches/sec Number of world switches per second. A context switch

occurs when a CPU switches the execution from one
world to another.

G MIG/s Migrates/sec Number of migrations per second. A world can be

migrated from a busy pCPU to a less loaded pCPU for load
balancing.

G QEXP/s Quantum Number of quantum expirations per second. This

happens when a time quantum given to a currently
Key Metric Name Description

Expires/sec running world expires. It is common that a world changes

its state to WAIT before the current time quantum
expires (50ms).

G WAKE/s Wakeups/sec Number of world wakeups per second. A world wakes up

when its state is changes from WAIT to READY

H AMIN Alloc Min Resource pool, virtual machine, or world attribute

reservation. A value of 0 means no reservation.

H AMAX Alloc Max Resource pool, virtual machine, or world attribute limit. A
value of -1 means unlimited.

H ASHRS Alloc Shares Resource pool, virtual machine, or world attribute Shares.
Default shares are -4 (High), -3 (Normal) and -2 (Low).
When a custom share is configured, the value is
displayed.

H AMLMT Alloc Min Undocumented parameter.

Limited

H AUNITS Allocated unit Allocation unit (MHz for virtual machines)

I %LAT_C Latency CPU The percentage of time the Resource Pool/World was
ready to run but was not scheduled to run because of cpu
resource contention.

I %LAT_M Latency The percentage of time the Resource Pool/World was

Memory ready to run but was not scheduled to run because of
memory resource contention.

I %DMD Demand The CPU demand in percentage. It represents the average

active CPU load in the past 1 minute.
Key Metric Name Description

I EMIN Effective Min Effective minimum cpu allocation in case of resource

contention. This is the amount of MHz the world will
receieve when virtual machines are competing for
resources.

I TIMER/s Timer/sec Timer rate for this world.

I AFFINITY_BIT_MASK Affinity Bit Bit mask showing the current scheduling affinity for the
Mask world.

I CPU CPU The physical or logical processor on which the world was
running when resxtop (or esxtop) obtained this
information.

I HTSHARING HT Sharing Current hyperthreading configuration.

I HTQ HT Indicates whether the world is currently quarantined or

Quarantine not. N means no and Y means yes.

J POWER Power Usage Current CPU power consumption for a virtual machine (in
Watts Watts). Usage calculation is based on the value provided
by the power supply.

Memory Panel

Key Metric Name Description

A ID ID The actual world ID is cloaked and the group ID is displayed. As

you can't expand the group in the memory screen you can't see
the real world ID.

B GID Group ID The Group ID of a running world. A group is sometimes also

referred as Resource Pool which has nothing in common with
the Resource Pools you can configure in a DRS Cluster.
Key Metric Name Description

Press 'N' to sort by GID column.

C LWID Leader World ID The Leader World ID, also known as VMX Cartel ID for Virtual
Machines. The LWID typcally the first world that has been
started in a group.

D NAME Name The name of a World's group. The number appended to the
name is the LWID. Virtual Machines do not have a number
appended.

E NWLD Number of The number of worlds running in the Group. NWLD is 1 when
Worlds the Group as a single process.

F AMIN Alloc Min Resource pool, virtual machine, or world attribute reservation.
A value of 0 means no reservation.

F AMAX Alloc Max Resource pool, virtual machine, or world attribute limit. A
value of -1 means unlimited.

F ASHRS Alloc Shares Resource pool, virtual machine, or world attribute Shares.
Default shares are -4 (High), -3 (Normal) and -2 (Low). When a
custom share is configured, the value is displayed.

F AMLMT Alloc Min Limited Undocumented parameter.

F AUNITS Allocated unit Allocation unit (kilobytes for virtual machines)

G NHN Numa Home Current home node for the resource pool or virtual machine.
Nodes This statistic is applicable only on NUMA systems. If the virtual
machine has no home node, a dash (-) appears. A virtual
machine runs only on processors within its home node, and its
newly allocated memory comes from the home node as well. A
virtual machine can have multiple home nodes when the
Key Metric Name Description

number of virtual CPUs exceeds the number of cores per

physical socket.

G NMIG Numa Rebalance

Count Delta

G NRMEM Numa Remote Amount of remote memory allocated to the virtual machine.
Memory MBytes

G NLMEM Numa Local Amount of local memory allocated to the virtual machine.
Memory MBytes

G N%L Numa % Local Percentage of local memory allocated to the virtual machine.
This value should be close to 100%. A reason for poor NUMA
locality can be that a virtual machine has more memory
configured than it is available for each processor. Accessing
remote memory causes increased latency.

H MEMSZ Memory Size Amount of physical memory allocated to a virtual machine.

MBytes This is the configured virtual machine memory.
MEMSZ = GRANT + MCTLSZ + SWCUR + untouched

Press 'M' to sort by MEMSZ column.

H GRANT Memory Granted Amount of physical memory mapped to the virtual machine. If
Size MBytes GRANT is lower than MEMSZ, either the guest has never used
all of its configured memory or if it has been reclaimed by the
balloon driver.

H CNSM Memory Amount of the memory consumed by the virtual machine. The
Consumed Size memory currently consumed by the virtual machine is equal to
MBytes the amount of memory that the VM guest operating system
currently uses, excluding the amount of memory saved by
transparent page sharing and memory compression.
Key Metric Name Description

H SZTGT Target Size Targeted amount of physical memory to be allocated. It is

MBytes possible that SZTGT is higher than MEMSZ because it includes
the virtual machine overhead memory.

H TCHD Touched MBytes The amount of physical memory recently used (read or write)
by the virtual machine. Touched memory is a working set
estimate, which indicates how actively the VM is using its
memory. This value is similar to the active memory reported by
the guest OS.

H TCHD_W Touched Write The amount of physical memory recently written by the virtual
MBytes machine.

I %ACTV Active Estimate Percentage of active guest physical memory, current value.

I %ACTVS Active Slow Percentage of active guest physical memory, slow moving
Estimate average.

I %ACTVF Active Fast Percentage of active guest physical memory, fast moving
Estimate average.

I %ACTVN Active Next Percentage of active guest physical memory, predict of what
Estimate %ACTVF will be at next sample.

J MCTL? Memctl? Memory balloon driver is installed or not. N means no, Y means
yes. The ballooning driver is part of VMware Tools.

J MCTLSZ Memctl MBytes Amount of physical memory reclaimed from the virtual
machine by ballooning. To decrease host memory pressure, the
ballon driver inflates inside the virtual machine and makes the
physical memory available for other virtual machines. The
performance impact caused by ballooning is small and
therefore preferred over swapping. Another reason for
ballooning can be a configured memory limit (AMAX).
Key Metric Name Description

Press 'B' to sort by MCTLSZ column.

J MCTLTGT Memctl Target Amount of physical memory the ESXi system attempts to
MBytes reclaim from the virtual machine by way of ballooning.

J MCTLMAX Memctl Max The maximum amount of guest physical memory reclaimable
MBytes by the balloon driver.

K SWCUR Swapped MBytes Current swap usage by this virtual machine.

K SWTGT Swap Target Target where the ESXi host expects the swap usage by the
MBytes virtual machine to be.

K SWR/s Swap Read Rate at which the ESXi host swaps in memory from disk for the
MBytes/sec virtual machine.

K SWW/s Swap Written Rate at which the ESXi host swaps virtual machine memory to
MBytes/sec disk.

L LLSWR/s Llswap Read The rate at which memory is read from the host cache. When
MBytes/sec host caching is enabled, the ESXi host is able to swap to a local
SSD, instead of the virtual machines datastore, which
significantly reduces the impact caused by swapping.

L LLSWW/s Llswap Written Rate at which memory is written to the host cache. When host
MBytes/sec caching is enabled, the ESXi host is able to swap to a local SSD,
instead of the virtual machines datastore, which significantly
reduces the impact caused by swapping.

M CPTRD Checkpoint Read The amount of data read from the checkpoint file. Data from a
MBytes checkpoint file is read when a virtual machine is resumed from
a suspended state.
Key Metric Name Description

M CPTTGT Checkpoint Target The size of checkpoint file. The checkpoint file is used when a
MBytes virtual machine is suspended.

N ZERO Zero MBytes The size of virtual machine physical pages that are zeroed. A
zero page is simply a memory page that is all zeros which can
be easily used for page sharing.

N SHRD Shared MBytes Amount of guest physical memory that is shared.

N SHRDSVD Shared Saved Amount of physical memory that is saved due to page sharing.
MBytes

N COWH Copy On Write Amount of guest physical hint pages for page sharing.
Hint MBytes

O OVHDUW Overhead UW Amount of overhead memory reserved for the vmx user world.
MBytes

O OVHD Overhead MBytes Amount of overhead memory currently consumed by the

virtual machine.

O OVHDMAX Overhead Max Amount of reserved overhead memory for the entire VM.
MBytes

P MCMTTGT Min Commit

Target MBytes

P CMTTGT Commit Target

MBytes

P CMTCHRG Commit Charged

MBytes
Key Metric Name Description

P CMTPPS Commit Pages Per

Share

Q CACHESZ Compressed Current size of compressed memory by this virtual machine.

Memory MBytes

Q CACHEUSD Used Compressed Used compression memory cache.

Memory MBytes

Q ZIP/s Compression Compressed memory per second.

MBytes/sec

Q UNZIP/s Decompression Decompressed memory per second.

MBytes/sec

Disk (Virtual Machine) Panel

Key Metric Name Description

A ID ID The virtual SCSI Disk ID. The ID is typically cloaked and the
group ID is displayed, unless the group is expanded with the
"e" command. It is the same ID as used by vscsiStats.

B GID Group ID The Group ID of the virtual machine.

C VMNAME Virtual Machine The display name of the virtual machine.

Name
Press 'N' to sort by VMNAME column.

D VDEVNAME Virtual Device Name of the VSCSI device. Only displayed when the group is
Name expanded with the "e" command.

E NVDISK Number of Virtual Number of VSCSI devices.

Key Metric Name Description

Disks

F CMDS/s Commands/sec Number of commands issued per second.

F READS/s Reads/sec Number of read commands issued per second.

Press 'r' to sort by READS/s column.

F WRITES/s Writes/sec Number of write commands issued per second.

Press 'w' to sort by WRITES/s column.

F MBREAD/s MBytes Read/sec Megabytes read per second.

Press 'R' to sort by MBREAD/s column.

F MBWRTN/s MBytes Megabytes written per second.

Written/sec
Press 'T' to sort by MBWRTN/s column.

G LAT/rd Read Latency Average latency (in milliseconds) per read.

H LAT/wr Write Latency Average latency (in milliseconds) per write.

Disk (Device) Panel

Key Metric Name Description

A DEVICE Device Name of the storage device.

B PATH/WORLD/PARTITION Path/World/Partition This column is only visible when expanded

Id with either "e" for Disk World Statistics, "P"
for Disk Path Statistics or "t" for Disk
Key Metric Name Description

Partition Statistics.

C NPH Number of Paths Number of paths to the storage device.

C NWD Number of Worlds Number of worlds that are running on the

storage device. Expand the device with the
"e" command to display statistics for
individual worlds.

C NPN Number of Partitions Number of partitions on the storage

device.

D SHARES Shares Number of shares. Only displayed when the

device is expanded with the "e" (Disk
World Statistics) command.

E BLKSZ Block Size (Bytes) Physical Device block size in bytes.

E NUMBLKS Number of Blocks Number of blocks of the device.

F DQLEN Device Q Depth Current device queue depth of the storage

device.

F WQLEN World Q Depth World queue depth. This is the maximum

number of ESXi VMkernel active commands
that the world is allowed to have. This is a
per device maximum for the world. It is
valid only if the corresponding device is
expanded to worlds.

F ACTV Active Commands Number of commands in the ESXi

VMkernel that are currently active. This
statistic applies to only worlds and devices.
Key Metric Name Description

F QUED Queued Commands Number of commands in the ESXi

VMkernel that are currently queued. This
statistic applies to only worlds and devices.

F %USD % Used Percentage of the queue depth used by

ESXi VMkernel active commands. This
statistic applies to only worlds and devices.

F LOAD Load Ratio of ESXi VMkernel active commands

plus ESXi VMkernel queued commands to
queue depth. This statistic applies to only
worlds and devices.

G CMDS/s Commands/sec Number of commands issued per second.

G READS/s Reads/sec Number of read commands issued per

second.

G WRITES/s Writes/sec Number of write commands issued per

second.

G MBREAD/s MBytes Read/sec Megabytes read per second.

G MBWRTN/s MBytes Written/sec Megabytes written per second.

H RESV/s Reserves/sec

H CONS/s Conflicts/sec

I DAVG/cmd Average Driver This is the average response time in

MilliSec/Command milliseconds per command being sent to
the device. The DAVG value is part of
GAVG.
Key Metric Name Description

I KAVG/cmd Average Kernel This is the amount of time the command

MilliSec/Command spends in the VMkernel. The KAVG value is
part of GAVG.

I GAVG/cmd Average Guest "This is the response time as it is perceived

MilliSec/Command by the guest operating system. This
number is calculated with the formula:
DAVG + KAVG = GAVG"

I QAVG/cmd Average Queue Average queue latency per command in

MilliSec/Command milliseconds.

J DAVG/rd Average Driver Average device driver read latency per read
MilliSec/Read operation in milliseconds.

J KAVG/rd Average Kernel Average ESXi VMkernel read latency per

MilliSec/Read read operation in milliseconds.

J GAVG/rd Average Guest Average guest operating system read

MilliSec/Read latency per read operation in milliseconds.

J QAVG/rd Average Queue Average queue read latency per read

MilliSec/Read operation in milliseconds.

K DAVG/wr Average Driver Average device write latency per write

MilliSec/Write operation in milliseconds.

K KAVG/wr Average Kernel Average ESXi VMkernel write latency per

MilliSec/Write write operation in milliseconds.

K GAVG/wr Average Guest Average guest operating system write

MilliSec/Write latency per write operation in milliseconds.
Key Metric Name Description

K QAVG/wr Average Queue Average queue write latency per write

MilliSec/Write operation in milliseconds.

L FCMDS/s Failed Commands/sec

L FREAD/s Failed Reads/sec

L FWRITE/s Failed Writes/sec

L FMBRD/s Failed Bytes Read/sec

L FMBWR/s Failed Bytes

Written/sec

L FRESV/s Failed Reserves/sec

L ABRTS/s Aborts/sec Number of commands aborted per second.

L RESETS/s Resets/sec Number of commands reset per second.

M PAECMD/s PAE Commands/sec Number of PAE commands per second. This

statistic applies to only paths.

M PAECP/s PAE Copies/sec Number of PAE copies per second. This

statistic applies to only paths.

N SPLTCMD/s Split Commands/sec Number of split commands per second.

This statistic applies to only paths.

N SPLTCP/s Split Copies/sec Number of split copies per second. This

statistic applies to only paths.
Key Metric Name Description

O CLONE_RD VAAI Clone Reads

O CLONE_WR VAAI Clone Writes

O CLONE_F VAAI Clone Failed

O MBC_RD/s MBytes Clone

Reads/sec

O MBC_WR/s MBytes Clone

Writes/sec

O ATS Atomic Test and Set

O ATSF Atomic Test and Set

Failed

O ZERO Zeros

O ZERO_F Zeros Failed

O MBZERO/s MBytes Zeroed/sec

O DELETE Delete

O DELETE_F Delete Failed

O MBDEL/s Mbytes Delete/sec

P CAVG/suc Average Success

Latency ms/Clone
Key Metric Name Description

P CAVG/f Average Failure

Latency ms/Clone

P AAVG/suc Average Success

Latency ms/ATS

P AAVG/f Average Failure

Latency ms/ATS

P ZAVG/suc Average Success

Latency ms/Zero

P ZAVG/f Average Failure

Latency ms/Zero

Disk (Adapter) Panel

Key Metric Name Description

A ADAPTR Adapter

B PATH Path

C NPTH Number of Paths

D AQLEN Adapter Q Depth

E CMDS/s Commands/sec

E READS/s Reads/sec

E WRITES/s Writes/sec
Key Metric Name Description

E MBREAD/s MBytes Read/sec

E MBWRTN/s MBytes Written/sec

F RESV/s Reserves/sec

F CONS/s Conflicts/sec

G DAVG/cmd Average Driver MilliSec/Command

G KAVG/cmd Average Kernel MilliSec/Command

G GAVG/cmd Average Guest MilliSec/Command

G QAVG/cmd Average Queue MilliSec/Command

H DAVG/rd Average Driver MilliSec/Read

H KAVG/rd Average Kernel MilliSec/Read

H GAVG/rd Average Guest MilliSec/Read

H QAVG/rd Average Queue MilliSec/Read

I DAVG/wr Average Driver MilliSec/Write

I KAVG/wr Average Kernel MilliSec/Write

I GAVG/wr Average Guest MilliSec/Write

Key Metric Name Description

I QAVG/wr Average Queue MilliSec/Write

J FCMDS/s Failed Commands/sec

J FREAD/s Failed Reads/sec

J FWRITE/s Failed Writes/sec

J FMBRD/s Failed Bytes Read/sec

J FMBWR/s Failed Bytes Written/sec

J FRESV/s Failed Reserves/sec

J ABRTS/s Aborts/sec

J RESETS/s Resets/sec

K PAECMD/s PAE Commands/sec

K PAECP/s PAE Copies/sec

L SPLTCMD/s Split Commands/sec

L SPLTCP/s Split Copies/sec

Network Panel

Key Metric Name Description

A PORT-ID Port ID The port ID used on the virtual switch. This ID is used for
example for network traces with pktcap-uw.
Key Metric Name Description

Press 'N' to sort by PORT-ID column.

B UPLINK Uplink? This virtual switch port is a physical uplink port. N means
no, Y means yes.

C UP Link Up? Y means the corresponding link is up. N means it is not.

Only visible for uplink ports.

C SPEED Link Speed (Mb/s) Link speed in Megabits per second. Only visible for uplink
ports.

C FDUPLX Full Duplex? Y means the corresponding link is operating at full duplex.
N means it is not. Only visible for uplink ports.

D USED-BY Used By Indicates what is connected to the virtual switch port.

Connected devices can be virtual machine NICs, VMkernel
ports (vmk#), physical NICs (vmnic#) or ports used for
health checks (Shadow of vmnic#).

E TEAM-PNIC Team Uplink Physical The physical NIC that the corresponding device is actively
NIC Name using. This information is helpful for network
troubleshooting.

F DNAME Device Name Virtual Switch name. Either the display name for virtual
switches or DvsPortset-# for distributed switches.

G PKTTX/s Packets Number of packets transmitted per second.

Transmitted/sec
Press 't' to sort by PKTTX/s column.

H MbTX/s MBits MegaBits transmitted per second.

Transmitted/sec
Key Metric Name Description

Press 'T' to sort by MbTX/s column.

I PSZTX Average Packet Size Average size of transmitted packets in bytes.

Transmitted (Bytes)

J PKTRX/s Packets Received/sec Number of packets received per second.

Press 'r' to sort by PKTRX/s column.

K MbRX/s MBits Received/sec MegaBits received per second.

Press 'R' to sort by MbRX/s column.

L PSZRX Average Packet Size Average size of received packets in bytes.

Received (Bytes)

M %DRPTX % Outbound Packets Percentage of transmit packets dropped.

Dropped

N %DRPRX % Received Packets Percentage of receive packets dropped.

Dropped

O ACTN/s Actions Posted/sec Number of Vmkernel actions posted per second. This is an
VMware internal counter with no further documentation.

P PKTTXMUL/s Multicast Packets Number of multicast packets transmitted per second.

Transmitted/sec

P PKTRXMUL/s Multicast Packets Number of multicast packets received per second.

Received/sec

Q PKTTXBRD/s Broadcast Packets Number of broadcast packets transmitted per second.

Transmitted/sec
Key Metric Name Description

Q PKTRXBRD/s Broadcast Packets Number of broadcast packets received per second.

Received/sec