A Complete List of VMware VAAI Primitives

VMware’s introduced the “vStorage APIs for Array Integration” (VAAI) in vSphere 4.1, and block-heads like me went nuts. We’ve been trying to integrate storage and servers for decades, and VMware’s APIs finally allowed this to work in truly seamless fashion. But the world of VAAI is a thicket of bizarre naming and puzzling functionality. Some VAAI primitives are ignored or even hidden! Let’s take a look at the complete list.

VAAI in 4.1: Three For Block (Plus One More)

vSphere 4.1 officially includes three VAAI primitives: Full Copy, Hardware Assisted Locking, and Block Zeroing. But it was intended to include a fourth (Thin Provisioning Stun) and variations in nomenclature have confused administrators.

Here’s the complete list of VAAI primitives in vSphere 4.1, including a little more about what they really do.

Atomic Test & Set (ATS) (AKA Hardware Assisted Locking or HardwareAcceleratedLocking)

This is one of the most valuable storage technologies to come out of VMware, but it’s usually overlooked. It allows compare and write of SCSI blocks in one operation using proprietary array-specific op-codes. This enables granular locking of block storage devices beyond the basic full-LUN “reservations” included in SCSI from days of yore. This is huge for performance when LUNs are shared, with HP reporting at VMworld allowing 6 times more VMs per LUN once VAAI is turned on. NFS doesn’t really need this since locking is a non-issue and VM files aren’t shared the same way LUNs are.

To verify from the CLI:

esxcfg-advcfg -g /VMFS3/HardwareAcceleratedLocking

Cloning Blocks (AKA Full Copy, Extended Copy, or HardwareAcceleratedMove)

This is the signature VAAI command. Instead of reading data from the array then writing it back, the hypervisor can command the array to make a mirror of a range of data on its behalf. If supported and enabled, operations like Clone and VMotion can be lightning quick, since many high-end arrays are very capable of this sort of action. This leverages the SCSI XCOPY command (or other proprietary alternatives enabled with a VMware plug-in) and defaults to a 4 MB block size.

To verify from the CLI:

esxcfg-advcfg -g /DataMover/HardwareAcceleratedMove

Zeroing File Blocks (AKA Block Zeroing or HardwareAcceleratedInit)

Thin provisioning is difficult to get right because storage arrays have no idea what’s going on in the hosts. VAAI includes a generic interface for communication of free space, allowing vast ranges of blocks to be zeroed out at once. Along with the generic T10 “WRITE_SAME” command, VAAI allows vendors to use “UNMAP” or their own mechanisms with a plug-in. VMware defaults to a 1 MB block size for zeroing operations. But zeroing only works for supported block arrays, and only for capacity inside a VMDK: Clones and vMotions leave the array guessing.

To verify from the CLI:

esxcfg-advcfg -g /DataMover/HardwareAcceleratedInit

Out of Space Condition (AKA Thin Provisioning Stun)

Running out of capacity is a catastrophe, but it’s easy to ignore the alerts in vCenter until it’s too late. This command allows the array to notify vCenter to “stun” (suspend) all virtual machines on a LUN that is running out of space due to thin provisioning over-commit. This is the “secret” fourth primitive that wasn’t officially acknowledged until vSphere 5 but apparently existed before. In vSphere 5, this works for both block and NFS storage. Signals are sent using SCSI “Check Condition” operations.

VAAI Commands in vSphere 4.1

esxcli corestorage device list

esxcli vaai device list

esxcli corestorage plugin list

VAAI in 5.0: Lots More

vSphere 5 includes a new generation of VAAI primitives, including support for NFS storage. Once again, VMware includes a “secret” primitive not yet implemented, and puzzlingly lumps some of the official entries together in company documentation. But VMworld changed that, with one presentation including insight into the real list of VAAI primitives.

vSphere 5 includes and enhances all the VAAI primitives from vSphere 4.1 (including “Out of Space Condition”) as well as the following new capabilities.

vSphere 5 includes the above primitives, plus these

UNMAP (AKA Space Reclamation or EnableBlockDelete)

Zeroing File Blocks is great, but VMFS operations like cloning and vMotion didn’t include any hints to the array. This means that some of the biggest storage operations aren’t accelerated or thinned out! UNMAP is a SCSI command, and it was added to VAAI to allow thin-capable arrays to clear unused VMFS space as well. But vCenter couldn’t handle waiting for the array to return “UNMAP” command status, so the use of this primitive is officially suspended by VMware until this issue is resolved.

To verify from the CLI:

esxcfg-advcfg -g /VMFS3/EnableBlockDelete

For now, disable it:

esxcfg-advcfg —s 0 /VMFS3/EnableBlockDelete

Quota Exceeded Behavior (AKA TP Soft Threshold)

It would be nice if vSphere could react before an out-of-space condition is reached. This primitive allows the array to notify the hypervisor preemptively, possibly triggering a Storage DRS re-balance, for example. It also allows the hypervisor to query the array for its threshold values using SCSI “mode sense”.

TP LUN Reporting (AKA Report Thin Capacity)

A very simple command, this allows the hypervisor to query the array to determine if a LUN is thinly provisioned. On block arrays, the SCSI “TPE” bit is used, but this also works with NFS.

NFS Full File Clone (AKA Hardware Snap)

NFS already did a nice job with storage provisioning and presentation, but VMware couldn’t make use of hardware snapshot capability. This changes in vSphere 5, with a “Full Copy”-like mechanism appearing for NAS. But the NFS version doesn’t support Storage vMotion operations like its block cousin.

NFS Space Reservation (AKA Zeroedthick Files)

NFS is always thin, which is a good and bad thing. This new primitive adds two new file type for NFS-based VMDK files: “Eager” and “Lazy” Zeroed Thick. These are similar to the old default “thick” volume in block storage, but thick provisioning and space reservation is brand new territory for NFS admins. This primitive uses the NFS “reserve_space” command.

NFS Extended Stat (AKA Extended Status)

One-upping the SCSI block world, vSphere 5 can query NAS servers for all sorts of information, including file size. This should make NFS in vCenter even more friendly!

NFS Space Reclaim (AKA Space Reclamation)

Although the block-only UNMAP feature doesn’t work, the NFS equivalent Space Reclaim seems fine. This enables the hypervisor to unlink old files once they’re no longer used. This primitive uses the NFS “unlink” command.

NFS Fast File Clone (AKA Linked Clone Offload)

This “secret” VAAI primitive allows a future release of VMware View (and only View) to offload creation of linked clones to a NAS array. Hopefully we’ll see more of this in the future!

VAAI Commands in vSphere 5

esxcli storage core device list

esxcli storage core device vaai status get

esxcli storage core plugin list