Currently Being Moderated

UUID Background

UUIDs are used in many places in technology, especially in VMware, wherever an object needs a universally unique identifier, hence the name UUID.  Within VMware, they are used to identify VMs internally to vCenter and to themselves and network provisioning systems in the form of BIOS UUIDs.  They identify a virtual disk to a VM, a LUN to ESX, a datastorewithin the storage, and give the ESX server itself a unique ID.  Autodeploy can use a physical server's BIOS UUID as an identifier to determine which specific ESXi image to apply.


In most cases, especially according to the standards, a UUID is a 16-byte number, usually represented by groups of hexadecimal digits separated by dashes.

~# grep /system/uuid /etc/vmware/esx.conf

/system/uuid= "4db5a322-5a40-57c7-43fd-00262d06d39e"

 

Datastore UUIDs

The storage infrastructure of ESX and ESXi uses UUIDs to uniquely and persistently identify a datastore in order to consistently mount and access it.  Looking in /vmfs/volumes, you can see these UUIDs along with the human readable labels or names created mostly by the administrator.  Note that the label is simply a symlink to the UUID.

 

/vmfs/volumes# ls -l

drwxr-xr-t    1 root    root               1260 Apr 2523:18 4daa9d0d-5dadd97c-80b8-00262d06d39f

drwxr-xr-t    1 root    root               1260 Sep 2323:50 4deea4c3-065c4358-760f-001517dc13d9

drwxr-xr-t    1 root    root               1260 Sep 2303:56 4e7bba36-ca2d8566-7306-001517dc13db

drwxr-xr-t    1 root    root               1260 Sep 2322:43 4e7ceaca-0f58e6e0-808f-001517dc13db

drwxr-xr-x    1 root    root               4096 Sep 2223:38 7681e7dd-de42acdd

lrwxr-xr-x    1 root    root                 35 Sep 2800:30 datastore1 -> 4daa9d0d-5dadd97c-80b8-00262d06d39f

lrwxr-xr-x    1 root    root                 35 Sep 2800:30 f32big -> 4e7bba36-ca2d8566-7306-001517dc13db

lrwxr-xr-x    1 root    root                 35 Sep 2800:30 f32small -> 4e7ceaca-0f58e6e0-808f-001517dc13db

drwxrwxrwx    1 root    root               8192 Sep 2716:21 fffa8c3f-ee853c65

lrwxr-xr-x    1 root    root                 35 Sep 2800:30 frogstar1 -> 4deea4c3-065c4358-760f-001517dc13d9

lrwxr-xr-x    1 root    root                 17 Sep 2800:30 infra -> 7681e7dd-de42acdd

lrwxr-xr-x    1 root    root                 17 Sep 28 00:30 stuff -> fffa8c3f-ee853c65

 

 

Wait a minute!  You said they were 16-bytes, so the long ones make sense.  But what are the shorter ones?

 

 

I'll come back to that.

 

 

The datastore path including UUID is used in all references to VMs and their files internally in ESX.  The main index of VMs is in /etc/vmware/hostd/vmInventory.xml

~# cat /etc/vmware/hostd/vmInventory.xml

<ConfigRoot>

  <ConfigEntry id="0000">

    <objID>1</objID>

    <vmxCfgPath>/vmfs/volumes/7681e7dd-de42acdd/esxitest/esxitest.vmx</vmxCfgPath>

  </ConfigEntry>

  <ConfigEntry id="0001">

    <objID>2</objID>

   <vmxCfgPath>/vmfs/volumes/7681e7dd-de42acdd/XP20110514/XP20110514.vmx</vmxCfgPath>

  </ConfigEntry>

</ConfigRoot>

 

 

VM files like the vmx, vmdk descriptor, snapshot descriptor and others, also use full path including UUID for any reference to a file not in the VM working dir.  An example is a VMDK or vswap file on a different datastore from the VM working dir.

sched.swap.derivedName= "/vmfs/volumes/4e7bba36-ca2d8566-7306-001517dc13db/winnebago-e56940a3.vswp"

 

 

Why use UUID?  The main reason is they can't be easily or arbitrarily changed by the VMware administrator. The human readable name can be changed at any time.  Imagine the havoc it would create if the datastore name changed and all the pointers to VMs and their component files got scrambled.  Yes, that has happened anyway due to changes on SANs and NAS storage networks, but it's not something that happens because somebody clicked a datastore name twice and typed something by accident.


 

So what's with the short UUIDs?

 

NFS Datastore UUIDs

 

Those are NFS datastores.  The NFS datastore UUID is generated very differently from other UUIDs.  Unlike other UUIDs which are composed from different elements such as a MAC address from the server that created it, timestamp, and random elements, the NFS datastore is generated as a hash of the admin-specified parameters passed to the mount call, whether through CLI, GUI or script/API.  The parameters are IP, hostname or FQDN of the NFS server, and the export path as specified on that NFS server.  Although many NFS servers, including NetApp, offer a unique volume ID, they are not used by ESX.  For example, looking at my NAS datastores from the command line

/vmfs/volumes# esxcfg-nas -l

stuff is /vol/stuff from 192.168.42.60 mounted available

 

fffa8c3f-ee853c65is a hash of "/vol/stuff" and "192.168.42.60".

 

There's an important point in here.  If you specify a different identifier for either of these, you get a different hash which means a different UUID, even if it all resolves to the same export on the same NFS server.  So, while "labrat" and "labrat.skunkworks.netapp.com" might both resolve to "192.168.42.60", you get a different hash / UUID, and ESX thinks it's a different datastore.  Same goes for the export path.  Using export aliases with the -actual export option or a trailing / in the export name also generates a different UUID.  These things, especially the trailing / played havoc in a lab at a VMworld event a couple years ago.  Watch this:

My normal "stuff" datastore

 

/vmfs/volumes# esxcfg-nas -l

stuff is /vol/stuff from 192.168.42.60 mounted available

/vmfs/volumes# ls -l stuff

lrwxr-xr-x    1 root    root                 17 Sep 2903:49 stuff -> fffa8c3f-ee853c65

 

Unmount it:

/vmfs/volumes# esxcfg-nas -d stuff

NAS volume stuff deleted.

 

Now mount with variations of NFS server and export

/vmfs/volumes# esxcfg-nas -a -o labrat-stg.vgibu.eng.netapp.com -s /vol/stuff stuff

Connecting to NAS volume: stuff

stuff created and connected.

/vmfs/volumes# ls -l stuff

lrwxr-xr-x    1 root    root                 17 Sep 2903:50 stuff -> 590fde0e-8fa5838f

/vmfs/volumes# esxcfg-nas -d stuff

NAS volume stuff deleted.

 

/vmfs/volumes# esxcfg-nas -a -o labrat-stg -s /vol/stuff stuff

Connecting to NAS volume: stuff

stuff created and connected.

/vmfs/volumes# ls -l stuff

lrwxr-xr-x    1 root    root                 17 Sep 2903:50 stuff -> 7fa9d18e-7b4829a0

/vmfs/volumes# esxcfg-nas -d stuff

NAS volume stuff deleted.

 

/vmfs/volumes# esxcfg-nas -a -o labrat-stg -s /vol/stuff/ stuff

Connecting to NAS volume: stuff

stuff created and connected.

/vmfs/volumes# ls -l stuff

lrwxr-xr-x    1 root    root                 17 Sep 2903:50 stuff -> bb5cc5d6-6e612683

 

Did you notice the trailing / on the share on the last example?  I missed it the first few times in that lab, too, then had one of those "Holy smokes!" moments.

 

Ever since VMware first supported NFS for VM datastores, we have recommended that the same designation be used for host and export, consistently, across all servers using the same storage.  We've generally recommended using IP addresses for a private storage network or VLAN, in order to eliminate dependencies on DNS.  However, there are situations where it may be desirable or necessary to use a different IP address for the same storage for different servers. This may be to trick the load balancing algorithms, or in the case where servers are on multiple VLANs or subnets accessing a common storage device with connections on both networks.  At VMworld 2007, we talked about this a little.  You can have /etc/hosts entries with the same hostname for the filer, or use DNS, either round-robin or configured to provide specific, different storage IPs to servers in response to their queries. You enter the same hostname or FQDN on the command line or GUI for all servers, so the UUID works out the same, but you get the desired IP and resulting network path.  As far as ESX is concerned, that's all fine.

 

Breaking vMotion and HA

The gotcha is that vCenter comes along and looks deeper than just UUID and datastore name.  When vCenter assimilates an ESX server and its datastores, or new datastores on existing servers, if two ESX servers have what otherwise appear to be the same datastore, except using hostname or FQDN resolving to different IPs, it will rename the second (and subsequent) occurrence(s) of the datastore appending"(1)", similar to what is seen with default local datastores.  The result is that what we intended to be the same datastore is treated as two different datastores, and advanced features like vMotion and HA will fail for VMs using that datastore.

 

Changing IPs

Another problem with the way NFS UUIDs are generated is that if a storage network is changed, perhaps to solve a problem with IP depletion, to support a 1g to 10g migration, or to implement best practices separating storage traffic from other traffic, and IP addresses used by ESX for storage change, the datastore UUIDs will also change. In the vSphere Client, "all your VM are belong to us" - I mean inaccessible.  This is because the datastore path to the VMX as specified in vmInventory.xml no longer exists.  If you have to make such a change that changes UUIDs, you will have to fix VMs, either by removing and re-adding each VM, or by editing vmInventory.xml.  You'll also have to change other references in VMX and other files.  Editing vmInventory.xml is risky because other things change it on an ongoing basis, like registering/removing VMs, vMotion, HA and DRS.  If you have it open in vi, and one of those other tools adds or removes a VM, you will end up with an inconsistent vmInventory.  I've written a script to search datastores, and reconcile what's in vmInventory.xml with what's really on the datastores, but I doubt it's something VMware will want to support.

 

So, what's new in vSphere 5?

With vSphere 5, VMware has made some changes in NFS datastore IP, UUID and name handling.  First, nothing has changed in ESXi itself. Datastore UUIDs are still generated using the same hash of NFS server spec and export/share.  What's different is vCenter no longer second-guesses admins who trick ESX(i) with host or DNS entries that resolve a different IP for different ESXi servers going to the same NFS storage.  Now, if you're using these tricks, NFS datastores do NOT get renamed, they are treated as the same datastore, and vMotion, HA, DRS all work as expected.

 

So, they fixed the vCenter issue, but the underlying UUID generation algorithm in ESXi is still the same.  In fact, all the examples in this post were done using ESXi 5.0 GA.

 

Share and enjoy!

Comments