• Powering on a virtual machine fails.
  • Unable to power on a virtual machine.
  • Adding an existing virtual machine disk (VMDK) to a virtual machine that is already powered on fails.

You see the error:

Cannot open the disk ‘/vmfs/volumes/UUID/VMName/Test-000001.vmdk’ or one of the snapshot disks it depends on. Reason: Failed to lock the file.

Cause:

+++++++

This issue occurs when one of the files required by the virtual machine has been opened by another application.

During a Create or Delete Snapshot operation while a virtual machine is running, all the disk files are momentarily closed and reopened. During this window, the files could be opened by another virtual machine, management process, or third-party utility. If that application creates and maintains a lock on the required disk files, the virtual machine cannot reopen the file and resume running.

Resolution:

+++++++++++

If the file is no longer locked, try to power on the virtual machine again. This should succeed. To determine the cause of the previously locked file, review the VMkernel, hostd, and vpxa log files and attempt to determine:

  • When the hostd and vpxa management agents open VMDK descriptor files, they log messages similar to:info ‘DiskLib’] DISKLIB-VMFS : “/vmfs/volumes/UUID/VMName/Test-000001.vmdk” : open successful (21) size = 32227695616, hd = 0. Type 8
    info ‘DiskLib’] DISKLIB-VMFS : “/vmfs/volumes/UUID/VMName/Test-000001.vmdk” : closed.
  • When the VMkernel attempts to open a locked file, it reports:31:16:46:55.498 cpu7:8715)FS3: 2928: [Requested mode: 2] Lock [type 10c00001 offset 11401216 v 2035, hb offset 3178496
    gen 26643, mode 1, owner 4ca72d14-84dc8dd4-0da3-0017a4770038 mtime 2213195] is not free on volume ‘norr_prod_vmfs_data08’
  • The file may have been locked by third-party software running on an ESXi/ESX host or externally. Review the logs of any third-party software that may have acted on the virtual machine’s VMDK files at the time.

Situation 1:

++++++++

Error : Failed to get exclusive lock on the configuration file, another VM process could be running, using this configuration file

Solution : This issue may occur if there is a lack of disk space on the root drive. The ESX host is unable start a virtual machine because there is insufficient disk space to commit changes.

Situation 2:

++++++++

Error : Failed to lock the file when creating a snapshot

Solution :

To work around this issue in ESX or earlier ESXi releases, Use the vmkfstools -D command to identify the MAC address of the machine locking the file, then reboot or power off the machine that owns that MAC address to release the lock.

Notes: 

  • If the vmkfstools -D test-000001-delta.vmdk command does not return a a valid MAC address in the top field (returns all zeros), review the RO Owner line below it to see which MAC address owns the read only/multi writer lock on the file.
  • In some cases, it may be a Service Console-based lock, an NFS lock, or a lock generated by another system or product that can use or read the VMFS file systems. The file is locked by a VMkernel child or cartel world and the offending host running the process/world must be rebooted to clear it.
  • After you have identified the host or backup tool (machine that owns the MAC) locking the file, power it off or stop the responsible service and then restart the management agents on the host running the virtual machine to release the lock.

Situation 3:

+++++++++

Error : Failed to add disk scsi0:1. Failed to power on scsi0:1

To prevent concurrent changes to critical virtual machine files and file systems, ESXi/ESX hosts establish locks on these files. In certain circumstances, these locks may not be released when the virtual machine is powered off. The files cannot be accessed by the servers while locked, and the virtual machine is unable to power on.

These virtual machine files are locked during runtime:

  • VMNAME.vswp
  • DISKNAME-flat.vmdk
  • DISKNAME-ITERATION-delta.vmdk
  • VMNAME.vmx
  • VMNAME.vmxf
  • vmware.log

>> There is a manual procedure to locate the host and virtual machine holding locks.

To work around this issue, run the vmfsfilelockinfo script from the host experiencing difficulties with one or more locked files:

  1. To find out the IP address of the host holding the lock, run the /bin/vmfsfilelockinfo Python script. The script takes these parameters:
    • File being tested
    • Username and password for accessing VMware vCenter Server (when tracing MAC address to ESX host.)For example:

      Run this command:

      ~ # vmfsfilelockinfo -p /vmfs/volumes/iscsi-lefthand-2/VM1/VM1_1-000001-delta.vmdk -v 192.168.1.10 -uadministrator@vsphere.local

      You see ouput similar to:

      vmfsflelockinfo Version 1.0
      Looking for lock owners on “VM1_1-000001-delta.vmdk”
      “VM1_1-000001-delta.vmdk” is locked in Exclusive mode by host having mac address [‘xx:xx:xx:xx:xx:xx’]
      Trying to make use of Fault Domain Manager
      ———————————————————————-
      Found 0 ESX hosts using Fault Domain Manager.
      ———————————————————————-
      Could not get information from Fault domain manager
      Connecting to 192.168.1.10 with user administrator@vsphere.local
      Password: xXxXxXxXxXx
      ———————————————————————-
      Found 3 ESX hosts from Virtual Center Server.
      ———————————————————————-
      Searching on Host 192.168.1.178
      Searching on Host 192.168.1.179
      Searching on Host 192.168.1.180
      MAC Address : xx:xx:xx:xx:xx:xx

      Host owning the lock on the vmdk is 192.168.1.180, lockMode : Exclusive

      Total time taken : 0.27 seconds.

      Note: During the life-cycle of a powered on virtual machine, several of its files transitions between various legitimate lock states. The lock state mode indicates the type of lock that is on the file. The list of lock modes is:

    • mode 0 = no lock
    • mode 1 = is an exclusive lock (vmx file of a powered on virtual machine, the currently used disk (flat or delta), *vswp, and so on.)
    • mode 2 = is a read-only lock (For example on the ..-flat.vmdk of a running virtual machine with snapshots)
    • mode 3 = is a multi-writer lock (For example used for MSCS clusters disks or FT VMs)
  2. To get the name of the process holding the lock, run the lsof command on the host holding the lock and filter the output for the file name in question:~ # lsof | egrep ‘Cartel|VM1_1-000001-delta.vmdk’

    You see output similar to:

    Cartel | World name | Type | fd | Description
    36202 vmx FILE 80 /vmfs/volumes/556ce175-7f7bed3f-eb72-000c2998c47d/VM1/VM1_1-000001-delta.vmdk

    This shows that the file is locked by a virtual machine having Cartel ID 36202. Now display the list of active Cartel IDs by executing this command:

    ~ # esxcli vm process list

    This displays information for active virtual machines grouped by virtual machine name and having a format similar to:

    Alternate_VM27
    World ID: 36205
    Process ID: 0
    VMX Cartel ID: 36202
    UUID: 56 4d bd a1 1d 10 98 0f-c1 41 85 ea a9 dc 9f bf
    Display Name: Alternate_VM27
    Config File: /vmfs/volumes/556ce175-7f7bed3f-eb72-000c2998c47d/Alternate_VM27/Alternate_VM27.vmx
    Alternate_VM20
    World ID: 36207
    Process ID: 0
    VMX Cartel ID: 36206
    UUID: 56 4d bd a1 1d 10 98 0f-c1 41 85 ea a5 dc 94 5f
    Display Name: Alternate_VM20
    Config File: /vmfs/volumes/556ce175-7f7bed3f-eb72-000c2998c47d/Alternate_VM20/Alternate_VM20.vmx

    The virtual machine entry having VMX Cartel ID 36202 shows the display name of the virtual machine holding the lock on file VM1_1-000001-delta.vmdk, which in this example, is Alternate_VM27.

  3. Shut down the virtual machine holding the lock to release the lock.

Related Information

This script performs these actions in this sequence:

  1. Identifies locked state Exclusive, Read-Only, not locked.
  2. Identifies MAC address of locking host [‘xx:xx:xx:xx:xx:xx’].
  3. Queries the Fault Domain Manager (HA) for information on discovered MAC address.
  4. Queries vCenter Server for information on discovered MAC address.
  5. Outputs final status.
    For example:

Host owning the lock on the vmdk is 192.168.1.180, lockMode : Exclusive.

  • The script outputs total execution time when it terminates.

 

Notes:

  • The script does not attempt to break/remove locks. The script only identifies the potential ESX host which holds the lock.
  • If not run with vCenter Server username and password, it prompts for the same, after querying the Fault Domain Manager.
  • This script works on a single file parameter, without wildcards. If multiple queries are required, you must execute the script repeatedly in a wrapper script.

For further clarifications please follow : https://kb.vmware.com/s/article/10051