This is pain point for most of us on how to isolate and understand who is the culprit on performance issue. I will try to add all the points together and summarize .

Symptoms:
++++++++++
>>Services running in guest virtual machines respond slowly.
>>Applications running in the guest virtual machines respond intermittently.
>>The guest virtual machine may seem slow or unresponsive.

There are 4 major things which deal with the performance:

1: CPU
2: Memory
3: Storage
4: Networking

1:CPU:

>>Use the esxtop command to determine if the ESXi/ESX server is being overloaded.
>>Examine the load average on the first line of the command output.
A load average of 1.00 means that the ESXi/ESX Server machine’s physical CPUs are fully utilized, and a load average of 0.5 means that they are half utilized. A load average of 2.00 means that the system as a whole is overloaded.
>>Examine the %READY field for the percentage of time that the virtual machine was ready but could not be scheduled to run on a physical CPU.

For more info please follow : https://kb.vmware.com/s/article/1033115?r=2&KM_Utility.getArticleLanguage=1&KM_Utility.getArticle=1&KM_Utility.getGUser=1&KM_Utility.getArticleData=1&Quarterback.validateRoute=1

If the load average is too high, and the ready time is not caused by CPU limiting, adjust the CPU load on the host. To adjust the CPU load on the host, either:

Increase the number of physical CPUs on the host
OR
Decrease the number of virtual CPUs allocated to the host. To decrease the number of virtual CPUs allocated to the host, either:

Reduce the total number of CPUs allocated to all of the virtual machines running on the ESX host. For more information, see Determining if multiple virtual CPUs are causing performance issues (1005362).
OR
Reduce the number of virtual machines running on the host.

2:Memory:

>>Use the esxtop command to determine whether the ESXi/ESX server’s memory is overcommitted.
>>Examine the MEM overcommit avg on the first line of the command output. This value reflects the ratio of the requested memory to the available memory, minus 1.

Examples:

If the virtual machines require 4 GB of RAM, and the host has 4 GB of RAM, then there is a 1:1 ratio. After subtracting 1 (from 1/1), the MEM overcommit avg field reads 0. There is no overcommitment and no extra RAM is required.
If the virtual machines require 6 GB of RAM, and the host has 4 GB of RAM, then there is a 1.5:1 ratio. After subtracting 1 (from 1.5/1), the MEM overcommit avg field reads 0.5. The RAM is overcommited by 50%, meaning that 50% more than the available RAM is required.

>>Determine whether the virtual machines are ballooning and/or swapping.

To detect any ballooning or swapping:

++Run esxtop.
++Type m for memory
++Type f for fields
++Select the letter J for Memory Ballooning Statistics (MCTL)
++Look at the MCTLSZ value.

MCTLSZ (MB) displays the amount of guest physical memory reclaimed by the balloon driver.

++Type f for Field
++Select the letter for Memory Swap Statistics (SWAP STATS).
++Look at the SWCUR value.

SWCUR (MB) displays the current Swap Usage.

3:Storage:

To determine whether the poor performance is due to storage latency:

>>Determine whether the problem is with the local storage. Migrate the virtual machines to a different storage location.
>>Reduce the number of Virtual Machines per LUN.
>>Look for log entries in the Windows guests that look like this:

The device, \Device\ScsiPort0, did not respond within the timeout period.
>>Using esxtop, look for a high DAVG latency time
>>Determine the maximum I/O throughput you can get with the iometer command.
>>Compare the iometer results for a VM to the results for a physical machine attached to the same storage.
>>Check for SCSI reservation conflicts.
>>If you are using iSCSI storage and jumbo frames, ensure that everything is properly configured.
>>If you are using iSCSI storage and multipathing with the iSCSI software initiator, ensure that everything is properly configured.

4:Network:

Network performance can be highly affected by CPU performance. Rule out a CPU performance issue before investigating network latency.

To determine whether the poor performance is due to network latency:

>>Test the maximum bandwidth from the virtual machine with the Iperf tool. This tool is available from https://github.com/esnet/iperf

Note: VMware does not endorse or recommend any particular third-party utility.

While using Iperf, change the TCP windows size to 64 K. Performance also depends also on this value. To change the TCP windows size:

On the server side, enter this command:

#iperf -s

On the client side, enter this command:

#iperf.exe -c sqlsed -P 1 -i 1 -p 5001 -w 64K -f m -t 10 900M

 

Some additional Information on ESXTOP:
++++++++++++++++++++++++++++++++++++++

Configuring monitoring using esxtop:
====================================
To monitor storage performance per HBA:

>>Start esxtop by typing esxtop at the command line.
>>Press d to switch to disk view (HBA mode).
>>To view the entire Device name, press SHIFT + L and enter 36 in Change the name field size.
>>Press f to modify the fields that are displayed.
>>Press b, c, d, e, h, and j to toggle the fields and press Enter.
>>Press s and then 2 to alter the update time to every 2 seconds and press Enter.

To monitor storage performance on a per-LUN basis:

>>Start esxtop by typing esxtop from the command line.
>>Press u to switch to disk view (LUN mode).
>>Press f to modify the fields that are displayed.
>>Press b, c, f, and h to toggle the fields and press Enter.
>>Press s and then 2 to alter the update time to every 2 seconds and press Enter.

 

For more information please follow the VMware KB : https://kb.vmware.com/s/article/1008205