Here's a quick way of finding which system has a specified IP address assigned to it. You have a rogue IP address and don't know which VM is assigned to it.
$match_address = "192.168.10.27"
Get-VM | %{
$vmIPs = $_.Guest.IPAddress
foreach($ip in $vmIPs) {
if ($ip -eq $match_address) {
"Found VM with matching address: {0}" -f $_.Name
}
}
}
Monday, November 23, 2015
PowerCLI script to query VM for SnapShots and sizes.
Here's a simple script to query all your VM's that currently have snapshots. 5 columns are created starting from VM, SnapShotName, Snapshot Description, Size (Snapshot), Date Created.
$Report = Get-VM | Get-Snapshot | Select VM,Name,Description,@{Label="Size";Expression={"{0:N2} GB" -f ($_.SizeGB)}},Created
If (-not $Report)
{ $Report = New-Object PSObject -Property @{
VM = "No snapshots found on any VM's controlled by $VIServer"
Name = ""
Description = ""
Size = ""
Created = ""
}
}
$Report = $Report | export-csv -path c:\tmp\vmware\snapshotsDAYMONTH.csv
Append "| export-csv -path c:\tmp\vmware\vm_tools_upgrade.csv" to the end of the script for viewing and reporting purposes.
PS > Snapshot.ps1
$Report = Get-VM | Get-Snapshot | Select VM,Name,Description,@{Label="Size";Expression={"{0:N2} GB" -f ($_.SizeGB)}},Created
If (-not $Report)
{ $Report = New-Object PSObject -Property @{
VM = "No snapshots found on any VM's controlled by $VIServer"
Name = ""
Description = ""
Size = ""
Created = ""
}
}
$Report = $Report | export-csv -path c:\tmp\vmware\snapshotsDAYMONTH.csv
Append "| export-csv -path c:\tmp\vmware\vm_tools_upgrade.csv" to the end of the script for viewing and reporting purposes.
PS > Snapshot.ps1
PowerCLI script to show VM's needing VMware Tools upgraded.
I started looking for a way to inventory all of our VM's currently running within our 6-7 clusters.
Most of our VM's had the latest version of VMware tools but there were plenty that were out of compliance. I didn't want to take the time to individually check each VM (500+ VM's) so after some searching I found a script that showed me the tools version on the VM's that were out of compliance.
# Select Cluster you wish to scan "User VMs"#
PS > get-view -ViewType virtualmachine -SearchRoot (get-cluster "User VMs").id -Filter @{'Summary.Guest.ToolsVersionStatus'='guestToolsNeedUpgrade'} | Select Name,@{N='VMware Tools Version';E={$_.Config.Tools.ToolsVersion}},@{N='Tools Status';E={$_.Summary.Guest.ToolsVersionStatus}}
Append "| export-csv -path c:\tmp\vmware\vm_tools_upgrade.csv" for viewing and reporting purposes.
The output would show:
Name VMware Tools Version Tools Status
VM_1 8394 guestToolsNeedUpgrade
VM_2 9344 guestToolsUpgrade
Most of our VM's had the latest version of VMware tools but there were plenty that were out of compliance. I didn't want to take the time to individually check each VM (500+ VM's) so after some searching I found a script that showed me the tools version on the VM's that were out of compliance.
# Select Cluster you wish to scan "User VMs"#
PS > get-view -ViewType virtualmachine -SearchRoot (get-cluster "User VMs").id -Filter @{'Summary.Guest.ToolsVersionStatus'='guestToolsNeedUpgrade'} | Select Name,@{N='VMware Tools Version';E={$_.Config.Tools.ToolsVersion}},@{N='Tools Status';E={$_.Summary.Guest.ToolsVersionStatus}}
Append "| export-csv -path c:\tmp\vmware\vm_tools_upgrade.csv" for viewing and reporting purposes.
The output would show:
Name VMware Tools Version Tools Status
VM_1 8394 guestToolsNeedUpgrade
VM_2 9344 guestToolsUpgrade
PowerCLI script to collect "Computers that have a specific OS installed"
I started looking for a way to inventory all of our VM's currently running within our 6-7 clusters. There was such a variety of OS's running that I didn't want to take the time to individually check the setting on each VM (500+ VM's). So after looking at different scripts, I took a combination of two or three and after trail and error I was able to filter my search results for exactly what I was looking for. In the end the script basically spits out the version of OS currently installed and IP Address of the VM. The IP address was critical in my search because we are running several different subnets and I needed to know exactly where the VM resided.
I then exported the results to my local C: drive in CSV format for viewing and reporting purposes.
# The *Cent* is changed to reflect what flavor of OS you may be looking for. *Red Hat*, *Server*, *Workstation*,etc..#
PS > get-vm | get-view | where {$_.Guest.GuestFullName -like "*Cent*"} | select Name,@{N="OS";E={@($_.Guest.GuestFullName)}},@{N="IP";E={@($_.Guest.IPAddress)}} | export-csv -path c:\tmp\vmware\CentOS.csv
I then exported the results to my local C: drive in CSV format for viewing and reporting purposes.
# The *Cent* is changed to reflect what flavor of OS you may be looking for. *Red Hat*, *Server*, *Workstation*,etc..#
PS > get-vm | get-view | where {$_.Guest.GuestFullName -like "*Cent*"} | select Name,@{N="OS";E={@($_.Guest.GuestFullName)}},@{N="IP";E={@($_.Guest.IPAddress)}} | export-csv -path c:\tmp\vmware\CentOS.csv
Tuesday, November 17, 2015
VMWare View Windows 7 Optimization Install
Preparing a new virtual machine
1. Create New Virtual Machine – FILE > NEW > VIRTUAL MACHINE (CTRL+N)
2. Under Configuration select CUSTOM.
3. Select a Name, Folder, Host, Cluster, and Storage.
4. Virtual Machine Version: vmx-10
5. Under Guest Operating System: Windows
a. Version: Windows 7 64bit
6. Number of virtual sockets: 1
a. Number of cores per virtual socket: 1
b. Total number of cores: 1
7. Memory Configuration 4GB
8. Select which Network (VLAN) and under Adapter select VMXNET 3.
9. SCSI Controller: LSI Logic SAS
10. Disk: Create a new virtual disk
a. Configure disk size to 32GB THIN provisioned.
11. Advanced Options keep defaults
12. Check “Edit the virtual machine settings before completion” and then Continue
13. Video Card: Do not “Auto detect” , set to 2 displays and 128 MB video memory Video Card: Enable 3D support
14. Remove Floppy Drive 1.
15. Go to Boot Options and check "The next time the virtual machine boots, force entry into the BIOS setup screen.
16. Click Finish
17. Power on virtual machine and open Console – virtual machine should be loaded into the BIOS screen.
2. Under Configuration select CUSTOM.
3. Select a Name, Folder, Host, Cluster, and Storage.
4. Virtual Machine Version: vmx-10
5. Under Guest Operating System: Windows
a. Version: Windows 7 64bit
6. Number of virtual sockets: 1
a. Number of cores per virtual socket: 1
b. Total number of cores: 1
7. Memory Configuration 4GB
8. Select which Network (VLAN) and under Adapter select VMXNET 3.
9. SCSI Controller: LSI Logic SAS
10. Disk: Create a new virtual disk
a. Configure disk size to 32GB THIN provisioned.
11. Advanced Options keep defaults
12. Check “Edit the virtual machine settings before completion” and then Continue
13. Video Card: Do not “Auto detect” , set to 2 displays and 128 MB video memory Video Card: Enable 3D support
14. Remove Floppy Drive 1.
15. Go to Boot Options and check "The next time the virtual machine boots, force entry into the BIOS setup screen.
16. Click Finish
17. Power on virtual machine and open Console – virtual machine should be loaded into the BIOS screen.
BIOS.
1. Under MAIN, Legacy Diskette A: change to Disabled.
2. Keyboard Features: change Numlock to ON.
3. Under ADVANCED, I/O Device Configuration: disable all Serial, and Parallel ports, as well as Floppy Controller.
4. Exit and Save Changes.
5. After Reboot Power Off and Enter Edit settings again.
6. CD/DVD: Device Type Datastore ISO File
a. Select the WIN764bit ISO
b. Check Connect at Power On
7. Click on OPTIONS tab and then General, UN-Check "Enable logging".
8. Finish
9. Power On
2. Keyboard Features: change Numlock to ON.
3. Under ADVANCED, I/O Device Configuration: disable all Serial, and Parallel ports, as well as Floppy Controller.
4. Exit and Save Changes.
5. After Reboot Power Off and Enter Edit settings again.
6. CD/DVD: Device Type Datastore ISO File
a. Select the WIN764bit ISO
b. Check Connect at Power On
7. Click on OPTIONS tab and then General, UN-Check "Enable logging".
8. Finish
9. Power On
Installing Windows 7
1. Boot from your Windows 7 installation media.
2. Click the "Next" button.
3. Click the "Install now" button.
4. Accept the terms and click the "Next" button.
5. Choose "Custom: Install (advanced)."
6. At “Where do you want to install Windows” screen, press Shift + F10 to open a command prompt.
7. At the command prompt, enter following commands:
a. Diskpart
b. Select disk 0
c. Create Partition Primary
d. assign
e. Exit
f. Exit
8. Click Refresh
9. Select "Disk 0 Partition 1"
10. Complete the install on the disk formatted with the 8192 allocation size.
2. Click the "Next" button.
3. Click the "Install now" button.
4. Accept the terms and click the "Next" button.
5. Choose "Custom: Install (advanced)."
6. At “Where do you want to install Windows” screen, press Shift + F10 to open a command prompt.
7. At the command prompt, enter following commands:
a. Diskpart
b. Select disk 0
c. Create Partition Primary
d. assign
e. Exit
f. Exit
8. Click Refresh
9. Select "Disk 0 Partition 1"
10. Complete the install on the disk formatted with the 8192 allocation size.
Installing VMWare Tools
1. Right click on virtual machine and select GUEST > Install/Upgrade VMware Tools.
2. Don’t Restart instead Shutdown virtual machine.
3. Enter Edit Settings once again and remove ISO on CD and then Remove CD/DVD Drive
2. Don’t Restart instead Shutdown virtual machine.
3. Enter Edit Settings once again and remove ISO on CD and then Remove CD/DVD Drive
Windows 7 Optimization
1. Go to Control panel and turn off all Action Center Settings and User Account Controls
2. Run Windows updates
3. After updates run Command.bat as Administrator found in the VMView Windows 7 Optimization http://www.vmware.com/resources/techresources/10157. I have also included it as an attachment.
4. Turn off Drive Indexing
a. Open the Start Menu
b. Click Computer
c. Right click C: drive and select Properties and uncheck drive indexing
d. Click apply changes to drive C:\, subfolders and files (ignore all when comes up)
5. Shrink Page File
a. Open the Start Menu
b. Right click Computer
c. Click Properties
d. Click Advanced System Settings (it’s on the left side)
e. Under performance click Settings
f. Go to the Advanced tab
g. Under Virtual memory click Change
h. Uncheck the "Automatically manage paging file" box at the top
i. Select the C: drive
j. Click "Custom size"; initial size 512MB, maximum size 2048MB
k. Click "Set"
l. Click OK
m. Click OK
n. Start closing Open Windows and it will ask to restart.
6. RESTART Virtual Machine
7. Change Power Options
a. Open the Start Menu, in the search line, type Power Options and press Enter
b. Select the "High performance" power plan
c. Click "change plan settings"
d. Click "change advanced power settings"
e. Expand the Hard disk option and change setting to "never"
f. Expand the Sleep option and change setting to "never"
g. Click OK
8. Uninstall Tablet PC, Windows Gadgets, and Media Center
a. Open Control Panel > Programs and Features > Turn Windows features on or off.
b. Uncheck Media Center, Tablet PC, and Windows Gadgets.
9. Disable Windows Themes
a. Control Panel > Personalization > Select Windows Classic.
10. Set Run for Best Performance
a. Right click on Computer and select Properties
b. Select Advanced system settings
c. Under the Advanced Tab, click Settings next to Performance
d. Select "Adjust for best performance"
11. Disable System Maintenance
a. Control Panel > Troubleshooting > Change Settings and select OFF under Computer Maintenance.
12. Disable Screensaver 1.Control Panel > Personalization > Screen Saver and change to NONE.
13. Disable Windows Sounds 1.Control Panel > Personalization > Sounds > and change Sound Scheme to No Sounds.
14. Speed up the Menu Show delay time
a. Open Regedit
b. Go to: HKEY_CURRENT_USER\Control Panel\Desktop
c. Modify MenuShowDelay from 400 to 1
15. Disable Un-needed Services
a. Telephony
b. Volume Shadow Copy Service – Not needed since System Restore is disabled.
16. Startup and Recovery
a. Open the Start Menu
b. Right click Computer
c. Click Properties
d. Click Advanced System Settings (it’s on the left side)
e. Under Startup and Recovery click Settings
f. Set the “Time to display list of operating systems” to 1
g. Disable “Write an event to the system log”
h. Click OK
i. Click OK
17. Boot GUI
a. Run msconfig
b. Under Boot tab Enable “No GUI boot” and “Base Video”
18. Restart
2. Run Windows updates
3. After updates run Command.bat as Administrator found in the VMView Windows 7 Optimization http://www.vmware.com/resources/techresources/10157. I have also included it as an attachment.
4. Turn off Drive Indexing
a. Open the Start Menu
b. Click Computer
c. Right click C: drive and select Properties and uncheck drive indexing
d. Click apply changes to drive C:\, subfolders and files (ignore all when comes up)
5. Shrink Page File
a. Open the Start Menu
b. Right click Computer
c. Click Properties
d. Click Advanced System Settings (it’s on the left side)
e. Under performance click Settings
f. Go to the Advanced tab
g. Under Virtual memory click Change
h. Uncheck the "Automatically manage paging file" box at the top
i. Select the C: drive
j. Click "Custom size"; initial size 512MB, maximum size 2048MB
k. Click "Set"
l. Click OK
m. Click OK
n. Start closing Open Windows and it will ask to restart.
6. RESTART Virtual Machine
7. Change Power Options
a. Open the Start Menu, in the search line, type Power Options and press Enter
b. Select the "High performance" power plan
c. Click "change plan settings"
d. Click "change advanced power settings"
e. Expand the Hard disk option and change setting to "never"
f. Expand the Sleep option and change setting to "never"
g. Click OK
8. Uninstall Tablet PC, Windows Gadgets, and Media Center
a. Open Control Panel > Programs and Features > Turn Windows features on or off.
b. Uncheck Media Center, Tablet PC, and Windows Gadgets.
9. Disable Windows Themes
a. Control Panel > Personalization > Select Windows Classic.
10. Set Run for Best Performance
a. Right click on Computer and select Properties
b. Select Advanced system settings
c. Under the Advanced Tab, click Settings next to Performance
d. Select "Adjust for best performance"
11. Disable System Maintenance
a. Control Panel > Troubleshooting > Change Settings and select OFF under Computer Maintenance.
12. Disable Screensaver 1.Control Panel > Personalization > Screen Saver and change to NONE.
13. Disable Windows Sounds 1.Control Panel > Personalization > Sounds > and change Sound Scheme to No Sounds.
14. Speed up the Menu Show delay time
a. Open Regedit
b. Go to: HKEY_CURRENT_USER\Control Panel\Desktop
c. Modify MenuShowDelay from 400 to 1
15. Disable Un-needed Services
a. Telephony
b. Volume Shadow Copy Service – Not needed since System Restore is disabled.
16. Startup and Recovery
a. Open the Start Menu
b. Right click Computer
c. Click Properties
d. Click Advanced System Settings (it’s on the left side)
e. Under Startup and Recovery click Settings
f. Set the “Time to display list of operating systems” to 1
g. Disable “Write an event to the system log”
h. Click OK
i. Click OK
17. Boot GUI
a. Run msconfig
b. Under Boot tab Enable “No GUI boot” and “Base Video”
18. Restart
GPO Changes
19. Disable success logins
a. Computer Configuration > Policies > Windows Settings > Security Settings > Local Policies > Audit Policy
b. Under Audit account logon events settings Properties select the Security Policy Setting tab.
c. Select the Failure checkbox.
20. Set maximum Event Log size
a. Computer Configuration > Administrative Templates > Event Log Service > Specific Event Log
b. Maximum application, security, and system log size = 1024
21. Empty IE temp files
a. User Configuration > Administrative Templates > Windows Components > Internet Explorer > Internet Control Panel > Advanced Page
b. Empty Temporary Internet Files folder when browser is closed = Enabled
22. Disabled IE First Run Customize Wizard
a. User Configuration > Administrative Templates > Windows Components > Internet Explorer
b. Prevent performance of First Run Customize settings – Enabled
23. Disable RSS Feeds
a. User Configuration > Administrative Templates > Windows Components > RSS Feeds
b. Turn off background sync feeds and web slices = Enabled
24. Disable Windows Slideshow
a. User Configuration > Administrative Templates > Windows Components >Windows Slideshow
b. Turn off Windows Slideshow = Enabled
25. Turn off Build-to-Lossless feature
a. Computer Configuration > Administrative Templates > Classic Administrative Templates (ADM) > PCoIP Session Variables
b. Turn off Build-to-Lossless feature = Enabled
26. Limit audio bandwidth
a. Computer Configuration > Administrative Templates > Classic Administrative Templates (ADM) > PCoIP Session Variables
b. Change Configure the PCoIP session audio bandwidth limit to 100
27. Set PCoIP frame rate and image quality
a. Computer Configuration > Administrative Templates > Classic Administrative Templates (ADM) > PCoIP Session Variables
b. Maximum Initial image quality = 70
c. Minimum Image Quality = Leave default
d. Maximum Frame Rate = 15
a. Computer Configuration > Policies > Windows Settings > Security Settings > Local Policies > Audit Policy
b. Under Audit account logon events settings Properties select the Security Policy Setting tab.
c. Select the Failure checkbox.
20. Set maximum Event Log size
a. Computer Configuration > Administrative Templates > Event Log Service > Specific Event Log
b. Maximum application, security, and system log size = 1024
21. Empty IE temp files
a. User Configuration > Administrative Templates > Windows Components > Internet Explorer > Internet Control Panel > Advanced Page
b. Empty Temporary Internet Files folder when browser is closed = Enabled
22. Disabled IE First Run Customize Wizard
a. User Configuration > Administrative Templates > Windows Components > Internet Explorer
b. Prevent performance of First Run Customize settings – Enabled
23. Disable RSS Feeds
a. User Configuration > Administrative Templates > Windows Components > RSS Feeds
b. Turn off background sync feeds and web slices = Enabled
24. Disable Windows Slideshow
a. User Configuration > Administrative Templates > Windows Components >Windows Slideshow
b. Turn off Windows Slideshow = Enabled
25. Turn off Build-to-Lossless feature
a. Computer Configuration > Administrative Templates > Classic Administrative Templates (ADM) > PCoIP Session Variables
b. Turn off Build-to-Lossless feature = Enabled
26. Limit audio bandwidth
a. Computer Configuration > Administrative Templates > Classic Administrative Templates (ADM) > PCoIP Session Variables
b. Change Configure the PCoIP session audio bandwidth limit to 100
27. Set PCoIP frame rate and image quality
a. Computer Configuration > Administrative Templates > Classic Administrative Templates (ADM) > PCoIP Session Variables
b. Maximum Initial image quality = 70
c. Minimum Image Quality = Leave default
d. Maximum Frame Rate = 15
Friday, November 13, 2015
PowerCLI script to collect "Computers that have not logged on since specified date"
I needed to collect and Inventory all VM's that have not logged on since a specified date. In my case I was looking for systems that hadn't logged on in the past 90 days. Substitute the "$DaysInactive" with what suits your case.
# Gets time stamps for all computers in the domain that have NOT logged in since after specified date
$domain = "YouDomain.Name"
$DaysInactive = 90
$time = (Get-Date).Adddays(-($DaysInactive))
# Get all AD computers with lastLogonTimestamp less than our time
Get-ADComputer -Filter {LastLogonTimeStamp -lt $time} -Properties LastLogonTimeStamp |
# Output hostname and lastLogonTimestamp into CSV
select-object Name,@{Name="Stamp"; Expression={[DateTime]::FromFileTime($_.lastLogonTimestamp)}} | export-csv -path c:\tmp\vmware\LastLogon_90_Days.csv -notypeinformation
# Gets time stamps for all computers in the domain that have NOT logged in since after specified date
$domain = "YouDomain.Name"
$DaysInactive = 90
$time = (Get-Date).Adddays(-($DaysInactive))
# Get all AD computers with lastLogonTimestamp less than our time
Get-ADComputer -Filter {LastLogonTimeStamp -lt $time} -Properties LastLogonTimeStamp |
# Output hostname and lastLogonTimestamp into CSV
select-object Name,@{Name="Stamp"; Expression={[DateTime]::FromFileTime($_.lastLogonTimestamp)}} | export-csv -path c:\tmp\vmware\LastLogon_90_Days.csv -notypeinformation
Tuesday, October 20, 2015
Understanding VMware CPU over commitment
Thanks to Manish Kumar Jha aka Alex Hunt for providing this insight to over commitment.
Over commitment in its simplest term means allocating more resources to virtual workloads then what is available at physical level. Most common resources that are over committed are memory and cpu.
A simple example of over commitment can be running 3 VM each with 4 GB RAM on an Esxi host which has only 8 GB RAM. In this case we have allocated 12 GB RAM to all VM’s collectively but at physical level (Esxi host) we have only 8 GB RAM available.
It is a general belief by most novice VMware admins that allocating more resources to virtual machines means better performance. When I started working with VMware I also used to think in the same way.
It was the vSphere Optimize and Scale training, where I learned this is not true and how over commitment can badly affect VM performance.
In this post I am trying to demonstrate the negative affects of CPU over commitment.
Before diving into demonstration lets recall a few terms which will help understanding concepts more clearly.
The host on which these VM are running has 6 GB of RAM and has 8 CPU X 2.4 GHz.
CPU allocation for the VM’s are as follows:
Server 2003-1: 1 vCPU
Server 2003-2: 1 vCPU
RHEL Server: 1 vCPU
vShield Manager: 2 vCPU
Lets begin with the demonstration:
1: Power on all VM’s
2: Run Esxtop
In general:
4: Change the number of cores per socket to 8 for both VM
Select 1 sockets and 8 cores per socket to make total vCPU=8
5: Power on the 2 VM and look at the esxtop stats again
Gees, look at the %RDY and %CSTP time.
For Linux VM there is one script available at GitHub
Note: Before running the cpuload.sh script make sure the Linux VM has ‘stress‘ and ‘cpulimit‘ rpm’s installed
Copy the code from GitHub and save it as cpuload.sh and run following commands:
# chmod +x cpuload.sh
# ./cpuload.sh [cpu load in percent] [duration in seconds]
For e.g to force cpu utilization to 75% for 50 seconds run the command as shown below:
# ./cpuload.sh 75 50
On running cpuload.sh on linux VM, if you fire top command you will see the cpu stats as below:
If we look at the esxtop stats, we will see something like as shown below:
Holy crap. Look at the jump in %RDY and %CSTP values
For Windows OS there is a nice utility called Load Storm. This utility can be downloaded from Here
Note: Load Storm requires dotnet framework 3.5 to be pre-installed.
To generate load on cpu adjust the Number of threads value and Approx core load per thread. Also you can choose the checkbox Generate load for if you wish to keep cpu busy for more than default one minute.
On generating 100% workload on cpu, the windows reported CPU utilization as 100%
Lets see the esxtop stats
As expected you can see the rise in value for %RDY and %CSTP for Windows VM
If we check the Esxi host CPU stats in GUI we can see CPU is almost maxed out
Also the performance chart will show a sudden rise in CPU utilization when we have run the cpu load script on both windows and linux vm together.
In my case GUI was reporting almost full utilization of physical cpu. But in production environment it is very common to see Esxi host utilization somewhere near 50-60% but still VM’s are reporting high %RDY and %CSTP values in Esxtop.
High %RDY and high %CSTP indicate an over-allocation of CPU resources – i.e. too many vCPU for the job at hand
Now I am going to power off the RHEL and Windows VM and set the vCPU to 2 and re-run the cpu load scripts to see if there is any difference in %RDY and %CSTP values.
I choosed to run 4 threads on 2 vCPU in windows VM.
After changing the cpu count, esxtop has following readings
You can see there is significant drop in %RDY and %CSTP values for both VM’s.
Conclusion
Assigning fewer vCPUs to a VM than it has threads is more efficient than assigning more vCPUs; but if we assign the exact number of vCPUs as threads, we risk increasing %RDY and %CSTP if we over commit the total number of vCPUs in the environment.
Over commitment in its simplest term means allocating more resources to virtual workloads then what is available at physical level. Most common resources that are over committed are memory and cpu.
A simple example of over commitment can be running 3 VM each with 4 GB RAM on an Esxi host which has only 8 GB RAM. In this case we have allocated 12 GB RAM to all VM’s collectively but at physical level (Esxi host) we have only 8 GB RAM available.
It is a general belief by most novice VMware admins that allocating more resources to virtual machines means better performance. When I started working with VMware I also used to think in the same way.
It was the vSphere Optimize and Scale training, where I learned this is not true and how over commitment can badly affect VM performance.
In this post I am trying to demonstrate the negative affects of CPU over commitment.
Before diving into demonstration lets recall a few terms which will help understanding concepts more clearly.
- pCPU- Total physical CPU X Number of cores. In this example I have a host with 2 CPU and 2 cores and thus total of 4 pCPU.
- vCPU- CPU’s allocated to a Virtual Machine.
- Esxtop- Performance analysis utility, usually run from Esxi Console or SSH.
- %RDY- CPU Ready Time is the amount of time a virtual machine is ready to use CPU but was unable to schedule time because all physical CPU are busy.
- %CSTP- The percentage of time that the virtual machine is ready to execute commands but that it is waiting for the availability of physical CPU’s as the virtual machine is configured to use multiple vCPUs.
- %WAIT- It is the percentage of time the virtual machine was waiting for some VMkernel activity to complete (such as I/O) before it can continue.
- %IDLE- The percentage of time a world is in idle loop.
The host on which these VM are running has 6 GB of RAM and has 8 CPU X 2.4 GHz.
CPU allocation for the VM’s are as follows:
Server 2003-1: 1 vCPU
Server 2003-2: 1 vCPU
RHEL Server: 1 vCPU
vShield Manager: 2 vCPU
Lets begin with the demonstration:
1: Power on all VM’s
2: Run Esxtop
In general:
- %RDY (CPU Ready) should be low
- %CSTP should be 0.00
- %IDLE should be high – indicating high percentages of resources waiting for something to do
4: Change the number of cores per socket to 8 for both VM
Select 1 sockets and 8 cores per socket to make total vCPU=8
5: Power on the 2 VM and look at the esxtop stats again
Gees, look at the %RDY and %CSTP time.
- High %RDY value indicates that vCPUs are waiting for actual physical CPUs to schedule their threads.
- High %CSTP indicates that ESXi is asking the vCPUs to wait – A.K.A. co-stopping them for scheduling purposes.
For Linux VM there is one script available at GitHub
Note: Before running the cpuload.sh script make sure the Linux VM has ‘stress‘ and ‘cpulimit‘ rpm’s installed
Copy the code from GitHub and save it as cpuload.sh and run following commands:
# chmod +x cpuload.sh
# ./cpuload.sh [cpu load in percent] [duration in seconds]
For e.g to force cpu utilization to 75% for 50 seconds run the command as shown below:
# ./cpuload.sh 75 50
On running cpuload.sh on linux VM, if you fire top command you will see the cpu stats as below:
If we look at the esxtop stats, we will see something like as shown below:
Holy crap. Look at the jump in %RDY and %CSTP values
For Windows OS there is a nice utility called Load Storm. This utility can be downloaded from Here
Note: Load Storm requires dotnet framework 3.5 to be pre-installed.
To generate load on cpu adjust the Number of threads value and Approx core load per thread. Also you can choose the checkbox Generate load for if you wish to keep cpu busy for more than default one minute.
On generating 100% workload on cpu, the windows reported CPU utilization as 100%
Lets see the esxtop stats
As expected you can see the rise in value for %RDY and %CSTP for Windows VM
If we check the Esxi host CPU stats in GUI we can see CPU is almost maxed out
Also the performance chart will show a sudden rise in CPU utilization when we have run the cpu load script on both windows and linux vm together.
In my case GUI was reporting almost full utilization of physical cpu. But in production environment it is very common to see Esxi host utilization somewhere near 50-60% but still VM’s are reporting high %RDY and %CSTP values in Esxtop.
High %RDY and high %CSTP indicate an over-allocation of CPU resources – i.e. too many vCPU for the job at hand
Now I am going to power off the RHEL and Windows VM and set the vCPU to 2 and re-run the cpu load scripts to see if there is any difference in %RDY and %CSTP values.
I choosed to run 4 threads on 2 vCPU in windows VM.
After changing the cpu count, esxtop has following readings
You can see there is significant drop in %RDY and %CSTP values for both VM’s.
Conclusion
Assigning fewer vCPUs to a VM than it has threads is more efficient than assigning more vCPUs; but if we assign the exact number of vCPUs as threads, we risk increasing %RDY and %CSTP if we over commit the total number of vCPUs in the environment.
Subscribe to:
Posts (Atom)