Automate UCS ESXi host firmware updates with PowerCLI/PowerTool 3

The situation:

While working in an environment that utilizes Cisco UCS blades as the underlying hardware for a large vSphere deployment, it became apparent that automation around upgrading the blade firmware is a necessity.  For those of you familiar with Cisco UCS, you know that firmware updates introduce new functionality, fix bugs (such as this nasty Java bug), as well as keep you in a supported state with Cisco’s technical support.

Cisco UCS firmware updates happen first at the management / FI layer, and trickle downwards after that.  It is critical that you keep your host firmware (the actual blades in this case) up-to-date with the upstream components.

Note:  It is *highly* recommended that you check the Cisco Hardware and Software Compatibility Matrix before upgrading any of the firmware within the UCS environment.  The VMware HCL should also be consulted to ensure proper support and compatibility.

The environment:

In this environment, it is standard practice to use service profile templates to provision the service profiles of the ESXi hosts.  These profiles are of the “initial-template” type — they are not “updating” templates.  This decision was made to reduce the risk of accidental changes that could take down all hosts at once.  Additionally, all service profiles are deployed with a “user-ack” policy that requires manual intervention before a service profile will reboot after a user-initiated change.  The “user-ack”‘ policy also reduces the risk of unintentional downtime.

We are utilizing Auto Deploy in this environment, however, it is irrelevant to this process.

The solution:

The firmware update process involves a few different steps:

  1. Set the desired firmware policy on the service profile template.  This will be inherited by the service profiles as we unbind / bind them back to the template.
  2. Evacuate the ESXi host of any running VMs (done using maintenance mode — assuming full DRS functionality).
  3. Power off the ESXi host.
  4. Set the desired power state for the service profile to “off”.
  5. Initiate the firmware upgrade on the host by unbinding the service profile from its existing template, then re-binding it back to the same template.
  6. Acknowledge the user action (this is due to the “user-ack” policy described above).
  7. Wait for the firmware upgrade process to complete.  This could take up to 45 minutes or more!
  8. Once the upgrade is complete, set the desired power state for the service profile to “on”, and power up the host.

To automate this process, I have created the following PowerCLI / PowerTool script.  I have adapted it from an example that Cisco has posted online here.

Note:  It is required that you have both the PowerCLI and Cisco PowerTool components installed.  Before running this script it is assumed that you are logged into both the vSphere environment that contains the hosts you want to update, as well as the UCS environment containing the service profiles / hosts to update.  You can download the Cisco PowerTool components here.

 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
# Script to update ESXi host UCS firmware in a rolling fashion inside of a vSphere cluster.
#
# Assumptions:
#    1. You have assigned the new firmware package to the service profile template defined in the variable below. 
#    2. The service profile template is an "initial" template, and not an "updating" template.
#    3. You have already connected to the appropriate vSphere server and UCS environment in PowerShell/PowerCLI.
#
# Author: Tim Patterson <tim@pc-professionals.net>
# Last Updated: 2014-02-03
# 
# Adapted from Cisco example found here: https://communities.cisco.com/docs/DOC-36050
 
[CmdletBinding()]
Param(
	[Parameter(Mandatory=$True, HelpMessage="ESXi Cluster to Update")]
	[string]$ESXiCluster,
 
	[Parameter(Mandatory=$True, HelpMessage="ESXi Host(s) in cluster to update. Specify * for all hosts.")]
	[string]$ESXiHost,
 
	[Parameter(Mandatory=$True, HelpMessage="UCS Service Profile Template Name")]
	[string]$SPTemplate
)
 
Write-Host "Starting process at $(date)"
Write-Host "Working on ESXi Cluster: $ESXiCluster"
Write-Host "Using service profile template: $SPTemplate"
 
try {
	Foreach ($VMHost in (Get-Cluster $ESXiCluster | Get-VMHost | Where { $_.Name -like "$ESXiHost" } )) {
		Write-Host "vC: Placing ESXi Host: $($VMHost.Name) into maintenance mode"
		$Maint = $VMHost | Set-VMHost -State Maintenance -Evacuate
 
		Write-Host "vC: Waiting for ESXi Host: $($VMHost.Name) to enter Maintenance Mode"
		do {
			Sleep 10
		} until ((Get-VMHost $VMHost).State -eq "Maintenance")
 
		Write-Host "vC: ESXi Host: $($VMHost.Name) now in Maintenance Mode, shutting down Host"
		$Shutdown = $VMHost.ExtensionData.ShutdownHost($true)
 
		Write-Host "UCS: Correlating ESXi Host: $($VMHost.Name) to running UCS Service Profile (SP)"
 
		$vmMacAddr = $VMhost.NetworkInfo.PhysicalNic | where { $_.name -ieq "vmnic0" }
 
		$sp2upgrade =  Get-UcsServiceProfile | Get-UcsVnic -Name eth0 |  where { $_.addr -ieq  $vmMacAddr.Mac } | Get-UcsParent 
 
		Write-Host "UCS: ESXi Host: $($VMhost.Name) is running on UCS SP: $($sp2upgrade.name)"
		Write-Host "UCS: Waiting for UCS SP: $($sp2upgrade.name) to gracefully power down"
	 	do {
			if ( (get-ucsmanagedobject -dn $sp2upgrade.PnDn).OperPower -eq "off")
			{
				break
			}
			Sleep 60
		} until ((get-ucsmanagedobject -dn $sp2upgrade.PnDn).OperPower -eq "off" )
		Write-Host "UCS: UCS SP: $($sp2upgrade.name) powered down"
 
		Write-Host "UCS: Setting desired power state for UCS SP: $($sp2upgrade.name) to down"
		$poweron = $sp2upgrade | Set-UcsServerPower -State "down" -Force | Out-Null
 
		# Unbind / Bind to SP template, as this will force FW update action:
		$sp2upgrade | Set-UcsServiceProfile -srctemplname '' -force
		$sp2upgrade | Set-UcsServiceProfile -srctemplname "$SPTemplate" -force
 
		Write-Host "UCS: Acknowledging any User Maintenance Actions for UCS SP: $($sp2upgrade.name)"
		if (($sp2upgrade | Get-UcsLsmaintAck| measure).Count -ge 1)
			{
				$ackuserack = $sp2upgrade | get-ucslsmaintack | Set-UcsLsmaintAck -AdminState "trigger-immediate" -Force
			}
 
		Write-Host "UCS: Waiting for UCS SP: $($sp2upgrade.name) to complete firmware update process..."
		do {
			Sleep 40
		} until ((Get-UcsManagedObject -Dn $sp2upgrade.Dn).AssocState -ieq "associated")
 
		Write-Host "UCS: Host Firmware Pack update process complete.  Setting desired power state for UCS SP: $($sp2upgrade.name) to 'up'"
		$poweron = $sp2upgrade | Set-UcsServerPower -State "up" -Force | Out-Null
 
		Write "vC: Waiting for ESXi: $($VMHost.Name) to connect to vCenter"
		do {
			Sleep 40
		} until (($VMHost = Get-VMHost $VMHost).ConnectionState -eq "Connected" )
	}
}
Catch 
{
	 Write-Host "Error occurred in script:"
	 Write-Host ${Error}
	 Write-Host "Finished process at $(date)"
         exit
}
Write-Host "Finished process at $(date)"