Updating ThinkAgile VX environments

to Best Recipe levels using LXCI and

vLCM

Version 23.1b
Contents
Introduction ................................................................................................................. 1
Before you begin ......................................................................................................... 2
Installing and configuring the LXCI Appliance ......................................................... 3
Creating and configuring vLCM cluster images ....................................................... 6
Validated Cluster Image Example Based On Major/Minor BR Release with a
Repository Pack .................................................................................................. 7
Validated Cluster Image Example Based on Minor BR Release without an
available Repository Pack .................................................................................. 9
Exporting vLCM cluster images............................................................................... 14
Remediating ESXi hosts using cluster images....................................................... 15
Troubleshooting Tips ............................................................................................... 15
"Compliance status is unknown" error is shown during the cluster compliance check. ..................................... 15
“Hosts are incompatible with the image” is being displayed in the Cluster > Updates View .............................. 16

Appendix A - Trademarks ......................................................................................... 18

Introduction
This document explains how to update ThinkAgile VX nodes running ESXi 7.x or 8.x in a
VMware vSAN cluster to the driver and firmware levels associated with the ThinkAgile VX
Best Recipe.

Beginning with ESXi 7.x or 8.x, use LXCI and vLCM to update the ESXi version on the
server as well as the drivers and firmware by generating a single cluster image, which can
be applied to all the servers in the cluster.

Note: If you are updating from ESXi 6.7 or earlier, you must use the LXCI rolling updates
feature for firmware upgrades and the established VMware Update Manager process for
upgrading ESXi hosts. For more information about rolling updates, see the this page

This document may provide links to Lenovo or external sources. Should conflicting
information be observed between this guide and sources outside this document, assume
the outside sources are correct.

Make sure that you are familiar with the following terms:

VMware LifeCycle The successor to VMware Update Manager (VUM), it allows

Manager (vLCM) ESXi 7.x or 8.x hosts to be upgraded, patched, and have their
firmware levels updated from a single user interface. It
integrates with vCenter and will ensure workloads are migrated
off ESXi hosts prior to being upgraded. It is built into vCenter, so
no additional installation is required.
Lenovo xClarity It allows vLCM to perform firmware updates on Lenovo physical
Integrator (LXCI) servers. It hosts the ThinkAgile VX Repository Pack and the
ThinkAgile VX vendor Add-On. It also includes a list of
supported ESXi versions and their corresponding Repository
Pack and Lenovo Custom Image Add-Ons.
vLCM cluster image An enhanced version of a VMware Update Manager (VUM)
“baseline,” that enables the application of ESXi versions,
Lenovo Repository Packs, and customization Add-Ons on all
hosts in a VSAN cluster in a consistent manner.
Lenovo ThinkAgile An archive bundle containing all the firmware packages that are
VX Repository Pack supported with a ThinkAgile VX Best Recipe release (e.g.
December 2021 release – 21.4). These bundles can be used
with LXCA or LXCI. They are hosted online and can also be
imported manually if LXCI systems have no internet access.
Lenovo A bundle of vLCM components, released by Lenovo with a
Customization Add- specific Best Recipe.
On (Vendor Add-On)
vLCM components A vLCM construct analogous to “vibs” from previous ESXi
versions. These are individual update packages that are
published by Lenovo and VMware. They contain device drivers
and tools specific for a VX Best Recipe.

Page 1
 VX Best Recipe This is typically a quarterly release of VX Best Recipe that
Release contains changes to most system and device code along with
new ESXi release support, new machine types, new devices,
new Lenovo Software. They are labeled in the format YY.X (e.g.
22.3 – 3rd best recipe release for calendar year 2022). The
major releases always come with an associated VX Repository
Pack.
VX Best Recipe A Best Recipe based on the major release, that contains small
Minor Release changes to either system or device code, support for a new ESXi
patch version or Lenovo Software patch. They are labeled
YY.X[a-z] (e.g. 22.3a for code released after the release of 22.3,
before 23.1). Minor releases are cumulative, 22.3c would
include all fixes included in 22.3a and 22.3b.
Minor releases may not be associated with their own Repository
Pack when released, but the fixes will get eventually bundled up
in a Minor Repository Pack. See minor release example for
more details.

Before you begin

Before upgrading ESXi and firmware for a host, make sure you completed following steps:

• Update vCenter to the applicable version. vCenter versions are listed on the Best
Recipe page.

• Update any 3rd-party software that is used in the ThinkAgile VX environment (VDI,
Backup, Monitoring, and Security)

• Make sure that LXCI is installed and registered with vLCM. See "Installing and
configuring the LXCI appliance" for more information.

Upgrades should be scheduled in environments where at least an N+1 redundancy is

provided by the vSAN clusters in terms of CPU, RAM, network and storage resources and
policies. Also, updates should be scheduled outside of peak hours to avoid impact on
workload performance.

Page 2
Installing and configuring the LXCI Appliance
Complete the following steps to install and configure LXCI:

1. Install LXCI and make sure that vLCM is enabled. For detailed instructions on
installing LXCI, see the following topic:
https://sysmgt.lenovofiles.com/help/topic/com.lenovo.lxci_vcenter.doc/vcenter_usin
g_vsphere_web_client_work_rsu_function.html

2. Optionally, apply updates. Check the information on the Best Recipe page to
determine if an update is required. In some cases, Lenovo Support might provide a
custom page.

To apply an update or patch, see the following topic:

https://sysmgt.lenovofiles.com/help/topic/com.lenovo.lxci_vcenter.doc/update_man
agement_server_software.html

3. Make sure that all ThinkAgile VX systems in the vSAN cluster are managed by
LXCI. For each system in the cluster, use the following procedure:
https://sysmgt.lenovofiles.com/help/topic/com.lenovo.lxci_vcenter.doc/vcenter_disc
ov_manage_bmc_direct.html

4. Synchronize the vLCM inventory:

a. From the VMware main menu, click Lifecycle Manager.
b. Click Actions→Sync HCL.
c. Upon completion, click Actions→Sync Updates.

Page 3
5. Update the LXCI repository. From the menu, click Lenovo xClarity Integrator →
Manage Firmware Packages. Then complete one of the following steps:
• If LXCI has internet access, select the Repository Pack matching the version
in the Best Recipe. Click Download and wait for the download to complete.

• If LXCI does not have internet access or if you have been provided a private
Repository Pack by Lenovo Support, manually download the Repository Pack
that is appropriate for your cluster. Copy the repository pack to a shared folder
accessible to the LXCI appliance.

Then, click the Import button to configure the Remote Repository. Provide the
shared folder details along with access credentials. Select the Lenovo
ThinkAgile VX Repository Pack version for your cluster. Click Download and
wait for the download to complete.
If synchronized, vLCM should have the Lenovo vendor Add-On that matches the
Best Recipe information. If the Add-Ons listed in the Best Recipe are not available
in the list, manually download the vendor Add-On from the Best Recipe, and then
import the vendor Add-On into vLCM:

a. From the VCSA menu, click Lifecycle Manager →Actions→Import

Updates.
b. Browse for the .zip file that contains the Add-Ons and click Import.
c. Once the import is completed the package should be available under the
vendor Add-Ons page.
In rare situations you may need to manually import vLCM components, as they are
not present in the Lenovo vendor addon. To do this, locate the required package
on the corresponding Machine Type Best recipe page, for your server model.
Download the package (typically a zip bundle) and determine if the zip bundle has
following folder structure.

If this is not the folder structure, ensure that you have the correct package and look
in zip files within the archive for a file matching this folder structure. This is required
to ensure you will be importing a valid component file into vLCM.
Import the updates. From the VSCA menu, click LifeCycle
Manager→Actions→Import Updates. Choose the required zip bundle and click
Import.

Page 4
Note: Starting with LXCI 7.6, you can create copies of the Lenovo Repository Packs,
and replace/delete firmware packages. This is not required for most customers. Only
modify the firmware and drivers add-on when directed by Lenovo Support or when such
guidance is given on the Best Recipe.

Page 5
Creating and configuring vLCM cluster images

Lenovo provides customers regularly with a “Lenovo ESXi Custom Image”. This
image takes a base official ESXi Image version and layers on top the Lenovo specific
drivers and utilities that have been validated against a specific release of ESXi (e.g.
7.0U3) on Lenovo hardware.
While this image works for most customer environments, there may be special
circumstances where a validated cluster image needs to be built with a different set of
drivers or a different base ESXi version (e.g. patches like ESXi 7.0U3f). Furthermore, the
Lenovo Custom Image cannot be imported into vLCM, as the custom image construct is
not tied to vLCM, so to leverage vLCM to upgrade systems to a specific Best Recipe level,
you must configure a vLCM cluster image.
You can create a validated cluster image that includes ESXi, driver updates, firmware
updates, and optionally, vLCM components that are listed in the Best Recipe. Then, you
can apply this image to all nodes in the cluster.

Complete the following steps to create the cluster image:

1. From VSCA, click on a cluster. Then click Updates→Image→Manage clusters with

a single image
2. If your intent is to use the ESXi build version mentioned in the best recipe:
a. Download the “*ClusterImage.json” link in the Solution Stack Guidelines
section of the BR, for your corresponding release (e.g. 7.0U3)
b. Select your Cluster→Updates→Image→ “…”(top right side)
→Import→Provide the JSON file from above →Next
This will prepopulate the various pieces that make up the cluster image.
3. If you wish to select other ESXi version or have different requirements, select the
options listed in the Image Definition section for this Best Recipe. You will need to
select the following options:
a. ESXi version
b. Vendor Addon
c. Firmware and Drivers Addon
d. (Optional) vLCM Components as indicated in the best recipe
4. Click Save.
5. Click Finish Image Setup

Note: If your hardware configuration contains PCIe switches, please disable VMD from
UEFI. You will need to restart the system after changing VMD state. There is no impact to
VSAN configuration when changing this setting. VMD can be enabled once the firmware
upgrade is complete.

Page 6
Validated Cluster Image Example Based On Major/Minor
BR Release with a Repository Pack
Here’s an example of how starting from the VX Best Recipe you can create a validated
cluster image. For this example, we will use April 2022 best recipe release and 7.0U3 as
the ESXi release, trying to create an image identical to the Lenovo Custom Image in this
release. Going to the Solution Stack Guidelines page, these are the validated image
definitions:

This release provides following vLCM validated image definitions:

ESXi Version 7.0u3 - any vmware official release

Customization Addon for Lenovo ThinkSystem 7.0.3-
Vendor Addon
LVO.703.10.3
Firmware and Drivers
Lenovo ThinkAgile VX Repository Pack 22.1a
Addon
Components No additional components

A json file, matching the Lenovo custom image for this release is provided in the solution
stack page:

Lenovo 7.0U3 Cluster Image Definition 7.0U3_22.3_ClusterImage.json

You can either

• Import this into your cluster’s image via vCenter Inventory > Select Cluster >
Updates > Image > Import and provide the file from above:

Page 7
OR
• Manually setup the image by using vCenter Inventory > Select Cluster > Updates >
Image > Setup, and then select each of the components in the image:
ESXi Version:

Select vendor addon, 702.10.3:

Next select the firmware and driver addon, by selecting the LXCI instance that acts
as the HSM, and the 22.1a repository pack.

Page 8
 Lastly hit Save or validate the image as needed.
You now have a vLCM cluster image built to match the 2022 VX April Best Recipe.

Validated Cluster Image Example Based on Minor BR

Release without an available Repository Pack
Here’s an example of how starting from the VX Best Recipe minor version you can
create a validated cluster image. Use this example when you require certain fix_ids added,
as deviations from existing repository packs, as directed by Lenovo Support.
For this example, we will use a fictitious minor best recipe release, post 22.3
release (for this example, it would have a different 7Y94 XCC firmware) and 7.0U3 as the
ESXi release, trying to create an image identical to the Lenovo Custom Image in this
release. Going to the Solution Stack Guidelines page, these are the validated image
definitions:

This release provides following vLCM validated image definitions:

ESXi Version 7.0u3 - any vmware official release

Customization Addon for Lenovo ThinkSystem 7.0.3-
Vendor Addon
LVO.703.10.7
Firmware and Drivers
Lenovo ThinkAgile VX Repository Pack 22.3
Addon
Components No additional components

This example assume you have refreshed the list of available repository packs, and you
have imported 22.3 Repository Pack inside LXCI.

Next, review the VX Best Recipe Page for each machine type in scope for upgrade, for
fix_ids (upgrade packages) that are not bundled in the Repository Pack. Such fix_ids have
a relevant comment in the notes section. Download the required fix_ids files from the
download page. Store these on a shared folder, then repeat this step for all packages in
scope for the respective VX release and machine type. In this example we would use this
firmware
lnvgy_fw_xcc_cdi3a8n-9.40_anyos_noarch to add to the existing 22.3 Repo

In the LXCI GUI > Manage Firmware Packages > Import > Provide the shared folder
where all the individual fix_id packages are located > Complete the Import wizard

Page 9
In the LXCI GUI > Manage Firmware Packages > Select the 22.3 Repository Pack >
Copy > This will create a new Repository Pack called

A new Repo Pack Entry will be created with a name like:

Next, Select the custom Repo Pack > Edit

You will be presented with the list of MTs in the Custom Repo. Browse to the MT of
interest, in this example, we use 7Y94 and replace the existing XCC code with
“lnvgy_fw_xcc_cdi3a8n-9.40_anyos_noarch” fix id

Page 10
Once you click Replace you will be presented with an import form to provide a shared
folder where the replacement firmware can be picked up from:

For v3 systems and newer the import screen is simpler, you need to provide the zip
bundle as you would from the basic Windows “Open view”

Page 11
Once you hit finish, the Repository Pack highlights the changes from the original copied
repo pack.

Once you click Apply, the Repo Pack has been modified to include the new XCC code.

Next you can go ahead and

Page 12
Now you can go ahead and remediate host with this modified Repo Pack, as directed by
either the Best Recipe Notes or Lenovo Support.

Page 13
Exporting vLCM cluster images
Cluster images can be exported for use in other vCenter systems, or clusters. There are 2
formats available, JSON (just a manifest of what the cluster image should contain), and an
ISO file:

JSON
Download the image as a JSON file that can be imported into other clusters
managed by images. Note that this only contains metadata about the image, not the actual
software packages.

ISO
Download an installable ISO from the image to reuse this in other clusters
managed using Baselines, or to image new hosts.

ZIP (offline bundle)

Download a ZIP offline bundle that contains all components (software packages)
included in this image that can be imported into Lifecycle Manager's depot.

Choose the option suitable for your needs and wait for the export to complete.

Page 14
Remediating ESXi hosts using cluster images
For each cluster in scope of upgrade do following:
1. Run a cluster compliance check to get an overview of which systems require
updates

2. Run a remediation pre-check to reveal any potential issues with the

update/upgrade process

3. Run Remediation on the servers, selecting specific parameters according to the

availability and failures tolerated by the environment

4. Once the remediation is complete, run through the VSAN Health Check for the
cluster, and ensure any leftover configuration, like VSAN format upgrades is
performed
Tip:
After the firmware and drivers for a server are updated, the server will:
1. Reboot to the System Setup menu to make sure that the server powers on
2. Reboot to the System Setup menu again to activate all firmware
3. Reboot again to ESXi
This process is normal, and does not require any user interaction

Troubleshooting Tips

"Compliance status is unknown" error is shown during the cluster

compliance check.
Make sure that the following conditions are met:
• The vendor addon is selected
• The hosts are managed in LXCI before attempting to create the image
• The Repository Pack and Add-on are compatible

Page 15
 “Hosts are incompatible with the image” is being displayed in the
Cluster > Updates View
For a VX validated cluster image, it can happen that the patch version for ESXi
installed on the hosts introduced a newer VMware Tools Package than your target ESXi
version level.
In this case please ensure you select a high enough Cluster Image ESXi base
version so that VMware Tools package is equal or newer than your installed version.

In the example above, ESX 8.0GA was selected. Reviewing the vmware tools
versions at this link, shows that ESXi 8.0 GA shipped with a lower VMware Tools version
than ESXi 7.0U3i, which is also included inside 7.0U3k

https://packages.vmware.com/tools/versions

To correct this situation, we would need to select version 8.0 that is at least higher than 8.0P01
build 21203435. Looking up build versions on this article returns:
ESXi 8.0b ESXi 8.0b 2023/02/14 21203435 NA ISO
So to go from ESXi 7.0U3k to ESXi 8.0, you must use base image of at least ESXi 8.0b. Below
is the compliance view for this version:

Page 16
Page 17
Appendix A - Trademarks
• LENOVO and THINKAGILE are trademarks of Lenovo.
• All other trademarks are the property of their respective owners. © 2022 Lenovo

Page 18