Clearpath Insights: Managed Services | Cloud Services | IT Solutions

VMware NSX-T 2.5 to 3.0 Upgrade Error Fix

vmware-partner-logo-3

(Blog article co-authored by Ryan West and Felix Vargas)

During VMworld 2020 there was a substantial focus on the Networking and Security part of the portfolio. NSX took that lead in that category and NSX-T 3.1 is the latest release that comes with some much-anticipated features including Federation and Distributed IPS to name a few.

At Clearpath we do everything in our power to help mitigate risk for our customers. Part of that approach involves testing software when it becomes generally available (GA) to go through the deployment and upgrade process. We do this to give our customers peace of mind that a deployment or upgrade will not have a negative impact on their environment. To that end, we performed an NSX-T 2.5 to 3.1 migration in our lab environment and encountered some issues with we've detailed below. 

Lab Environment Details:

Product Pre-Upgrade Post-Upgrade
vCenter 6.7U3 7.0.1
ESXi 6.7U3 7.0.1
NSX-T 2.5 3.1

NSX-T Upgrade High Level Steps:

  1. Upgraded vCenter from 6.7 U3 to 7.0.1 Build: 17168206 

  2. Upgraded ESXi from 6.7U3 to 7.0.1 Build: 17005016 

  3. Uploaded upgrade bundle file to NSX Managers which also upgrades Upgrade Coordinator

  4. Begin upgrade process via Upgrade Coordinator

    1. Upgrade Coordinator does the below:

      1. Edges

      2. Hosts

      3. NSX Manager Cluster

Note: Upgrading vCenter and ESXi to 7.0.1 was not required since 6.7u3 fully supports NSX-T 3.1 but we planned on a migration from the N-VDS to VDS which does require a ESXi/vCenter upgrade to 7.0.1 as NSX-T 2.5 is not supported when running vCenter/ESXi 7.0.1.

The Problem:

After the hosts were upgraded, they never fully connected back with the managers and the NSX bits had references to 6.7-3.1.0. Applying the profile that was originally created in 2.5.1 would not upgrade the hosts and trying to list the Transport Node Profiles with a general error.

The Solution:

The solution involved working with the NSX-T API through Postman (a GUI based API client for REST calls – www.postman.com) to list out all the Transport Node Profiles and delete all but one that was working. The following is the step by step analysis and resolution of the upgrade issue:

The Process:

From NSX Manager, I could not list or edit any Transport Node Profile’s, I could only create new ones.

Transport Node Profiles list failed: General error

After a few failed attempts with CLI and switching manager nodes, I decided to go to the API. A Post GET query returned all 7 profiles.

Postmant GET https://nsx-mgmt/api/v1/transport-node-profiles query

With the “id” and “display_name” next to each other, it was fairly straightforward to issue Postman DELETE commands to remove all but the working Transport Node Profile.  Here is an example, the value of 1 returned means the command was successful.

Postmant DELETE https://nsx-mgmt/api/v1/transport-node-profiles/ID

Now NSX-T GUI showed the proper results, it was time to create the NSX-T backed VDS switch and manage the adapters and MTU in vCenter.

Transport Node Profile listing with N-VDS and VDS conversion profiles

Here is the finished product:

NSX_upgrade-final

I hope this helps anyone in the lab or field if you get stuck after upgrading your ESXi hosts to 7.0U1.  Feel free to reach to me with any questions about the process. 

Clearpath's technical account managers and their supporting pre-sales engineers can help you make the most out of your VMware investment. We’ve helped hundreds of customers optimize their IT systems and save money by utilizing solutions like NSX-T discussed above in the blog. Reach out to our team today to see how we can help your organization.

Contact Clearpath's VMware Licensing and Professional Services Team

Documentation Used to Perform Upgrade:

API Documentation: https://code.vmware.com/apis/1083/nsx-t

NSX-T Upgrade: https://docs.vmware.com/en/VMware-NSX-T-Data-Center/3.1/nsxt_31_upgrade.pdf

Interoperability Matrix: https://www.vmware.com/resources/compatibility/sim/interop_matrix.php#interop&39=&175=

cta-engineers-customize

Our engineers customize
IT solutions for your business

The best hardware, software and cloud computing solutions are nothing without expert design and implementation.

Our team becomes your team. We take the time to understand your business needs first, and tailor high performance systems specifically for you.

Learn more
Live Chat Support Software