Upgrade ISE Cluster - 2.7 to 3.0 Overview & Tips

"The What?" - In this blog I want to cover a brief overview of migrating an ISE cluster from 2.7 to 3.0. Note that for this overview the cluster consisted of a 4 node ISE VM cluster from 2.7p5 to 3.0p5 & migrating all licenses from the old licensing model (base/plus/apex) to the new model (essentials/advantage/premier). This overview shared was an inline upgrade & not a backup/restore upgrade on new nodes.


"The Why?" - I want to share this with others in hope that it can help with their ISE upgrade journeys.


"The How?" - I intend on covering several items which includes:

  • Brief overview of preparation/planning

  • Bundle upgrade workflow + tips & tricks

  • Patch upgrade workflow

  • Post bundle & patch upgrade verification

Ok so first thing first, preparation/planning phase. I always urge customers to open tickets with TAC just in case you need to engage them during the upgrade for issues you hit & possibly cant figure out yourself. This saves you time especially when the unexpected may occur. Plus it helps TAC potentially identify bugs, etc. Honestly it's a win/win for both sides.


In a 2.x to 3.x migration you will 100% need TAC to aide with migrating your licenses so in a scenario like the one I am about to share you should open two separate cases. 1 for licensing migration & the other for the upgrade.


The next preparation step is to download the URT (upgrade readiness tool), and get it installed on your secondary PAN. Yes, you only need to run it on the secondary PAN & not all nodes. If you have a standalone node, then you run it on the standalone. Also, URT causes no service interruption. The URT goal is to detect any data upgrade issues prior to actually attempting the bundle upgrade. This is accomplished via cloning the database to perform upgrade readiness checks.


You have to install the URT which looks like this:

# application install ise-urtbundle-3.0.0.458-1.0.0.SPA.x86_64.tar.gz <repo name>

Overview of tool output on a successful readiness check:

Save the current ADE-OS running configuration? (yes/no) [yes] ?
Generating configuration...
Saved the ADE-OS running configuration to startup successfully

Getting bundle to local machine...
Unbundling Application Package...
Verifying Application Signature...
Initiating Application Install...

###########################################
# Installing Upgrade Readiness Tool (URT) #
###########################################

Checking ISE version compatibility
- Successful

Checking ISE persona
- Successful

Along with Administration, other services (MNT) are enabled on this node. Installing and running URT might consume additional resources.
Do you want to proceed with installing and running URT now (y/n):y

Checking if URT is recent(<45 days old)
- Note: URT is 456 days old and its version is 1.0.0. There might be a recent URT bundle on CCO, please verify on CCO
Do you want to proceed with this version which is 456 days old (y/n):y
Installing URT bundle
 - Successful
 
########################################
 # Running Upgrade Readiness Tool (URT) #
 ########################################
 This tool will perform following tasks:
 1. Clone config database
 2. Copy upgrade files
 3. Data upgrade on cloned database
 4. Time estimate for upgrade
 
Clone config database
 =====================
  [########################################] 100%  Successful
 
Copy upgrade files
 ==================
 - N/A
 
Data upgrade on cloned database
 ===============================
 Modifying upgrade scripts to run on cloned database
 - Successful
 
Running schema upgrade on cloned database
 - Running db sanity to check and fix if any index corruption
 - Auto Upgrading Schema for UPS Model
 - Upgrading Schema completed for UPS Model
 - Successful
 
Running sanity after schema upgrade on cloned database
 - Successful
 
Running data upgrade on cloned database
 - Data upgrade step 1/32, NSFUpgradeService(2.8.0.127)... Done in 11 seconds.
 - Data upgrade step 2/32, NetworkAccessUpgrade(3.0.0.10)... Done in 0 seconds.
 - Data upgrade step 3/32, UPSUpgradeHandler(3.0.0.10)... Done in 2 seconds.
 - Data upgrade step 4/32, GMTConfigRegistration(3.0.0.221)... Done in 0 seconds.
 - Data upgrade step 5/32, UPSUpgradeHandler(3.0.0.227)... Done in 0 seconds.
 - Data upgrade step 6/32, ERSDictionaryRegistration(3.0.0.240)... Done in 0 seconds.
 - Data upgrade step 7/32, CertDictAttribAddition(3.0.0.240)... Done in 0 seconds.
 - Data upgrade step 8/32, UPSUpgradeHandler(3.0.0.250)... Done in 11 seconds.
 - Data upgrade step 9/32, EPScriptOSConfigRegistration(3.0.0.307)... Done in 0 seconds.
 - Data upgrade step 10/32, AuthzUpgradeService(3.0.0.320)... Done in 0 seconds.
 - Data upgrade step 11/32, RegisterPostureTypes(3.0.0.325)... Done in 0 seconds.
 - Data upgrade step 12/32, KongCertRegistration(3.0.0.329)... Done in 1 seconds.
 - Data upgrade step 13/32, ESAgentlessPostureScriptRegistration(3.0.0.352)... Done in 0 seconds.
 - Data upgrade step 14/32, GuestAccessUpgradeService(3.0.0.355)... Done in 7 seconds.
 - Data upgrade step 15/32, UpnDictionaryCreation(3.0.0.368)... Done in 0 seconds.
 - Data upgrade step 16/32, UpnProfileCreation(3.0.0.368)... Done in 0 seconds.
 - Data upgrade step 17/32, SessionServiceAgentlessRegistration(3.0.0.375)... Done in 0 seconds.
 - Data upgrade step 18/32, PostureSettingsAgentlessRegistration(3.0.0.382)... Done in 0 seconds.
 - Data upgrade step 19/32, SxpConnectionUpgrade(3.0.0.382)... Done in 0 seconds.
 - Data upgrade step 20/32, RestIDStoreSettingsRegistration(3.0.0.385)... Done in 0 seconds.
 - Data upgrade step 21/32, AnyNadProfIdRegistration(3.0.0.388)... Done in 0 seconds.
 - Data upgrade step 22/32, AuthProfileUpgradeService(3.0.0.389)... Done in 0 seconds.
 - Data upgrade step 23/32, AccessSecretEncryptionUpgrade(3.0.0.436)... Done in 0 seconds.
 - Data upgrade step 24/32, ProvisioningRegistration(3.0.0.441)... Done in 4 seconds.
 - Data upgrade step 25/32, UPSUpgradeHandler(3.0.0.442)... Done in 3 seconds.
- Data upgrade step 26/32, RuleResultsSGTUpgradeService(3.0.0.450)... Done in 0 seconds.
 - Data upgrade step 27/32, SGTToTableMapper(3.0.0.450)... Done in 0 seconds.
 - Data upgrade step 28/32, NSFUpgradeService(3.0.0.458)... Done in 0 seconds.
 - Data upgrade step 29/32, ProfilerUpgradeService(3.0.0.458)... Done in 0 seconds.
 - Data upgrade step 30/32, GuestAccessUpgradeService(3.0.0.458)... Done in 5 seconds.
 - Data upgrade step 31/32, UPSUpgradeHandler(3.0.0.458)... Done in 0 seconds.
 - Data upgrade step 32/32, ESUpgradeService(3.0.0.458)... Done in 0 seconds.
 - Successful
 
Running data upgrade for node specific data on cloned database
 - Successful
 
Time estimate for upgrade
 =========================
 (Estimates are calculated based on size of config and mnt data only. Network latency between PAN and other nodes is not considered in calculating estimates)
 Estimated time for each node (in mins):
 MNT data is 17 GB, purging this data can reduce upgrade time
 NODE-X(PRIMARY PAP,MNT,PXG):148
 MNT data is 17 GB, purging this data can reduce upgrade time
 NODE-XX(SECONDARY PAP,MNT,PXG):151
 MNT data is 17 GB, purging this data can reduce upgrade time
 Each PSN(2 if in parallel):58
 

 Final cleanup before exiting...
 
Application successfully installed

An important note: If you need to re-run the URT tool more than once you have to uninstall the application & then reinstall. To verify install of urt tool:

#show application    --you should see urt

Then to uninstall URT:

#application remove urt

If you encounter errors you will see where the URT fails in the process, which at this point you can perform some troubleshooting or rely on the TAC case you already opened for the upgrade prep :)


Once you have your cases opened, confirmed that the cluster is ready via URT, & schedule the upgrade the next step is to perform at a minimum a configuration backup in case of an emergency. See Administration->System->Backup&Restore. You should also at a minimum run the health check from PAN GUI: System->Administration->Health Check as this will identify other possible issues such as expired certs, licensing issues, etc. Lastly, it is advised to definitely take a look at the Upgrade checklist to perform those tasks prior to upgrade (this will be mentioned later).


Some other helpful tasks that may help save upgrade time:

  • Upgrade to latest patch on current version

  • Purge logs (Administration->System->Maintenance: Operation Data Purge

At this point in the process you should have an idea of when you will be ready to make some moves on actually performing the bundle upgrade. So here is where the decision needs to be made on how you want to do the upgrade. What I mean by this is whether or not you want to do a split upgrade, which helps with keeping services up, or performing the upgrade with downtime (this saves upgrade time, but may be a no-no for you since services will go down). Also, whether or not you intend on using the GUI or CLI. Truth be told, I have done both several times. Personally in my opinion it is a preference thing, even though I think most would prefer CLI simply because of the control you have. However, in this blog I am sharing the experience with the GUI perspective.


Now I will cover an overview of using the GUI to perform the bundle upgrade:


Note these following steps assume that you have your bundle upgrade + patch placed in the repo that the nodes can see;

Navigating to Administration->System->Upgrade will present you with a checklist that contains additional checks you should perform:

Once you feel comfortable & are ready proceed. Next you will download the bundle to the nodes:

Then you will determine the order of operations. Always do the secondary PAN first. Once the sequence is determined (if doing split) then you can proceed with the upgrade. Once you kick the upgrade off you have a couple of ways to watch:

From CLI:

#show ver --this will depict something like this '%NOTICE: Identity Services Engine upgrade is in progress. . .'
#show logging application upgrade-uibackend-cliconsole.log --this is a more detailed step by step log to follow

Last little tip is that if you are concerned about potential PSN issues you can always increase the authz reauth timers in profiles so that the window of reauth is bigger which will present you with a longer window to get through the upgrade & potential issues.


The last leg of the cluster upgrade is the licensing migration, which to be honest is fairly straightforward and does not require much besides TAC doing their thing.


By no means is this the exact method you must follow. Remember that things may differ from environment to environment. This blog hopefully sheds some light on the process to take when attempting to move ISE from 2.x to 3.0.x.


Notable 3.0 bug I have seen during an upgrade: ISE 3.0 GUI certificate authentication - unsupported certificate purpose


Tons of other bugs have been reported in 3.0. Best bet is to apply latest patch after bundle upgrade.


Valuable/Helpful Cisco docs:

Products - ISE Licensing Migration Guide - Cisco

Cisco ISE 3.0 Upgrade Guide: Upgrade Method - Cisco


Thanks for reading, Cheers!

0 comments

Recent Posts

See All

In this tidbit I want to cover some high level notes on general trustsec items as well as some good-to-knows. A brief overview of what trustsec is: TrustSec provides scalable access controls by uniqu

"The What?" - In this blog I want to cover a project with Ansible that I created to automate parts of a workflow relating to an SDA edge node (EN) deployment. Now to breakdown the workflow I will be