Re: Catalyst 9800-L-F Web GUI extremely slow after code upgrade

pannick · ‎02-18-2025

Hello,

I have a Catalyst 9800-L-F that I just upgraded to 17.12.4 code from 17.12.3. Since the upgrade the web GUI is slow and at times non-responsive.

TAC informed me that I needed to restart the ip http server & ip http secure-server instances. Even though I had performed two reloads I did this with no luck. TAC then informed me that I needed to install the following patches.

Install required SMUs and APSP as you can find it here

I was able to install and active the SMU patch but after uploading the APSP patch and attempting to add it from the web gui I received the following response:

INSTALL ADD OPERATION: --- Analyzing file C9800-L-universalk9_wlc.17.12.04.CSCwn86874.SPA.apsp.bin --- Package Type is APSP Package type APSP and Operation type SMU do not match. Aborting install add.

Also, the device is very slow performing a tftp or stftp file transfer. They will go very slow and eventually time out. I had to upload the APSP from the USB slot.

Here is some log info:

CWW1-S-CFE05-R1249#show install summary
[ Chassis 1/R0 2/R0 ] Installed Package(s) Information:
State (St): I - Inactive, U - Activated & Uncommitted,
C - Activated & Committed, D - Deactivated & Uncommitted
--------------------------------------------------------------------------------
Type St Filename/Version
--------------------------------------------------------------------------------
IMG C 17.12.04.0.4826
SMU C bootflash:C9800-L-universalk9_wlc.17.12.04.CSCwn26561.SPA.smu.bin

--------------------------------------------------------------------------------
Auto abort timer: inactive
--------------------------------------------------------------------------------

CWW1-S-CFE05-R1249#show install version summary
Location: Chassis 1/R0
Version: 17.12.04.0.4826
Version Extension: 1721753583
Version State: Activated & Committed
Filename: /bootflash/C9800-L-universalk9_wlc.17.12.04.SPA.bin
CW_IMAGE: C9800-L-universalk9_wlc.17.12.04.SPA.bin
Checksum: e9ff767665489516152109b5241d715eb9a58abb
Package List:
Package Type: SMU
Package Name: /bootflash/C9800-L-universalk9_wlc.17.12.04.CSCwn26561.SPA.smu.bin
Package State: Activated & Committed

Location: Chassis 2/R0
Version: 17.12.04.0.4826
Version Extension: 1721753583
Version State: Activated & Committed
Filename: /bootflash/C9800-L-universalk9_wlc.17.12.04.SPA.bin
CW_IMAGE: C9800-L-universalk9_wlc.17.12.04.SPA.bin
Checksum:
Package List:
Package Type: SMU
Package Name: /bootflash/C9800-L-universalk9_wlc.17.12.04.CSCwn26561.SPA.smu.bin
Package State: Activated & Committed

CWW1-S-CFE05-R1249#show install log
[0|install_op_boot]: START Sat Feb 15 17:36:31 Universal 2025
[0|install_op_boot(INFO, )]: Mount IMG INI state base image
[0|install_op_boot]: END SUCCESS Sat Feb 15 17:36:32 Universal 2025
[0|install_op_boot(INFO, )]: cleanup_trap remote_invocation 0 operation install_op_boot .. 0 .. 0
[1|install_remove]: START Sun Feb 16 14:45:36 Eastern 2025
[1|install_remove(INFO, )]: [1 2]: Performing Collect_cleanlist
PID TTY TIME CMD
SUCCESS: Collect_cleanlist finished
[1|install_remove(INFO, )]: [1 2]: Performing Display_cleanlist
SUCCESS: Display_cleanlist finished
[1|install_remove(INFO, ignore user prompt)]: install_api
[remote|install_remove]: START Sun Feb 16 14:49:14 Eastern 2025
[remote|install_remove]: END SUCCESS Sun Feb 16 14:49:15 Eastern 2025
[remote|install_remove(INFO, )]: cleanup_trap remote_invocation 1 operation install_remove .. 0 .. 0
[1|install_remove(INFO, )]: [1 2]: Performing Remove_inactive
PID TTY TIME CMD
SUCCESS: Remove_inactive finished
[1|install_remove(INFO, )]: [1 2]: Performing Delete_cleanlist_files
PID TTY TIME CMD
SUCCESS: Delete_cleanlist_files finished
[remote|install_remove]: START Sun Feb 16 14:49:21 Eastern 2025
[remote|install_remove]: END SUCCESS Sun Feb 16 14:49:22 Eastern 2025
[remote|install_remove(INFO, )]: cleanup_trap remote_invocation 1 operation install_remove .. 0 .. 0
[1|install_remove(INFO, )]: [1 2]: Performing Post_Remove_Cleanup
PID TTY TIME CMD
SUCCESS: Post_Remove_Cleanup finished
[1|install_remove(INFO, )]: Remote output from chassis 1/R0
[1|install_remove(INFO, )]: install_remove: START Sun Feb 16 14:49:22 Eastern 2025
SUCCESS: install_remove Sun Feb 16 14:49:23 Eastern 2025
[1|install_remove(INFO, )]: Remote output from chassis 2/R0
[1|install_remove(INFO, )]: install_remove: START Sun Feb 16 14:49:21 Eastern 2025
SUCCESS: install_remove Sun Feb 16 14:49:22 Eastern 2025
[1|install_remove(INFO, )]: [1 2]: Performing ap_img_predownload
PID TTY TIME CMD
SUCCESS: ap_img_predownload finished
[1|install_remove]: END SUCCESS Sun Feb 16 14:49:28 Eastern 2025
[1|install_remove(INFO, )]: cleanup_trap remote_invocation 0 operation install_remove .. 0 .. 0
[2|get_profile_details]: START Sun Feb 16 15:03:16 Eastern 2025
[3|install_remove]: START Mon Feb 17 12:02:26 Eastern 2025
[3|install_remove(INFO, )]: [1 2]: Performing Collect_cleanlist
PID TTY TIME CMD
SUCCESS: Collect_cleanlist finished
[3|install_remove(INFO, )]: [1 2]: Performing Display_cleanlist
SUCCESS: Display_cleanlist finished
[3|install_remove(INFO, )]: [1 2]: Performing ap_img_predownload
PID TTY TIME CMD
SUCCESS: ap_img_predownload finished
[3|install_remove]: END SUCCESS Mon Feb 17 12:02:38 Eastern 2025
[3|install_remove(INFO, )]: cleanup_trap remote_invocation 0 operation install_remove .. 0 .. 0
[4|install_remove]: START Mon Feb 17 15:32:28 Eastern 2025
[4|install_remove(INFO, )]: [1 2]: Performing Collect_cleanlist
PID TTY TIME CMD
SUCCESS: Collect_cleanlist finished
[4|install_remove(INFO, )]: [1 2]: Performing Display_cleanlist
SUCCESS: Display_cleanlist finished
[4|install_remove(INFO, )]: [1 2]: Performing ap_img_predownload
PID TTY TIME CMD
SUCCESS: ap_img_predownload finished
[4|install_remove]: END SUCCESS Mon Feb 17 15:32:40 Eastern 2025
[4|install_remove(INFO, )]: cleanup_trap remote_invocation 0 operation install_remove .. 0 .. 0
[5|get_pkg_info_details]: START Tue Feb 18 11:04:35 Eastern 2025
[6|install_add]: START Tue Feb 18 11:04:36 Eastern 2025
[6|install_add(INFO, )]: Set INSTALL_TYPE to SMU
[6|install_add(CONSOLE, )]: Adding SMU
[6|install_add(CONSOLE, )]: Checking whether new add is allowed ....
[6|install_add(INFO, )]: [1 2]: Performing verify_smu
PID TTY TIME CMD
SUCCESS: verify_smu finished
[remote|install_add]: START Tue Feb 18 11:04:45 Eastern 2025
[remote|install_add(INFO, )]: Install SubType : NONE
[remote|install_add]: END SUCCESS /bootflash/C9800-L-universalk9_wlc.17.12.04.CSCwn26561.SPA.smu.bin Tue Feb 18 11:04:48 Eastern 2025
[remote|install_add(INFO, )]: cleanup_trap remote_invocation 1 operation install_add .. 0 .. 0
[6|install_add(INFO, )]: [1 2]: Performing SMU_ADD
SUCCESS: SMU_ADD finished
[6|install_add(INFO, )]: Remote output from chassis 1/R0
[6|install_add(INFO, )]:
[6|install_add(INFO, )]: Remote output from chassis 2/R0
[6|install_add(INFO, )]:
[6|install_add]: END SUCCESS Tue Feb 18 11:04:49 Eastern 2025
[6|install_add(INFO, )]: cleanup_trap remote_invocation 0 operation install_add .. 0 .. 0
[7|get_pkg_info_details]: START Tue Feb 18 12:58:27 Eastern 2025
[8|get_pkg_info_details]: START Tue Feb 18 19:38:21 Eastern 2025
[9|display_install_log]: START Tue Feb 18 19:44:27 Eastern 2025
[10|display_install_log]: START Tue Feb 18 19:48:59 Eastern 2025
[remote|install_activate]: START Tue Feb 18 19:54:06 Eastern 2025
[remote|install_activate(INFO, )]: Acquiring transaction lock...
[remote|install_activate(INFO, )]: global_trans_lock: /bootflash/.installer/install_global_trans_lock
[remote|install_activate(INFO, )]: tmp_global_trans_lock: /tmp/tmp_install_global_trans_lock
[remote|install_activate(INFO, )]: tmp lock does not exist: /tmp/tmp_install_global_trans_lock
[remote|install_activate(INFO, )]: Invoking op_activate_smu - remote: 1, filename bootflash:C9800-L-universalk9_wlc.17.12.04.CSCwn26561.SPA.smu.bin lnx_fileset /bootflash/C9800-L-universalk9_wlc.17.12.04.CSCwn26561.SPA.smu.bin abort_timer 360
[11|Exe_pre_post]: START Tue Feb 18 19:54:13 Eastern 2025
[11|Exe_pre_post(INFO, )]: Executing pre script, operation=install_activate, smu_id(s)=/bootflash/C9800-L-universalk9_wlc.17.12.04.CSCwn26561.SPA.smu.bin
[remote|install_activate(INFO, )]: [1 2]: Performing Exe_pre_scripts
SUCCESS: Exe_pre_scripts finished
[remote|install_activate(INFO, )]: KGV using kgv_update for SMU /bootflash/C9800-L-universalk9_wlc.17.12.04.CSCwn26561.SPA.smu.bin
[remote|install_activate(INFO, )]: active_smus
[remote|install_activate(INFO, )]: 70,/bootflash/C9800-L-universalk9_wlc.17.12.04.CSCwn26561.SPA.smu.bin,70_C9800-L-universalk9_wlc.17.12.04.CSCwn26561.SPA.smu.bin
[remote|install_activate(INFO, )]: prev_active_smus
[remote|install_activate(INFO, )]:
[remote|install_activate(INFO, )]: deactivated: changed SMU
[remote|install_activate(INFO, )]: activated: changed SMU /bootflash/C9800-L-universalk9_wlc.17.12.04.CSCwn26561.SPA.smu.bin
[remote|install_activate(INFO, )]: JSON file with impacted processes name doesn't exist. Not a combination SMU
[remote|install_activate(INFO, )]: Activating crdu smu /bootflash/C9800-L-universalk9_wlc.17.12.04.CSCwn26561.SPA.smu.bin
[remote|install_activate(INFO, )]: process_one_crdu_smu, patch_mount_dir /tmp/patch/smus, crdu_mnt_dir /tmp/patch/CRDU_mnt, this smu /bootflash/C9800-L-universalk9_wlc.17.12.04.CSCwn26561.SPA.smu.bin
[remote|install_activate(INFO, )]: creating /tmp/patch/CRDU_mnt
[remote|install_activate(INFO, )]: l_crdu_patch_name is CSCwn26561
[remote|install_activate(INFO, )]: l_activate_crdu_list is ,CSCwn26561
[remote|install_activate(INFO, )]: SMU Card Arch x86_64
[remote|install_activate(INFO, )]: chassis_role not supported yet.
[remote|install_activate(INFO, )]: SMU Card Arch x86_64
[remote|install_activate(INFO, )]: controller_launch not supported yet.
[remote|install_activate(INFO, )]: SMU Card Arch x86_64
[remote|install_activate(INFO, )]: SMU Card Arch x86_64
[remote|install_activate(INFO, )]: macaddr not supported yet.
[remote|install_activate(INFO, )]: SMU Card Arch x86_64
[remote|install_activate(INFO, )]: ncd_ha_update not supported yet.
[remote|install_activate(INFO, )]: SMU Card Arch x86_64
[remote|install_activate(INFO, )]: rif_mgr not supported yet.
[remote|install_activate(INFO, )]: SMU Card Arch x86_64
[remote|install_activate(INFO, )]: SMU Card Arch x86_64
[remote|install_activate(INFO, )]: rp_slot not supported yet.
[remote|install_activate(INFO, )]: SMU Card Arch x86_64
[remote|install_activate(INFO, )]: stack_mgr not supported yet.
[remote|install_activate(INFO, )]: SMU Card Arch x86_64
[remote|install_activate(INFO, )]: state not supported yet.
[remote|install_activate(INFO, )]: SMU Card Arch x86_64
[remote|install_activate(INFO, )]: List of crdu SMUs to be activated is CSCwn26561
[remote|install_activate]: END SUCCESS /bootflash/C9800-L-universalk9_wlc.17.12.04.CSCwn26561.SPA.smu.bin Tue Feb 18 19:54:18 Eastern 2025
[remote|install_activate(INFO, )]: cleanup_trap remote_invocation 1 operation install_activate .. 0 .. 0
[12|get_pkg_info_details]: START Tue Feb 18 19:55:29 Eastern 2025
[13|display_install_log]: START Tue Feb 18 20:20:07 Eastern 2025
[remote|install_commit]: START Tue Feb 18 20:21:36 Eastern 2025
[remote|install_commit(INFO, )]: Local transaction lock does not exist.
[remote|install_commit(INFO, )]: Releasing transaction lock...
[remote|install_commit]: END SUCCESS /bootflash/C9800-L-universalk9_wlc.17.12.04.CSCwn26561.SPA.smu.bin Tue Feb 18 20:21:38 Eastern 2025

Scott Fella · ‎02-18-2025

Did you try to downgrade at all? This is something that you really need to work with TAC and they have figure that out. I have not experienced slowness in the GUI and I upgrade my 9800-L's in my lab a lot. Other thing would be just to upgrade to a higher code train.

-Scott
*** Please rate helpful posts ***

marce1000 · ‎02-19-2025

- These commands could be useful :
show platform software system all
show platform resources
show facility-alarm status
show processes cpu platform sorted
show processes cpu platform sorted | exclude 0%
show processes cpu sorted
show platform software process slot chassis active R0 monitor (==> list process using the most CPU on top!)
show platform software process slot chassis active R0 monitor | include Mem
show processes memory platform sorted
show processes memory platform accounting (look at max_diff_calls , if on subsequent execution of the command, this parameter keeps increasing for a particular process then the process may have a memory leak)

It's also always advisable to checkup the controller's configuration using the CLI command
show tech wireless and feed the output from that into : Wireless Config Analyzer

M.

-- Each morning when I wake up and look into the mirror I always say ' Why am I so brilliant ? '
When the mirror will then always repond to me with ' The only thing that exceeds your brilliance is your beauty! '

Leo Laohoo · ‎02-19-2025

Are there any Tracebacks generated (%CLI_AGENT-1-NVGEN_ERR: Error while processing NVGEN command) when using the CLI?

If there are, this could be CSCwi78109 and SMU for 17.12.4 has been available since September 2024.

marce1000 · ‎02-19-2025

- Added : the https://software.cisco.com/download/home/286321399/type/282046477/release/Dublin-17.12.4 web page
also includes : All 17.12 deployments should run either: 17.12.4ESW13, at this hidden URL, or 17.12.4 CCO plus APSP4 (or above), and the CSCwj93876 and CSCwi78109 SMUs
Go for 17.12.4ESW13

Also from https://software.cisco.com/download/home/286321399/type , look at :

IOS XE Hardware Programmable Devices
IOS XE ROMMON Software

Upgrade these ones in too ,if applicable and or not yet done,
take a look at https://www.cisco.com/c/en/us/td/docs/wireless/controller/9800/config-guide/b_upgrade_fpga_c9800.html
for more info's on how to do that and or examine current versions (e.g.)

M.

-- Each morning when I wake up and look into the mirror I always say ' Why am I so brilliant ? '
When the mirror will then always repond to me with ' The only thing that exceeds your brilliance is your beauty! '

pannick · ‎02-19-2025

Figured it out. I forgot to activate and commit the SMU patch. Did that and everything was good to go.

Todd S · ‎02-27-2025

Having a similar issue with 9800-F-L running 17.9.6. I shouldn't need the 17.9.5 SMU patches if I'm running 17.9.6 correct?

marce1000 · ‎02-27-2025

- @Todd S That is correct ,for the rest visit look at my reply in the thread :

- These commands could be useful :
show platform software system all
  show platform resources
show facility-alarm status
  show processes cpu platform sorted
show processes cpu platform sorted | exclude 0%
show processes cpu sorted
    show platform software process slot chassis active R0 monitor (==> list process using the most CPU on top!)
show platform software process slot chassis active R0 monitor | include Mem
show processes memory platform sorted
show processes memory platform accounting (look at max_diff_calls , if on subsequent execution of the command, this parameter keeps increasing for a particular process then the process may have a memory leak)

It's also always advisable to checkup the controller's configuration using the CLI command
show tech wireless and feed the output from that into : Wireless Config Analyzer

M.

-- Each morning when I wake up and look into the mirror I always say ' Why am I so brilliant ? '
When the mirror will then always repond to me with ' The only thing that exceeds your brilliance is your beauty! '