Upgrading multiple 9800 WLCs using ISSU - outages experienced

[brian.mcdonagh@kerry.com]

Hi there, I am upgrading a bunch of WLCs at different sites from 17.9.4a to 17.9.5.

I am performing the upgrade using GUI (browser) with ISSU Upgrade tickbox ticked (AP upgrade per iteration set to 5% to minimise disruption).

Problem is, 2 of the sites reported a full wireless outage - and when I checked there were zero APs associated to the WLC - although it was upgraded to 17.9.5. I waited a few minutes but was seeing zero AP join activity on the WLC so I failed over to the standby WLC and then saw APs join and then users join back to wireless.

anyone seen this before? any workarounds? all the WLCs are all configured the exact same way so I am scratching my head as to why just 2 WLCs did not behave like the rest.

[marce1000]

  - Difficult to tell without detailed info's such as gathering logs on the controller(s) involved ; or better configure a (online) syslog server and enable global AP syslogging too (first). Lots of stuff to investigate afterwards!  ISSU has been known to be buggy at times , sometimes people avoiding it. 

                                       If everything 'has landed' ; you may find these commands useful :
                  https://www.cisco.com/c/en/us/support/docs/wireless/catalyst-9800-series-wireless-controllers/217738-monitor-catalyst-9800-kpis-key-performa.html#anc4
                 https://www.cisco.com/c/en/us/support/docs/wireless/catalyst-9800-series-wireless-controllers/217738-monitor-catalyst-9800-kpis-key-performa.html#anc5
                  https://www.cisco.com/c/en/us/support/docs/wireless/catalyst-9800-series-wireless-controllers/217738-monitor-catalyst-9800-kpis-key-performa.html#anc3

 M.
                          

[No Cables on Me]

@marce1000  is correct, you would need some log and a bit more information to start looking at what may have gone wrong.

The 2 WLC's that failed, can we assume they were in SSO, what was the state of SSO when you check?

[Rich R]

Consider yourself lucky that the upgrades worked at all.
You'll see many comments here from me and others on how problematic ISSU is.
We now just use reload for upgrades to avoid the hassle with the various screwed up ISSU situations which invariably require at least 1 reload, sometimes 2, to fix.

Leo actually recommends getting TAC on a WebEx *before* you start ISSU upgrade so they can help when it inevitably goes wrong.

If it happens again make sure to capture "show tech wireless" from the WLC, packet captures on WLC, and AP switchport, and logs from AP - as a minimum.  You probably also want to get a radioactive trace on one of the AP MAC addresses. TAC might still be able to get some info from the logs to work out why it went wrong now. 

[Leo Laohoo]

CSCwh76420

CSCwh29442