9800-CL - ha_port tx hang

[Jegan Rajappa]

Hi, I am not sure how many of you are seeing problem like me, In my 9800-CL HA SSO deployments active role switchover occurs from chassis 1 to chassis 2, show chassis says chassis 1 is removed, active chassis logs ha_port tx hang message, by looking into chassis 1 console it seems it is in recovery mode, reloading chassis 1 allows it to join HA SSO as standby and finally chassis stops logging ha_port tx hang message, sometimes need to reload both the chassis to bring it to normal state.

WLC1#
Feb  2 08:26:14.984 UTC: %IOSXE-3-PLATFORM: Chassis 2 R0/0: kernel: vmxnet3 0000:1b:00.0 ha_port: tx hang
WLC1#
Feb  2 08:26:20.617 UTC: %IOSXE-3-PLATFORM: Chassis 2 R0/0: kernel: vmxnet3 0000:1b:00.0 ha_port: tx hang
WLC1#
Feb  2 08:26:25.737 UTC: %IOSXE-3-PLATFORM: Chassis 2 R0/0: kernel: vmxnet3 0000:1b:00.0 ha_port: tx hang
WLC1#

WLC1#show chassis
Chassis/Stack Mac Address : 0050.56bb.9b87 - Foreign Mac Address
Mac persistency wait time: Indefinite
                                             H/W   Current
Chassis#   Role    Mac Address     Priority Version  State                 IP
-------------------------------------------------------------------------------------
 1       Member   0000.0000.0000     0      V02     Removed              169.254.137.82
*2       Active   0050.56bb.0c97     1      V02     Ready                169.254.137.83

WLC1#

[marce1000]

         - What kind of hypervisor solution(s) are you using ?

 M.

[Jegan Rajappa]

Vmware

[Scott Fella]

Recovery mode in SSO basically means that the image doesn't match, the SSO configuration is wrong, there is an issue with hitting the gateway or the RP connection is bouncing or not working and or a problem with the unit itself.  I had this issue once where only one would show up Ready and the other Removed and I basically had to rebuild the VM's.  I was able to build the one that showed Remove and it worked for a while, but was running into other issues.  So I ended up rebuilding both.  Could of been something when I first spun up the VM's or something I did while it was up.  I have spun up many instances after that with no issues.  What is nice about using a hypervisor is that you can always spin another up and join it to an existing controller after shutting down the broken one.

[Jegan Rajappa]

Both the appliances are having same image, I am seeing this issue for 2nd time, its different machine this time, so I will wait to gather more data before rebuilding it, In my eniv rebuilding is quite time taking job.

[Scott Fella]

I don't know how your environment is, but in mine, I have more than enough resources to build whatever I need.  This gives me flexibility which helps if I need to create a new vm for disaster recovery or testing.  If issues happen on different host, then that is a bit harder to identify.  If the issue happens on one host but not any other, then I would say its something with the host.  Either way, its something you need to be comfortable at especially with VM's as patching or hardware issues can become a big issue.  My environment is ESXi 6.7, ESXi 7 and Hyper-V 2019.

[Scott Fella]

I'm assuming you have seen this: https://kb.vmware.com/s/article/76808