Frequent Heart Beat Timer Expiry C9800-CL / C9130

[PBethek3]

Hi All,

Thanks in advance for hopping in and reading this and for any advice you can provide!

My scenario is extremely frustrating for me. I am running a virtualized C9800-CL (17.11.1) locally hosted on a Supermicro SYS-110D-8C-FRAN8TP, 8 CPU cores, 16GB RAM provided to the VM. I currently have 2x Cisco C9130AXI-B APs connected and joined to the controller, final deployment will have 14 C9130AXI-Bs, 5 C9130AXE-Bs. I am seeing consistent "heart beat timer expiry" events and AP join disjoins. This is causing service interruptions to my end devices in my test environment. (Access points are not overlapping in coverage as they are on opposite ends of the warehouse currently) 

Running through Cisco's Wireless Config Analyzer (https://cway.cisco.com/wireless-config-analyzer/), I'm not seeing any issues that could be causing this. The only "Error" it is providing is due to mDNS, which I have to disable. I have fixed the "AP ac4a.56c5.06c0did not present WLC name " and "AP ac4a.56c5.06c0did not present WLC IP address" issues as of this morning, they were remnant's from the last installation.

The APs are connected to both a C9300L (AP01) and a Catalyst 3750X (AP02). The C9800-CL is connected to the C9300L directly, and the 3750X is connected via an SFP to TE1/1/3 on the C9300L. Any help that can be provided would be greatly appreciated!

Aug  3 23:39:14.230: %CAPWAPAC_SMGR_TRACE_MESSAGE-5-AP_JOIN_DISJOIN: Chassis 1 R0/0: wncd: AP Event: AP Name: LL-C9130-I-AP01 Mac: 687d.b490.74e0 Session-IP: 10.8.0.31[5272] 10.8.0.8[5246] Ethernet MAC: 045f.b9c4.01d0 Joined

Aug  3 23:39:08.243: %CAPWAPAC_SMGR_TRACE_MESSAGE-5-AP_JOIN_DISJOIN: Chassis 1 R0/0: wncd: AP Event: AP Name: LL-C9130-I-AP01 Mac: 687d.b490.74e0 Session-IP: 10.8.0.31[5272] 10.8.0.8[5246] Disjoined Heart beat timer expiry

Aug  3 23:39:07.054: %CAPWAPAC_SMGR_TRACE_MESSAGE-5-AP_JOIN_DISJOIN: Chassis 1 R0/1: wncd: AP Event: AP Name: LL-C9130-I-AP02 Mac: ac4a.56c5.06c0 Session-IP: 10.8.0.32[5248] 10.8.0.8[5246] Ethernet MAC: 1416.9d2c.3708 Joined

Aug  3 23:39:06.580: %APMGR_TRACE_MESSAGE-4-WLC_CONFIG_CHECKER_WARNING: Chassis 1 R0/1: wncd: config check: AP ac4a.56c5.06c0 did not present WLC IP address in config status request during join.

Aug  3 23:39:06.580: %APMGR_TRACE_MESSAGE-4-WLC_CONFIG_CHECKER_WARNING: Chassis 1 R0/1: wncd: config check: AP ac4a.56c5.06c0did not present WLC name in config status request during join.

Aug  3 23:39:00.960: %CAPWAPAC_SMGR_TRACE_MESSAGE-5-AP_JOIN_DISJOIN: Chassis 1 R0/1: wncd: AP Event: AP Name: LL-C9130-I-AP02 Mac: ac4a.56c5.06c0 Session-IP: 10.8.0.32[5248] 10.8.0.8[5246] Disjoined Heart beat timer expiry

Aug  3 23:03:28.239: %CAPWAPAC_SMGR_TRACE_MESSAGE-5-AP_JOIN_DISJOIN: Chassis 1 R0/0: wncd: AP Event: AP Name: LL-C9130-I-AP01 Mac: 687d.b490.74e0 Session-IP: 10.8.0.31[5272] 10.8.0.8[5246] Ethernet MAC: 045f.b9c4.01d0 Joined

Aug  3 23:03:22.065: %CAPWAPAC_SMGR_TRACE_MESSAGE-5-AP_JOIN_DISJOIN: Chassis 1 R0/0: wncd: AP Event: AP Name: LL-C9130-I-AP01 Mac: 687d.b490.74e0 Session-IP: 10.8.0.31[5272] 10.8.0.8[5246] Disjoined Heart beat timer expiry

Aug  3 23:03:16.816: %CAPWAPAC_SMGR_TRACE_MESSAGE-5-AP_JOIN_DISJOIN: Chassis 1 R0/1: wncd: AP Event: AP Name: LL-C9130-I-AP02 Mac: ac4a.56c5.06c0 Session-IP: 10.8.0.32[5248] 10.8.0.8[5246] Ethernet MAC: 1416.9d2c.3708 Joined

Aug  3 23:03:16.328: %APMGR_TRACE_MESSAGE-4-WLC_CONFIG_CHECKER_WARNING: Chassis 1 R0/1: wncd: config check: AP ac4a.56c5.06c0 did not present WLC IP address in config status request during join.

Aug  3 23:03:16.328: %APMGR_TRACE_MESSAGE-4-WLC_CONFIG_CHECKER_WARNING: Chassis 1 R0/1: wncd: config check: AP ac4a.56c5.06c0did not present WLC name in config status request during join.

Aug  3 23:03:10.804: %CAPWAPAC_SMGR_TRACE_MESSAGE-5-AP_JOIN_DISJOIN: Chassis 1 R0/1: wncd: AP Event: AP Name: LL-C9130-I-AP02 Mac: ac4a.56c5.06c0 Session-IP: 10.8.0.32[5248] 10.8.0.8[5246] Disjoined Heart beat timer expiry

Thanks again,

~Paul

[Leo Laohoo]

Any operational reason(s) or requirement(s) why the controller have-to-be on 17.11.1?  

Because I would never recommend using a single release (like the 17.11.1) in production unless the objective was to "help Cisco find bugs".

[Scott Fella]

What are you using to virtualize the 9800, VMware, Hyper-V, etc?  You really need to make sure you read the implementation guide closely. There are certain requirements that might differ for another platform. Also with the 9800-CL, you want to make sure you implement FlexConnect and not use local mode. 

[Rich R]

Adding to what Leo said you should be using an extended support release like 17.9.x or 17.12.x - a good choice for now would be 17.9.4.  Standard support releases like 17.11 do not get any bug fixes at all - the only reason for ever using these is to get a new feature which isn't yet available in an extended support release, with the intention to migrate to an extended support release ASAP.
https://www.cisco.com/c/en/us/products/collateral/ios-nx-os-software/ios-xe-16/bulletin-c25-2378701.html