Solved: Re: Prime 3.10 with 9800 WLC - No reaction to bsnAPDisassociated SNMP

Ethan Grinnell · ‎10-17-2022

I've noticed that Prime doesn't seem to know when an AP connected to a 9800 WLC disconnects and reconnects. If the AP stays offline for a long time (Several hours) Prime eventually discovers this, and generates an alarm. As if there is a background job that runs to discover it. On an AireOS WLC Prime immediately generates an alarm when the AP goes down and responds appropriately when the the AP recovers.

I have this in production and have recreated it in the lab. The connection to the 9800 is healthy, telemetry is enabled and completed, SNMP is configured and all traps/wireless trap flags are enabled. I have client data in Prime, the 9800 coral service shows no issues, and all APs are discovered. I've tried Prime 3.9 and 3.10 both fully patched. Many different 9800 software versions (17.6.[234], 17.8.1, 17.9.1, and a few others). I've also tried with SNMP versions 1, 2, and 3.

Can anyone confirm that rebooting an AP connected to an 9800 WLC immediately generates the alarm in Prime like it does in the AireOS platform?

I've tried manually triggering all of the jobs in the Prime "Job Dashboard" that I am able to accelerate discovery of down APs, none seem to matter. I also did a packet capture and Prime debug log capture to confirm that Prime is in fact getting the bsnAPDisassociated SNMP trap from the WLC.

It appears that Prime is getting the bsnAPDisassociated SNMP trap in both cases, but it discards it from a 9800 because it claims it's from an unknown AP. The only difference that I see in the trap is that for AireOS the bsnAPEntry.bsnAPDot3MacAddress is 0 while for the 9800 it's a bogus MAC address that isn't the radio or Ethernet MAC. Maybe that's an issue, (seems likely to me), but is this working for anyone else?

9800-LF - SNMP Trap has bogus bsnAPDot3MacAddress - Actual radio MAC is 70:b3:17:7d:aa:e0:
From packet capture

        SNMPv2-MIB::sysUpTime.0 (1.3.6.1.2.1.1.3.0): 67111829
        SNMPv2-MIB::snmpTrapOID.0 (1.3.6.1.6.3.1.1.4.1.0): 1.3.6.1.4.1.14179.2.6.3.8 (AIRESPACE-WIRELESS-MIB::bsnAPDisassociated)
            Object Name: 1.3.6.1.6.3.1.1.4.1.0 (SNMPv2-MIB::snmpTrapOID.0)
            SNMPv2-MIB::snmpTrapOID: 1.3.6.1.4.1.14179.2.6.3.8 (AIRESPACE-WIRELESS-MIB::bsnAPDisassociated)
        AIRESPACE-WIRELESS-MIB::bsnAPMacAddrTrapVariable.0 (1.3.6.1.4.1.14179.2.6.2.20.0): Cisco_7d:aa:e0 (70:b3:17:7d:aa:e0)
            Object Name: 1.3.6.1.4.1.14179.2.6.2.20.0 (AIRESPACE-WIRELESS-MIB::bsnAPMacAddrTrapVariable.0)
            AIRESPACE-WIRELESS-MIB::bsnAPMacAddrTrapVariable: Cisco_7d:aa:e0 (70:b3:17:7d:aa:e0)
        AIRESPACE-WIRELESS-MIB::bsnAPName.6.112.179.23.124.41.12 (1.3.6.1.4.1.14179.2.2.1.1.3.6.112.179.23.124.41.12): 457468616e2d4f66666963652d323830322d31
            Object Name: 1.3.6.1.4.1.14179.2.2.1.1.3.6.112.179.23.124.41.12 (AIRESPACE-WIRELESS-MIB::bsnAPName.6.112.179.23.124.41.12)
                AIRESPACE-WIRELESS-MIB::bsnAPEntry.bsnAPDot3MacAddress: 06:70:b3:17:7c:29 (06:70:b3:17:7c:29)
            AIRESPACE-WIRELESS-MIB::bsnAPName: 457468616e2d4f66666963652d323830322d31

Prime log

==> /opt/CSCOlumos/logs/ncs-0-0.log <==
09/19/22 04:01:12.103 TRACE [faults] [ClientEnhancedTrapHandler] trapRecordData = Package:com.cisco.xmp.decap.base
Name:rawRcvSec;Kind:integer;Value:1663585272
Name:rawRcvUsec;Kind:integer;Value:87000
Name:trapOid;Kind:oid;Value:1.3.6.1.4.1.14179.2.6.3.8
Name:rcvSourcePort;Kind:integer;Value:49302
Name:notificationDeliveryMechanism;Kind:integer;Value:2
Name:trapSrcAddr;Kind:ipAddress;Value:192.168.20.10
Name:trapVersion;Kind:integer;Value:2
Name:trapPDU;Kind:ber;Value:TRAP[requestID=10817, errorStatus=Success(0), errorIndex=0, VBS[1.3.6.1.2.1.1.3.0 = 6 days, 1:44:27.33; 1.3.6.1.6.3.1.1.4.1.0 = 1.3.6.1.4.1.14179.2.6.3.8; 1.3.6.1.4.1.14179.2.6.2.20.0 = 70:b3:17:7d:aa:e0; 1.3.6.1.4.1.14179.2.2.1.1.3.6.112.179.23.124.41.12 = Ethan-Office-2802-1]]
Package:com.cisco.xmp.decap.event
Name:eventName;Kind:displayString;Value:bsnAPDisassociated
Package:com.cisco.xmp.decap.trap
Name:bsnAPMacAddrTrapVariable;Kind:macAddress;Value:70b3177daae0
Name:bsnAPName;Kind:adminString;Value:Ethan-Office-2802-1
09/19/22 04:01:12.103 TRACE [faults] [ClientEnhancedTrapHandler] trapOid : = 1.3.6.1.4.1.14179.2.6.3.8
09/19/22 04:01:12.104 TRACE [faults] [ClientEnhancedTrapHandler] trapType : = LRAD_DISASSOCIATED
...
==> /opt/CSCOlumos/logs/ncs-0-0.log <==
09/19/22 04:01:12.209 TRACE [faults] [IOBoundTrapHandler] Ignoring Lrad Associate / Disassociate Trap since the switch is unknown. AP: or EwlcSeries(9800) 70:b3:17:7d:aa:e0 senderIP: 192.168.20.10
09/19/22 04:01:12.401 TRACE [faults] [DefaultTrapHandler] trapRecordData = Package:com.cisco.xmp.decap.base

AireOS - SNMP Trap has 0-valued bsnAPDot3MacAddress - Actual radio MAC is 70:b3:17:7d:aa:e0:
From packet capture

    AIRESPACE-WIRELESS-MIB::bsnAPName.0 (1.3.6.1.4.1.14179.2.2.1.1.3.0): 457468616e2d4f66666963652d323830322d31
        Object Name: 1.3.6.1.4.1.14179.2.2.1.1.3.0 (AIRESPACE-WIRELESS-MIB::bsnAPName.0)
            AIRESPACE-WIRELESS-MIB::bsnAPEntry.bsnAPDot3MacAddress: 00:00:00_00:00:00 (00:00:00:00:00:00)
        AIRESPACE-WIRELESS-MIB::bsnAPName: 457468616e2d4f66666963652d323830322d31

Prime log - No error and immediately marks AP down

==> /opt/CSCOlumos/logs/ncs-0-0.log <==
09/19/22 00:26:43.677 TRACE [faults] [IOBoundTrapHandler] trapOid : = 1.3.6.1.4.1.14179.2.6.3.8
09/19/22 00:26:43.678 TRACE [faults] [IOBoundTrapHandler] trapType : = LRAD_DISASSOCIATED
09/19/22 00:26:43.678 TRACE [faults] [IOBoundTrapHandler] Trap Type LRAD_DISASSOCIATED /192.168.20.11
09/19/22 00:26:43.678 TRACE [faults] [RogueTrapHandler] trapRecordData = Package:com.cisco.xmp.decap.base
Name:rawRcvSec;Kind:integer;Value:1663572403
Name:rawRcvUsec;Kind:integer;Value:564000
Name:trapOid;Kind:oid;Value:1.3.6.1.4.1.14179.2.6.3.8
Name:rcvSourcePort;Kind:integer;Value:60315
Name:notificationDeliveryMechanism;Kind:integer;Value:2
Name:trapSrcAddr;Kind:ipAddress;Value:192.168.20.11
Name:trapVersion;Kind:integer;Value:2
Name:trapPDU;Kind:ber;Value:TRAP[requestID=108, errorStatus=Success(0), errorIndex=0, VBS[1.3.6.1.2.1.1.3.0 = 0:49:27.00; 1.3.6.1.6.3.1.1.4.1.0 = 1.3.6.1.4.1.14179.2.6.3.8; 1.3.6.1.4.1.14179.2.6.2.20.0 = 70:b3:17:7d:aa:e0; 1.3.6.1.4.1.14179.2.2.1.1.3.0 = Ethan-Office-2802-1]]
Package:com.cisco.xmp.decap.event
Name:eventName;Kind:displayString;Value:bsnAPDisassociated
Package:com.cisco.xmp.decap.trap
Name:bsnAPMacAddrTrapVariable;Kind:macAddress;Value:70b3177daae0
Name:bsnAPName;Kind:adminString;Value:Ethan-Office-2802-1

Ethan Grinnell · ‎10-17-2022

Replying to my own post, this might be related to CSCvy02956 which mentions needing support for " .1.3.6.1.4.1.14179.2.2.1.1.3 bsnAPName Name of Current AP", but is anyone else able to see immediate up/down status of APs on 9800 WLCs?

Edit: It wasn't the above bug, it turns out that it was https://bst.cloudapps.cisco.com/bugsearch/bug/CSCwc18004 It's fixed in 17.9.2 and also worked in 17.3.3. Versions from 17.3.3+ through 17.9.1 were broken. The bsnAPDot3MacAddress looks the same as before, so apparently the issue wasn't related to that. It had something with the trap and how Prime is processing it.

View solution in original post

Ethan Grinnell · ‎10-17-2022

Replying to my own post, this might be related to CSCvy02956 which mentions needing support for " .1.3.6.1.4.1.14179.2.2.1.1.3 bsnAPName Name of Current AP", but is anyone else able to see immediate up/down status of APs on 9800 WLCs?

Edit: It wasn't the above bug, it turns out that it was https://bst.cloudapps.cisco.com/bugsearch/bug/CSCwc18004 It's fixed in 17.9.2 and also worked in 17.3.3. Versions from 17.3.3+ through 17.9.1 were broken. The bsnAPDot3MacAddress looks the same as before, so apparently the issue wasn't related to that. It had something with the trap and how Prime is processing it.

Prime 3.10 with 9800 WLC - No reaction to bsnAPDisassociated SNMP trap