AP is up and down

[Leftz]

Hi We have c9800 and ap 3702e. Recently we get error message from network monitor device. The error message show like the below. but when we check ap status and wlc, we can see nothing wrong, meaning no ap up and down. so that we doubt if this is non wireless issue. Anyone can provide suggestion to make it clear? Thanks

10.1.1.111 (AP-10) is Up at least 5 min.,

May 09, 2023 06:18:17 PM

Device IP: 10.1.1.111

Device Name: AP-10

Down Monitors:

Sent from WhatsUp Gold

[Flavio Miranda]

Hi

 This can be some false positive, very common in Monitoring tools or this can be a flap, which means, the AP lost communication with the WLC for a short period and came back onlline. 

  You can check that by going to Monitoring >  AP statistics check for 

Time at last successful DTLS session

[Leftz]

Thank you Flavio for your reply!

I did not find "Time at last successful DTLS session" as you mentioned. Instead I went to Monitering ---> Wireless ----> Radio Statistics, where i can see "Uptime" for each AP. The issue is this Uptime is for many days long, but the error message is sent to us almost once every day. 

[Flavio Miranda]

Go to Monitoring, AP Statistics , Join Statistics >  In General you have: 

Time at which the AP joined this controller last time

And in Statistics you have

Time at last successful DTLS session

[David Ritter]

from the home page,

Monitor/Statistics/AP Join

unless the table has been cleared, your device MAC will appear.  Click on the MAC  

oops  thats for the 5500 class

9800

Monitor/Wireless/AP Statistics/Join Statistics (not General)

click on target MAC

[Leftz]

I found it. we have about 60 AP 3702e, and only about 10 of these APs have this issue. but all of these AP are normal like below. Is this correct? In other word, physical connection has no issue. and DTLS session looks not issue as well

[David Ritter]

Humm, could these 3702's be suffering from a cert expiry issue?  The easyest way would be to ssh into the device and check the logging.  looking for the point that they reload.

or they are downloading a new image, that fails (cert expiry) and they reload.  again and again.  I have an 1852 doing that currently.

[Leftz]

The below is impacted AP log info. Looks something wrong. I change mac add for security.

Drop the client self-originated pkt, mc2uc_dstMac=201e.xxx, srcMac=201e.vvv-
Drop the client self-originated pkt, mc2uc_dstMac=201e.xxx, srcMac=201e.vvv-

*Apr 4 18:17:17.931: %CAPWAP-1-ARP_ERROR: Could not arp the controller ip address, retry later
*Apr 5 14:31:55.619: %CAPWAP-1-ARP_ERROR: Could not arp the controller ip address, retry later
*Apr 6 07:14:01.803: %CAPWAP-1-ARP_ERROR: Could not arp the controller ip address, retry laterDrop the client self-originated pkt, mc2uc_dstMac=647yyyy, srcMac=6479tttt

*Apr 12 09:56:45.415: %CAPWAP-1-ARP_ERROR: Could not arp the controller ip address, retry laterDrop the client self-originated pkt, mc2uc_dstMac=a059.xxx, srcMac=a059.xxx
Drop the client self-originated pkt, mc2uc_dstMac=a059.xxx, srcMac=a059.xxx
Drop the client self-originated pkt, mc2uc_dstMac=a059.xxx, srcMac=a059.xxx

*May 10 09:12:18.419: %CAPWAP-1-ARP_ERROR: Could not arp the controller ip address, retry later
*May 11 02:38:10.207: %CAPWAP-1-ARP_ERROR: Could not arp the controller ip address, retry later
*May 11 17:33:11.267: %CAPWAP-1-ARP_ERROR: Could not arp the controller ip address, retry later

[Rich R · Accepted solution]

So the question really is how does your tool (WhatsUp Gold?) judge that an AP is up or down.  I'm guessing it's not clever enough to query the controller so you've probably just got it pinging the AP and when it misses 1 (or more) pings it marks the AP down?

Then looking at your log... AP saying it can't ARP the controller IP - that suggests you have a connectivity issue on the AP switch port.  And if it can't ARP it certainly won't be able to respond to ping either.  Obviously the connectivity is recovering before CAPWAP declares the DTLS down which is why you're not seeing the outage on the join stats.

So work out what is causing the connectivity loss on the AP port.  Check the switch logs, check the switch port stats, check the AP port stats for a start.

[Leftz]

@Rich R You are right. finally we found default gateway is incorrect. Thanks

[Rich R]

Ha ha - glad you worked it out eventually - that's a classic!
Always have to check the basics are right and troubleshoot methodically.