SSID disappears and after some seconds it appears again

[mateus.schott]

I'm having the following problem, ssid disappears from time to time, the client disconnects and after a few seconds the ssid appears again, it happens continuously. I think that can be a problem with rogue APs or some interference, has anyone ever had a similar case?

[Flavio Miranda]

Hi,

 It seems to be interference. There's a wireless behavior called fading where the wireless signal vary becoming weaker time to time due interference thus lowering the coverage and causing the SSID to disappear.

 Try to test on 5.0 GHz.

-If I helped you somehow, please, rate it as useful.-

[mateus.schott]

Guys, attached have a vídeo with the problem, SSID W-UNICORP appear more later, this happens frequently, disappears from time to time, happens with other SSIDs too. This happens more often in a region, where it has 6 APs, I have another 50 APs in other branches that connect in this WLC and in these it does not happen.

[Manish Mathur]

Mateus ,

How long does the SSID take to re-appear ? As u mentioned that the issue is specific to ALL the 6 APs in a particular site , can you check the association time of these APs on the WLC ? The command would be "show ap uptime".

Thanks,

Manish

[mateus.schott]

Manish,

It does not have a time pattern, sometimes it happens in five minutes, sometimes it takes more than an hour.

The problem is more noticeable in the APs that start with the prefix ADM, since they are the most used, maybe even happen in the other APs but since they are remote locations maybe the users do not notice or do not warn me.

show ap uptime

Number of APs.................................... 66
Global AP User Name.............................. admin
Global AP Dot1x User Name........................ Not Configured

AP Name Ethernet MAC AP Up Time Association Up Time
------------------ ----------------- ----------------------- -----------------------
ADM-AP06 a0:3d:6f:a6:af:80 45 days, 09 h 42 m 03 s 45 days, 09 h 40 m 29 s
F043-AP02 00:62:ec:2c:2e:20 45 days, 09 h 37 m 07 s 45 days, 07 h 22 m 57 s
F043-AP03 00:c8:8b:02:46:38 45 days, 09 h 37 m 54 s 45 days, 07 h 22 m 54 s
ADM-AP05 00:6b:f1:da:8b:a4 44 days, 21 h 59 m 10 s 44 days, 21 h 57 m 37 s
ADM-AP04 00:6b:f1:db:f0:d8 44 days, 21 h 59 m 06 s 44 days, 21 h 55 m 01 s
F043-AP01 00:62:ec:2c:2c:14 45 days, 09 h 03 m 39 s 37 days, 21 h 59 m 24 s
ADM-AP11 b0:aa:77:0c:24:35 19 days, 20 h 38 m 19 s 19 days, 20 h 36 m 44 s
ADM-AP10 b0:aa:77:0c:24:4e 7 days, 01 h 22 m 34 s 7 days, 01 h 21 m 00 s
F060-Outdoor 00:81:c4:88:65:02 27 days, 13 h 03 m 34 s 6 days, 17 h 45 m 24 s
F060-AP05 00:6b:f1:e0:eb:60 26 days, 21 h 17 m 27 s 6 days, 17 h 45 m 23 s
F060-AP02 00:6b:f1:e0:eb:8c 45 days, 02 h 10 m 07 s 6 days, 17 h 45 m 20 s
F060-AP03 00:6b:f1:e0:eb:94 45 days, 01 h 36 m 35 s 6 days, 17 h 45 m 19 s
F060-AP04 00:6b:f1:e3:34:d0 45 days, 02 h 03 m 07 s 6 days, 17 h 45 m 17 s
F060-AP01 70:7d:b9:ba:40:b0 36 days, 16 h 37 m 17 s 6 days, 17 h 45 m 16 s
F055-AP04 00:6b:f1:e3:34:cc 42 days, 02 h 37 m 33 s 6 days, 16 h 32 m 36 s

F055-AP03 00:6b:f1:da:8b:e8 39 days, 17 h 36 m 11 s 6 days, 16 h 32 m 32 s
F055-AP01 00:6b:f1:da:8b:b0 42 days, 03 h 45 m 10 s 6 days, 16 h 32 m 31 s
F059-AP02 00:c8:8b:02:46:44 28 days, 05 h 45 m 50 s 5 days, 18 h 19 m 33 s
F059-AP03 00:c8:8b:02:ac:74 28 days, 06 h 08 m 40 s 5 days, 18 h 19 m 31 s
F059-AP01 00:c8:8b:02:46:30 10 days, 16 h 46 m 48 s 5 days, 18 h 19 m 26 s
F059-AP04 00:c8:8b:02:45:e8 10 days, 16 h 42 m 59 s 5 days, 18 h 19 m 26 s
F055-AP02 00:6b:f1:e3:34:6c 4 days, 18 h 07 m 21 s 4 days, 18 h 05 m 47 s
F056-AP01 a0:3d:6f:a9:76:ec 11 days, 13 h 14 m 46 s 2 days, 06 h 06 m 59 s
CAD1-AP07 00:c8:8b:02:45:08 45 days, 03 h 50 m 38 s 0 days, 19 h 52 m 49 s
CAD2-AP01 00:c8:8b:02:9e:00 45 days, 00 h 00 m 00 s 0 days, 19 h 52 m 48 s
CAD2-AP08 00:c8:8b:02:45:e0 45 days, 04 h 26 m 16 s 0 days, 19 h 52 m 48 s
DEC-AP04 00:c8:8b:02:ac:c8 45 days, 01 h 22 m 08 s 0 days, 19 h 52 m 48 s
DEC-AP08 00:c8:8b:02:aa:f8 45 days, 03 h 30 m 31 s 0 days, 19 h 52 m 48 s
CAD2-AP09 00:c8:8b:02:9d:20 45 days, 04 h 03 m 20 s 0 days, 19 h 52 m 48 s
DEC-AP05 00:c8:8b:02:ac:dc 45 days, 03 h 45 m 14 s 0 days, 19 h 52 m 47 s
CAD1-AP13 00:c8:8b:02:ac:d0 45 days, 01 h 54 m 56 s 0 days, 19 h 52 m 47 s
CAD2-AP04 00:c8:8b:02:45:98 45 days, 02 h 45 m 04 s 0 days, 19 h 52 m 46 s
CAD1-AP12 00:c8:8b:02:ac:34 45 days, 02 h 44 m 42 s 0 days, 19 h 52 m 46 s
CAD2-AP11 00:c8:8b:02:ac:04 45 days, 02 h 43 m 15 s 0 days, 19 h 52 m 45 s
CAD1-AP01 00:c8:8b:02:9c:c0 45 days, 00 h 59 m 40 s 0 days, 19 h 52 m 45 s
DEC-AP12 00:c8:8b:02:46:28 45 days, 01 h 26 m 54 s 0 days, 19 h 52 m 44 s
CAD1-AP14 00:c8:8b:02:ac:88 44 days, 23 h 47 m 03 s 0 days, 19 h 52 m 44 s
CAD2-AP05 00:c8:8b:02:46:18 45 days, 01 h 24 m 23 s 0 days, 19 h 52 m 44 s
DEC-AP11 00:c8:8b:02:a9:60 45 days, 02 h 55 m 30 s 0 days, 19 h 52 m 44 s
CAD2-AP16 00:c8:8b:02:45:c0 45 days, 01 h 47 m 19 s 0 days, 19 h 52 m 44 s
DEC-AP01 00:c8:8b:02:45:cc 45 days, 00 h 13 m 51 s 0 days, 19 h 52 m 44 s
CAD2-AP15 00:c8:8b:02:45:2c 45 days, 03 h 18 m 16 s 0 days, 19 h 52 m 44 s
CAD2-AP14 00:c8:8b:02:46:34 45 days, 02 h 53 m 17 s 0 days, 19 h 52 m 42 s
CAD1-AP08 00:c8:8b:02:45:54 45 days, 03 h 27 m 07 s 0 days, 19 h 52 m 41 s
CAD1-AP15 00:c8:8b:02:aa:b4 45 days, 02 h 06 m 35 s 0 days, 19 h 52 m 41 s
DEC-AP06 00:c8:8b:02:ab:e0 45 days, 02 h 43 m 19 s 0 days, 19 h 52 m 36 s
CAD2-AP12 00:c8:8b:02:9d:18 45 days, 03 h 22 m 49 s 0 days, 19 h 52 m 38 s
CAD1-AP06 00:c8:8b:02:45:44 45 days, 02 h 42 m 32 s 0 days, 19 h 52 m 38 s
DEC-AP03 00:c8:8b:02:ab:e4 45 days, 03 h 07 m 15 s 0 days, 19 h 52 m 40 s
CAD1-AP04 00:c8:8b:02:45:04 45 days, 04 h 20 m 14 s 0 days, 19 h 52 m 40 s
CAD2-AP10 00:c8:8b:02:ab:b8 45 days, 04 h 09 m 14 s 0 days, 19 h 52 m 40 s
CAD1-AP10 00:c8:8b:02:45:14 45 days, 02 h 15 m 31 s 0 days, 19 h 52 m 40 s
CAD1-AP02 00:c8:8b:02:45:e4 45 days, 02 h 46 m 37 s 0 days, 19 h 52 m 37 s
DEC-AP07 00:c8:8b:02:ac:4c 45 days, 01 h 26 m 41 s 0 days, 19 h 52 m 40 s
CAD1-AP05 00:c8:8b:02:45:28 45 days, 01 h 55 m 19 s 0 days, 19 h 52 m 40 s
CAD2-AP06 00:c8:8b:02:45:1c 45 days, 02 h 26 m 18 s 0 days, 19 h 52 m 40 s
DEC-AP09 00:c8:8b:02:aa:dc 45 days, 02 h 53 m 21 s 0 days, 19 h 52 m 40 s
CAD1-AP09 00:c8:8b:02:45:30 45 days, 03 h 30 m 04 s 0 days, 19 h 52 m 40 s
CAD1-AP03 00:c8:8b:02:44:cc 45 days, 03 h 56 m 44 s 0 days, 19 h 52 m 40 s
CAD2-AP13 00:c8:8b:02:45:b8 45 days, 02 h 17 m 55 s 0 days, 19 h 52 m 40 s
DEC-AP02 00:c8:8b:02:ad:04 45 days, 00 h 18 m 58 s 0 days, 19 h 52 m 38 s
CAD2-AP07 00:c8:8b:02:aa:94 45 days, 04 h 05 m 35 s 0 days, 19 h 52 m 37 s
DEC-AP10 00:c8:8b:02:9d:58 45 days, 02 h 23 m 55 s 0 days, 19 h 48 m 15 s
CAD2-AP02 00:c8:8b:02:9d:38 45 days, 01 h 15 m 46 s 0 days, 19 h 48 m 12 s
CAD2-AP03 00:c8:8b:02:46:3c 45 days, 03 h 44 m 46 s 0 days, 19 h 48 m 11 s
CAD1-AP11 00:c8:8b:02:44:d0 45 days, 01 h 56 m 36 s 0 days, 01 h 06 m 59 s

[Scott Fella]

That information shows that the controller that the communication between the AP and the WLC is breaking. This is either a network issue between the two or possibly a code issue on the WLC. I have seen both but the latter was noticed after an upgrade. 

[mateus.schott]

This is interesting, because I recently upgraded the WLC to the version 8.3.133. I'm monitoring the AP ADM-AP11 which users are complaining more often, and both are on the same switch.

### WLC port
interface Port-channel9
description *** WLC-MTZ01 ***
switchport trunk encapsulation dot1q
switchport mode trunk

interface GigabitEthernet2/0/9
description *** WLC-MTZ01 - Port2 ***
switchport trunk encapsulation dot1q
switchport mode trunk
channel-group 9 mode on

interface GigabitEthernet1/0/9
description *** WLC-MTZ01 - Port1 ***
switchport trunk encapsulation dot1q
switchport mode trunk
channel-group 9 mode on

### AP port - AP is in mode local
interface GigabitEthernet2/0/2
description *** ADM-AP11 - Gig 0 ***

AP is in VLAN 1, actually everyone here in the array is in vlan 1, I have a large broadcast domain 172.16.0.0/16, it's a problem I know but we're walking to segment the network. But it has always been like that and recently the problem did not happen and we have not had very big changes on the network recently.

Thank's all for helping me.

[Scott Fella]

Well if you upgraded and see this issue, then upgrade or downgrade to fix it. There is not much you can do here beside that.

[mateus.schott]

Ok Scott, I'will do a downgrade and see what happen, for now, thank you!!

[Scott Fella]

Keep us posted. What I have seen in some of our sites is that we would upgrade and only a site or two would be affected. Majority of the other sites were stable on he same code. Downgrading helped and then I tried to upgrade again and ran into the same issue. I ended up upgrading to a higher code which was stable and then downgrading back tot he Code we want d and everything worked. This might be something to try is you want.

[Manish Mathur]

Hey guys,

I may be missing something (please correct me if I am wrong) here but at least for the ADM APs , I see that the "AP up time"is exactly the same as the "association time". Thus, I don't think that the communication between the ADM APs and the WLC is getting dropped.

Are these ADM APs in Flexconnect mode ? or local mode ?

What is the channel range on which these APs are working ? DFS channels ? 

Can u collect the events.log from any one AP which shows the issue ? and if possible , approximate time you saw the issue on this AP ? The events.log should be able to show if "beaconing is being stoped" by the AP frequently or the "Radio resets"/"DFS events"/"frequent channel changes" ??

Thanks,

Manish

[mateus.schott]

Hi Manish,

you're right with regard to association time.

What is the channel range on which these APs are working ? DFS channels ? 

2.4GHz - 1,6,11

5GHz - 56, 60, 149, 153, 157, 161

Attached is the event.log and show logg of 3 ADM APs.

[Manish Mathur · Accepted solution]

Hi Mateus and Scott ,

So, it is now clear that the APs are not getting dissociated from the WLC. We can rule out that possibility.

Next, the event logs from the AP clearly show that the radio interfaces are going in reset down mode frequently after these continuous messages:

*Dec 26 14:50:09.859: %DOT11-4-CCMP_REPLAY: Client 00b3.622a.2adf had 1 AES-CCMP TSC replays
*Dec 26 14:50:17.859: %DOT11-4-CCMP_REPLAY: Client 00b3.622a.2adf had 1 AES-CCMP TSC replays
*Dec 26 15:40:57.111: %DOT11-4-CCMP_REPLAY: Client 5ce0.c56a.4753 had 1 AES-CCMP TSC replays
*Dec 26 15:41:05.111: %DOT11-4-CCMP_REPLAY: Client 5ce0.c56a.4753 had 1 AES-CCMP TSC replays
*Dec 26 15:41:39.119: %DOT11-4-CCMP_REPLAY: Client 5ce0.c56a.4753 had 1 AES-CCMP TSC replays
*Dec 26 15:42:32.139: %DOT11-4-CCMP_REPLAY: Client 5ce0.c56a.4753 had 2 AES-CCMP TSC replays

Now, the reason behind these messages can be seen here:

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCsx50775/?referring_site=bugquickviewredir

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCur62548/?referring_site=bugquickviewredir

I remember facing similar issue on our infrastructure and this can be confirmed if you collect the output of "show controllers d1 | be Last " from the AP CLI. This will show the section for radio resets.

Solution:

1. If possible , please upgrade the drivers of the clients so that they stop bombarding the AP radio with these replays. You can get the MAC addresses of the clients in the event logs.

2. If not, you need to run a command on the AP which will not let the radio go into down state even when such replays are bombarded. I don't have that command on top of my mind. Will have to search.

Cheers

Manish

[mateus.schott]

Manish,

Thank you for your help, is clear for me, I'll search the command, I think that is the solution for me.

Any news I'll let you know.

[Manish Mathur]

Thanks Mateus.

I hope that the TAC case is still in progress. Meanwhile, I just wanted to correct myself so that people going through this thread are not misled.

WLC codes prior to (I guess 8.1) had the default config for the Radios to stop-on-failure . which means that if a Radio fails (due to any given reason including replay errors) , it will stop its function. We need to login and manually turn the radio up to get it working again.

This was recognized as a bug/issue by Cisco and was corrected in the later codes (I believe somewhere in the 8.2 train). The default setting was changed to reset-on-failure , which means that the radio will reset (flap) if it fails. This will obviously drop all the clients but at least the radio will recover and come back up. When the radio resets , the clients will jump to another nearby AP after losing connection (of course).

Hence, in your case , it looks like the config is "reset-on-failure" which is causing the radio to reset due to the failure. The root cause may be the replay errors that we are seeing on the AP. I hope that TAC would recommend an action plan to avoid those replay errors.

All the best. and thanks for keeping us updated.

Cheers,

Manish