WLC4402 drops network connection

[earlbwilliams]

We have 2 AIR-WLC4402-K9 devices at a remote location that will both drop their network connections some undetermined time after a reboot.  We cannot reach them via telnet, SSH or HTTP.  In fact, we cannot even ping them once they drop connection.  The only cure we've found so far is to power-cycle the controllers.  The controllers are in separate rooms and connected to separate switches.  We've confirmed the links are configured as trunks and have the correct speed/duplex settings.  I tried updating the IOS & boot loader on one of them but that had no effect on the problem. 

I've seen other posts on here related to this issue but so far there seems to be no solution.  Any help/advice is appreciated.

[Saravanan Lakshmanan]

post the msglog from external server for the affected wlc when issue occured.

from msglog, while the issue getting reproduced search for keyword - handle.

[Leo Laohoo]

We've confirmed the links are configured as trunks and have the correct speed/duplex settings.  I tried updating the IOS & boot loader on one of them but that had no effect on the problem.

Sorry Earl, you'll need to post more information.

1.  What firmware are the 4400 currently running?

2.  Do you have a link to the Service Port?

3.  How is the controllers connected to your network?  Fibre or copper?  If copper, what is the make and model of your switch?  If it's Cisco, what EXACT model?

[earlbwilliams]

Firmware version:  7.0.235.3

Service Port access:  no, but I'm working on it - device is remote

Connection:  copper to Cisco C3750G stack:

*    1 12    WS-C3750G-12S      12.2(55)SE6           C3750-IPBASEK9-M

     2 24    WS-C3750G-24T      12.2(55)SE6           C3750-IPBASEK9-M

     3 24    WS-C3750G-24T      12.2(55)SE6           C3750-IPBASEK9-M

     4 24    WS-C3750G-24T      12.2(55)SE6           C3750-IPBASEK9-M

     5 24    WS-C3750G-24T      12.2(55)SE6           C3750-IPBASEK9-M

Also, we are seeing the following errors in the switch log:

%SW_MATM-4-MACFLAP_NOTIF: Host 5894.6b53.19ec in vlan 1 is flapping between port Gi5/0/17 and port Po4

%SW_MATM-4-MACFLAP_NOTIF: Host 8c70.5aaa.ee66 in vlan 1 is flapping between port Po2 and port Gi5/0/17

%SW_MATM-4-MACFLAP_NOTIF: Host 0811.9621.519c in vlan 1 is flapping between port Gi5/0/17 and port Gi5/0/15

%SW_MATM-4-MACFLAP_NOTIF: Host 0811.9621.519c in vlan 1 is flapping between port Gi5/0/17 and port Gi5/0/15

%SW_MATM-4-MACFLAP_NOTIF: Host 0811.9621.519c in vlan 1 is flapping between port Gi5/0/17 and port Gi5/0/15

%SW_MATM-4-MACFLAP_NOTIF: Host 0811.96dd.7100 in vlan 1 is flapping between port Gi5/0/17 and port Po4

One of the controllers is connected on port Gi5/0/17.  The other controller is not connected to this stack at all.

Here is the config of port Gi5/0/17:

interface GigabitEthernet5/0/17
switchport trunk encapsulation dot1q
switchport mode trunk

Thanks for the help!

Earl

[Stephen Rodriguez]

are you purning the non-wireless VLAN from the port channel to the WLC?  It's possible the NPU is getting hammered by broadcast traffic.

HTH,
Steve

------------------------------------------------------------------------------------------------
Please remember to rate useful posts, and mark questions as answered

[earlbwilliams]

Hi Steve,

I'm not seeing high utilization on the WLC, switch or router at this site.  Also just FYI, they currently have very few wireless clients (around 5).

Thanks,

Earl

[Stephen Rodriguez]

Ok, but how many wired clients are there, multiple vlans?  Just want to make sure it's not causing a problem.

HTH,
Steve

------------------------------------------------------------------------------------------------
Please remember to rate useful posts, and mark questions as answered

[earlbwilliams]

There was only 1 VLAN (VLAN 1) but I've since added a separate VLAN for the wireless traffic and configured that virtual I/F on the controller.  However, we are still seeing the same issue.  The flapping error shown above seems to appear just prior to the I/F losing connection.

Thanks,

Earl

[smayanja123]

The WLAN 4402 has two ports , are both ports plugged in ? If so , unplug one of them. Also disable LAG on next reboot. That will fix your problem.

Sam
Sent from Cisco Technical Support iPad App

[earlbwilliams]

Well, it appears we had multiple hardware failures.  One controller has a bad port and the other appears to have a bad SFP.  We have been able to get both units to connect successfully but now here's another puzzle.

On the controller with a bad port (port 1) we moved the SFP into port 2 and changed the port configuration so that port is the primary and port 1 is the backup (even though we do not have anything connected to port 1).  The controller will see the APs but they will not join the controller.  If we move the SFP back to port 1 the APs will join the controller (at least until it loses network connectivity due to the defective port).

Any wisdom to share on this?

Thanks for everyone's input so far!

Earl

[George Stefanick]

On port 2 .. Do you state "tagging" by putting in a VLAN ID NUMBER. Or are you using 0 = untagged.

__________________________________________________________________________________________
"Satisfaction does not come from knowing the solution, it comes from knowing why." - Rosalind Franklin
__________________________________________________________________________________________
‎"I'm in a serious relationship with my Wi-Fi. You could say we have a connection."

[earlbwilliams]

We are using untagged for both the management and ap manager interfaces.

[George Stefanick]

I assume you are using the same cable from the switch to the WLC port 1 / 2 ?

I assume you configured port 1 and 2 identically ?

On port 2 -- remove the back up and make it 0 and see if there is any change. Also remove teh SFP from port1 when you test this.

__________________________________________________________________________________________
"Satisfaction does not come from knowing the solution, it comes from knowing why." - Rosalind Franklin
__________________________________________________________________________________________
‎"I'm in a serious relationship with my Wi-Fi. You could say we have a connection."

[earlbwilliams]

Yes, ports 1 & 2 are configured exactly the same.  I will try to remove the backup from port 2 to see if that helps.

Thanks,

Earl

[George Stefanick]

When was the last time you bounced this wlc?

__________________________________________________________________________________________
"Satisfaction does not come from knowing the solution, it comes from knowing why." - Rosalind Franklin
__________________________________________________________________________________________
‎"I'm in a serious relationship with my Wi-Fi. You could say we have a connection."