Cisco 3504 Wireless controller goes into maintenance mode

[Mikey John]

Hi Folks,

 I have 2 x Cisco 3504 Wireless controllers connected to switches. There are currently two ports active on the WLCs towards the switches (both on Active and Standby WLC). These ports are part of a LAG, and there is port channel configured on the switch end as well. These two ports seem to be functioning well.

But, when I try and enable the remaining two ports on the WLCs (Active & standby) towards the switch, and bring them into the port channel, I see the standby WLC loses peer connectivity, and goes into maintenance mode. 

(XXXX) >show redundancy summary

            Redundancy Mode = SSO ENABLED

                Local State = MAINTENANCE

                 Peer State = UNKNOWN - Communication Down

                       Unit = Primary

                    Unit ID = 70:18:A7:C9:3F:10

           Redundancy State = Non Redundant

               Mobility MAC = 70:18:A7:C9:3F:10

               Redundancy Port  = UP

               Link Encryption = ENABLED

Maintenance Mode = Enabled

Maintenance cause= Default Gateway not reachable

Appreciate your input on this issue.

Cheers

Mikey

[Scott Fella]

Check your configurations on the switch. Seems like the ether channel might be the issue. Adding or removing port connected to switches should not affect anything. Check the state of the port channel when you connect up the port.

[Mikey John]

Hi Scott,

 The port channel looks ok. I can see those two ports being added to port channel, and it looks all fine to me. Still the same issue.

Sw1:

Members in this channel: Gi 0/27(U) Gi 0/28(U) Gi 0/29(U) Gi 0/30(U)

Sw2:

Members in this channel: Gi 0/27(U) Gi 0/28(U) Gi 0/29(U) Gi 0/30(U)

Cheers

Mikey

[Scott Fella]

I don’t know… I have done that on a few models without any issues. I’m assuming you are using the correct ports to bundle not the mGig.

[Mikey John]

Yep, I have used only the first 4 ports on both the WLCs.

(XXXX) >show port summary

STP Admin Physical Physical Link Link
Pr Type Stat Mode Mode Status Status Trap POE
-- ------- ---- ------- ---------- ---------- ------ ------- ---------
1 Normal Forw Enable Auto 1000 Full Up Enable N/A
2 Normal Forw Enable Auto 1000 Full Up Enable N/A
3 Normal Forw Enable Auto 1000 Full Up Enable N/A
4 Normal Forw Enable Auto 1000 Full Up Enable N/A
5 Normal Disa Enable Auto Auto Down Enable N/A
RP Normal Forw Enable Auto Auto Up Enable N/A
SP Normal Disa Enable Auto Auto Down Enable N/A

(XXXX-Standby) >show port summary

STP Admin Physical Physical Link Link
Pr Type Stat Mode Mode Status Status Trap POE
-- ------- ---- ------- ---------- ---------- ------ ------- ---------
1 Normal Forw Enable Auto 1000 Full Up Enable N/A
2 Normal Forw Enable Auto 1000 Full Up Enable N/A
3 Normal Forw Enable Auto 1000 Full Up Enable N/A
4 Normal Forw Enable Auto 1000 Full Up Enable N/A
5 Normal Disa Enable Auto Auto Down Enable N/A
RP Normal Forw Enable Auto Auto Up Enable N/A
SP Normal Disa Enable Auto Auto Down Enable N/A

Thanks

Mikey

[patoberli]

You don't have them attached to a Nexus, right? I once had really weird issues because of that (wlc crashed and reload loop).

[Mikey John]

Nope, these WLCs are connected to Dell C150 series switches.

Cheers

Mikey

[Scott Fella]

Make sure that the load balancing is src-dst-ip. That is what the controller uses

[patoberli]

I know they are Dell switches, but could you check again the port configuration? Maybe there is some difference in the individual port configuration, which is overriding the LAG. I doubt this though and suggest opening a TAC.

You did once try to reboot the standby by hand?

[Leo Laohoo]

Redundancy Port is "UP".
What firmware are both units running on? Maybe they are running different versions.

[Mikey John]

Hi Leo,

If they were on different firmware versions, they wouldn't have been on active/standby role with the first two LAN ports active. right?

The WLC goes into maintenance mode only when I enable the remaining two LAN ports towards the switches.

Both the WLCs are running the same firmware version. See output below.

Active WLC
===========

(XXXX) >show sysinfo

Manufacturer's Name.............................. Cisco Systems Inc.
Product Name..................................... Cisco Controller
Product Version.................................. 8.8.120.0
RTOS Version..................................... 8.8.120.0
Bootloader Version............................... 8.5.103.0
Emergency Image Version.......................... 8.5.103.0

OUI File Last Update Time........................ N/A
Build Type....................................... DATA + WPS

Standby WLC
===========

(XXXX) >show sysinfo

Manufacturer's Name.............................. Cisco Systems Inc.
Product Name..................................... Cisco Controller
Product Version.................................. 8.8.120.0
RTOS Version..................................... 8.8.120.0
Bootloader Version............................... 8.5.103.0
Emergency Image Version.......................... 8.5.103.0

OUI File Last Update Time........................ N/A
Build Type....................................... DATA + WPS

Cheers

Mikey

[patoberli]

Besides the problem, do you really hit more than 1.5 Gbps in one direction on the WLC, or why do you want to add the other two ports?
Just asking, because here with ~3000 simultaneous users, I rarely have more than 1 Gbps of throughput (spread over > 150 APs).