Solved: Link flapping, C3750<->WLC5500

henrik-stryhn · ‎12-27-2010

I've got a simple setup with a C3750 connected with a WLC5500 on two fiberports.

Configuration:

interface GigabitEthernet1/0/1
switchport trunk encapsulation dot1q
switchport trunk native vlan 10
switchport mode trunk
speed nonegotiate
channel-group 1 mode on
end
Interface GigabitEthernet1/0/2
switchport trunk encapsulation dot1q
switchport trunk native vlan 10
switchport mode trunk
speed nonegotiate
channel-group 1 mode on
end

Ports on WLC are configured with LAG on.

Both ports on C3750 keeps changing states, 10-100 down's/up's in an hour.

I've tried to disable each link separetly, disabling channel-group, etc., nothing works.

I've upgraded the software to 12.2(52)SE. I've debugged etherchannel, link states, changed fibercabling, etc., nothing shows any errors.

The problem persists on both interfaces, no matter which interfaces are disabled, etc.

Log:

001124: Dec 27 13:48:12.257 GMT+1: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/1, changed state to down
001125: Dec 27 13:48:15.352 GMT+1: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/1, changed state to up
001126: Dec 27 13:48:26.954 GMT+1: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/1, changed state to down
001127: Dec 27 13:48:30.049 GMT+1: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/1, changed state to up
001128: Dec 27 13:48:43.664 GMT+1: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/1, changed state to down
001129: Dec 27 13:48:46.768 GMT+1: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/1, changed state to up
001130: Dec 27 13:49:03.680 GMT+1: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/2, changed state to down
001131: Dec 27 13:49:06.775 GMT+1: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/2, changed state to up
001132: Dec 27 13:49:33.912 GMT+1: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/1, changed state to down
001133: Dec 27 13:49:36.999 GMT+1: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/1, changed state to up
001134: Dec 27 13:50:36.282 GMT+1: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/2, changed state to down
001135: Dec 27 13:50:39.378 GMT+1: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/2, changed state to up

Any good ideas?

Peter Paluch · ‎12-28-2010

Henrik,

I believe that you have tried everything what we would have tried also. I guess the only option left is that indeed one of these two devices is faulty, or are they simply unable to talk to each other.

If you connected a different device (not a WLC) to the Gi1/0/1 and Gi1/0/2 of your switch, would the connection be stable? Also, if you connected the WLC to a physically different switch, would the problems reappear?

Best regards,

Peter

View solution in original post

blacktrack · ‎12-27-2010

Hi,

Just Type No keepalive on each interface.

regards

Hicham Azarou

henrik-stryhn · ‎12-27-2010

That didn't solve the problem.

001695: Dec 27 19:48:19.440 GMT+1: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/1, changed state to down
001696: Dec 27 19:48:22.552 GMT+1: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/1, changed state to up
001697: Dec 27 19:48:46.074 GMT+1: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/1, changed state to down
001698: Dec 27 19:48:49.186 GMT+1: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/1, changed state to up
001699: Dec 27 19:49:55.717 GMT+1: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/1, changed state to down
001700: Dec 27 19:49:58.829 GMT+1: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/1, changed state to up
001701: Dec 27 19:51:03.447 GMT+1: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/2, changed state to down
001702: Dec 27 19:51:06.560 GMT+1: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/2, changed state to up

Leo Laohoo · ‎12-27-2010

Disable one of the links and see if this still occurs.

henrik-stryhn · ‎12-27-2010

Already tried it.

Disabling channel-group. Links still flap.

Disabling Gi1/0/1. Gi1/0/2 are still flapping.

Disabling Gi1/0/2. Gi1/0/1 are still flapping.

It's a mystery to me. No log-messages appear, nothing show up on debug, everything passes on diagnostics.

Peter Paluch · ‎12-28-2010

Henrik,

Are the flaps also somehow logged on the WLC device? Does the WLC have its own fiber ports, or is there any media converter being used? Does the show interfaces show any errors or suspicious counter values on Gi1/0/1 and Gi1/0/2 interfaces?

EDIT: One more question: does the WLC use any EtherChannel signalling protocol to negotiate the creation of the bundle, i.e. LACP or PAgP? Does it at least support it?

Best regards,

Peter

henrik-stryhn · ‎12-28-2010

Gi1/0/1:

649 packets input, 280161 bytes, 0 no buffer
     Received 149 broadcasts (146 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 146 multicast, 0 pause input
     0 input packets with dribble condition detected
     3957155 packets output, 372142185 bytes, 0 underruns
     0 output errors, 0 collisions, 12 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out

Gi1/0/2:

13610 packets input, 5443240 bytes, 0 no buffer
     Received 189 broadcasts (151 multicasts)
     0 runts, 0 giants, 0 throttles
     6 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 151 multicast, 0 pause input
     0 input packets with dribble condition detected
     7217318 packets output, 672173930 bytes, 0 underruns
     0 output errors, 0 collisions, 11 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out

Ports on WLC shows 0's in all error types.

The flapping shows on the WLC as:

*Dec 28 11:25:10.275: %SIM-3-PORT_UP: sim.c:9547 Physical port 1 is up!.
*Dec 28 11:25:06.085: %SIM-3-PORT_DOWN: sim.c:9550 Physical port 1 is down!.

WLC:

>show port summary

1 Normal Forw Enable Auto 1000 Full Up Enable N/A 1000BaseSX
2 Normal Forw Enable Auto 1000 Full Up Enable N/A 1000BaseSX

>show lag summarty

LAG Enabled

You can nothing more than enable/disable LAG on the WLC, which is done. And the C3750 interfaces are combined in an etherchannel 'channel-group 1 mode on'.

Switch and WLC are directly connected via two 5m MM fiber cables with correct SFPs.

blacktrack · ‎12-28-2010

Try this :

1st : replace the speed nonegotiate by speed auto

if nothing change :

2nd : type this command on both interfaces : udld port aggressive

if nothing change :

3rd : replace your gbic or fiber.

regards

Hicham Azarou

Peter Paluch · ‎12-28-2010

Hicham,

I agree with your first suggestion but I think that introducing the UDLD would only complicate things further, and I do not see any compelling point to activate it here. Besides, I think that the WLC does not support UDLD at all.

What I would suggest - but it is only trying to exclude the possibility of a faulty SFP slot - to try using different slots. And one more idea comes to my mind: whether the output power of the SFPs in use is not too strong for a 5m MMF patch cable used here. In such case, the receiver could be overboosted.

Best regards,

Peter

henrik-stryhn · ‎12-28-2010

No, WLC doesn't support UDLD.

I've tried different slots, different SFPs, different cables, no luck and absolutely nothing in the log, besides Up/Down.

As far as I remember, 1000BaseSX supports UP TO about 220 metres which MM cables. Does there exist "weaker" modules? We've got other similar solutions, where the error doesn't exist on such short distances.

If it where the cables/SFPs or anything like that, what would be the odds of having the same error on both interfaces?

henrik-stryhn · ‎12-28-2010

Speed set to auto.

UDLD shows nothing, ports still flaps and no errdisable events. Debugging udld shows nothing strange.

Cables and SFPs have been replaced in C3750 and the WLC, the flapping still occur.

Is this some secret undiscovered bug somewhere?!

blacktrack · ‎12-28-2010

Hi,

Did you try without the etherchannel?

regards

Hicham Azarou

henrik-stryhn · ‎12-28-2010

I've tried with:

Gi1/0/1 to WLC, with and without etherchannel.

Gi1/0/2 shutdown

Gi1/0/1 shutdown

Gi1/0/2 to WLC, with and without etherchannel

Gi1/0/1 to WLC, with and without etherchannel

Gi1/0/2 to WLC, with and without etherchannel

Al possible combinations, just to see if it triggers anything. No matter which scenario is "active", the enabled/active ports just go up, down, up down, up down, etc.

Peter Paluch · ‎12-28-2010

Henrik,

I believe that you have tried everything what we would have tried also. I guess the only option left is that indeed one of these two devices is faulty, or are they simply unable to talk to each other.

If you connected a different device (not a WLC) to the Gi1/0/1 and Gi1/0/2 of your switch, would the connection be stable? Also, if you connected the WLC to a physically different switch, would the problems reappear?

Best regards,

Peter

Leo Laohoo · ‎12-28-2010

1. Disable etherchannel and LAG and disable one of the ports.

2. If issues still occur, "test" by moving the connection to another switch (or stack member).

3. If issues still occur, move the ports on the 5508 (5508 has 8 SFP ports).

4. If it still occurs, RMA the 5508.