Issue seems to be related to Cisco Bug: CSCsg35506 - EtherCahnnel flaping because of a port

[Jan Mierczi]

Hello,

Not long ago we got an incident. One of important WLAN controllers was down because of EtherChannel flapping. Turned out that one port caused inconsistency whole aggregated link. After disabling a faulty port issue disappeared.

• Switch: WS-C6509-E,
• Supervisor: WS-SUP720-3B
• IOS: s72033-ipservicesk9-mz.122-33.SXJ10.bin
• ROMMON: 12.2(18r)S1

There is a switch which connects via EtherChannel (4 links)  WLAN controller.

SW ------  EtherChannel 6 -------- WLAN controler.

EtherChannel contain ports on linecard 2 and 4.

Group  Port-channel  Protocol    Ports
------+-------------+-----------+-----------------------------------------------
6      Po6(SU)          -        Gi2/43(P)      Gi2/44(P)      Gi4/43(D)
                                 Gi4/44(P)

Once next logs appeared:

switch#sh logging | i Gi2/43|Gi2/44|Gi4/43|Gi4/44
000788: Aug 22 09:15:13: %EC-SP-5-COMPATIBLE: Gi2/43 is compatible with port-channel members
000794: Aug 22 09:15:13: %DTP-SP-5-NONTRUNKPORTON: Port Gi2/43 has become non-trunk
000797: Aug 22 09:15:17: %DTP-SP-5-TRUNKPORTON: Port Gi2/43 has become dot1q trunk
000801: Aug 22 09:15:50: %EC-SP-5-COMPATIBLE: Gi2/44 is compatible with port-channel members
000807: Aug 22 09:15:51: %DTP-SP-5-NONTRUNKPORTON: Port Gi2/44 has become non-trunk
000810: Aug 22 09:15:55: %DTP-SP-5-TRUNKPORTON: Port Gi2/44 has become dot1q trunk
000814: Aug 22 09:16:03: %EC-SP-5-COMPATIBLE: Gi4/44 is compatible with port-channel members
000820: Aug 22 09:16:04: %DTP-SP-5-NONTRUNKPORTON: Port Gi4/44 has become non-trunk
000823: Aug 22 09:16:07: %DTP-SP-5-TRUNKPORTON: Port Gi4/44 has become dot1q trunk
000835: Aug 22 09:19:49: %DTP-SP-5-NONTRUNKPORTON: Port Gi4/43 has become non-trunk
000838: Aug 22 09:19:52: %DTP-SP-5-TRUNKPORTON: Port Gi4/43 has become dot1q trunk
000842: Aug 22 09:21:08: %EC-SP-5-CANNOT_BUNDLE2: Gi4/43 is not compatible with Gi2/43 and will be suspended (flow control send of Gi4/43 is desired, Gi2/43 is off)
000847: Aug 22 09:21:09: %DTP-SP-5-NONTRUNKPORTON: Port Gi4/43 has become non-trunk
000850: Aug 22 09:21:12: %EC-SP-5-COMPATIBLE: Gi4/43 is compatible with port-channel members
000852: Aug 22 09:21:12: %DTP-SP-5-TRUNKPORTON: Port Gi4/43 has become dot1q trunk
000855: Aug 22 09:21:40: %EC-SP-5-CANNOT_BUNDLE2: Gi2/43 is not compatible with Gi2/44 and will be suspended (flow control send of Gi2/43 is desired, Gi2/44 is off)
000860: Aug 22 09:21:41: %DTP-SP-5-NONTRUNKPORTON: Port Gi2/43 has become non-trunk
000861: Aug 22 09:21:44: %EC-SP-5-CANNOT_BUNDLE2: Gi2/43 is not compatible with Gi2/44 and will be suspended (flow control send of Gi2/43 is on, Gi2/44 is off)
000862: Aug 22 09:21:44: %DTP-SP-5-TRUNKPORTON: Port Gi2/43 has become dot1q trunk
000864: Aug 22 09:21:45: %EC-SP-5-CANNOT_BUNDLE2: Gi2/44 is not compatible with Gi4/43 and will be suspended (flow control send of Gi2/44 is desired, Gi4/43 is off)
000870: Aug 22 09:21:45: %DTP-SP-5-NONTRUNKPORTON: Port Gi2/44 has become non-trunk
000871: Aug 22 09:21:48: %EC-SP-5-CANNOT_BUNDLE2: Gi2/44 is not compatible with Gi4/43 and will be suspended (flow control send of Gi2/44 is on, Gi4/43 is off)
000872: Aug 22 09:21:49: %DTP-SP-5-TRUNKPORTON: Port Gi2/44 has become dot1q trunk
000874: Aug 22 09:21:52: %EC-SP-5-CANNOT_BUNDLE2: Gi4/44 is not compatible with Gi4/43 and will be suspended (flow control send of Gi4/44 is desired, Gi4/43 is off)
000878: Aug 22 09:21:53: %DTP-SP-5-NONTRUNKPORTON: Port Gi4/44 has become non-trunk
000882: Aug 22 09:21:56: %EC-SP-5-CANNOT_BUNDLE2: Gi4/44 is not compatible with Gi4/43 and will be suspended (flow control send of Gi4/44 is on, Gi4/43 is off)
000883: Aug 22 09:21:56: %DTP-SP-5-TRUNKPORTON: Port Gi4/44 has become dot1q trunk
000892: Aug 22 09:25:18: %EC-SP-5-COMPATIBLE: Gi2/43 is compatible with port-channel members
000893: Aug 22 09:25:18: %EC-SP-5-COMPATIBLE: Gi2/44 is compatible with port-channel members
000894: Aug 22 09:25:18: %EC-SP-5-COMPATIBLE: Gi4/44 is compatible with port-channel members
000916: Aug 22 09:25:19: %DTP-SP-5-NONTRUNKPORTON: Port Gi2/43,Gi2/44,Gi4/43,Gi4/44 has become non-trunk
000935: Aug 22 09:25:23: %DTP-SP-5-TRUNKPORTON: Port Gi2/43,Gi2/44,Gi4/43,Gi4/44 has become dot1q trunk
000965: Aug 22 09:27:48: %DTP-SP-5-NONTRUNKPORTON: Port Gi2/43,Gi2/44,Gi4/43,Gi4/44 has become non-trunk
000972: Aug 22 09:27:51: %EC-SP-5-CANNOT_BUNDLE2: Gi4/44 is not compatible with Gi4/43 and will be suspended (flow control send of Gi4/44 is on, Gi4/43 is off)
000975: Aug 22 09:27:51: %EC-SP-5-CANNOT_BUNDLE2: Gi2/43 is not compatible with Gi4/43 and will be suspended (flow control send of Gi2/43 is on, Gi4/43 is off)
000976: Aug 22 09:27:51: %EC-SP-5-CANNOT_BUNDLE2: Gi2/44 is not compatible with Gi4/43 and will be suspended (flow control send of Gi2/44 is on, Gi4/43 is off)
000978: Aug 22 09:27:52: %DTP-SP-5-TRUNKPORTON: Port Gi2/43,Gi2/44,Gi4/43,Gi4/44 has become dot1q trunk
000989: Aug 22 09:31:28: %EC-SP-5-COMPATIBLE: Gi2/43 is compatible with port-channel members
000990: Aug 22 09:31:28: %EC-SP-5-COMPATIBLE: Gi2/44 is compatible with port-channel members
000991: Aug 22 09:31:28: %EC-SP-5-COMPATIBLE: Gi4/44 is compatible with port-channel members
000997: Aug 22 09:31:29: %DTP-SP-5-NONTRUNKPORTON: Port Gi4/43 has become non-trunk

After disabling port Gi4/43 issue has resolved. We found out about the bug:

Cisco Bug: CSCsg35506 - JQL: port-channel member in suspend due to flowcontrol mismatch '

My question is how I can troubleshoot it and fix the issue? At the moment the 4th link is down and I am not sure if I will enable it whole EtherChannel will work correctly. Is it caused by HW problems, SW problems? This aggregated link must contain at least 4 links according to design.

[Leo Laohoo]

Post the complete output to the following commands:

1.  sh interface status err; 

2.  sh run int Gi2/43;

3.  sh run int Gi2/44;

4.  sh run int Gi4/43; and

5.  sh run int Gi4/44

[Jan Mierczi]

switch#sh int status err-disabled

switch#sh run int Gi2/43
Building configuration...

Current configuration : 446 bytes
!
interface GigabitEthernet2/43
 description XXXXX
 switchport
 switchport trunk encapsulation dot1q
 switchport trunk allowed vlan 1,2,127,129,130,202,443,445,611,840,891
 switchport mode trunk
 logging event trunk-status
 wrr-queue bandwidth 30 30 40
 wrr-queue cos-map 1 2 0 1
 wrr-queue cos-map 2 2 2 4
 wrr-queue cos-map 3 2 3 6 7
 mls qos trust dscp
 no lldp transmit
 no lldp receive
 channel-group 6 mode on
end

switch#sh run int Gi2/44
Building configuration...

Current configuration : 446 bytes
!
interface GigabitEthernet2/44
 description XXXXX
 switchport
 switchport trunk encapsulation dot1q
 switchport trunk allowed vlan 1,2,127,129,130,202,443,445,611,840,891
 switchport mode trunk
 logging event trunk-status
 wrr-queue bandwidth 30 30 40
 wrr-queue cos-map 1 2 0 1
 wrr-queue cos-map 2 2 2 4
 wrr-queue cos-map 3 2 3 6 7
 mls qos trust dscp
 no lldp transmit
 no lldp receive
 channel-group 6 mode on
end

switch#sh run int Gi4/43
Building configuration...

Current configuration : 456 bytes
!
interface GigabitEthernet4/43
 description XXXXX
 switchport
 switchport trunk encapsulation dot1q
 switchport trunk allowed vlan 1,2,127,129,130,202,443,445,611,840,891
 switchport mode trunk
 logging event trunk-status
 shutdown
 wrr-queue bandwidth 30 30 40
 wrr-queue cos-map 1 2 0 1
 wrr-queue cos-map 2 2 2 4
 wrr-queue cos-map 3 2 3 6 7
 mls qos trust dscp
 no lldp transmit
 no lldp receive
 channel-group 6 mode on
end

switch#sh run int Gi4/44
Building configuration...

Current configuration : 446 bytes
!
interface GigabitEthernet4/44
 description XXXXX
 switchport
 switchport trunk encapsulation dot1q
 switchport trunk allowed vlan 1,2,127,129,130,202,443,445,611,840,891
 switchport mode trunk
 logging event trunk-status
 wrr-queue bandwidth 30 30 40
 wrr-queue cos-map 1 2 0 1
 wrr-queue cos-map 2 2 2 4
 wrr-queue cos-map 3 2 3 6 7
 mls qos trust dscp
 no lldp transmit
 no lldp receive
 channel-group 6 mode on
end

[Leo Laohoo]

(flow control send of Gi2/44 is on, Gi4/43 is off)

I think I know what this is and I've seen this before.  

Particularly with 6500, when you think you've removed a line of configuration but the supervisor card still think it's there. 

Try this: 

conf t

default interface G4/43

interface G4/43

<re-enter the configurations again>

[Jan Mierczi]

Hi,

Thank you for your answer. :) But could you explain me please verbose what do you mean?

The flow control never has not been configured on all interfaces. I need just to understand: what, why etc.

Best Regards