Cisco 9000 Port-Channel Suspended Error Due To Different Speed

[Mano11]

Hi All,

Recently, the Cisco 9000 series switch is displaying logs below and has caused the interface to be suspended.
The issue only affects up-links with SFP-10/25GBase-CSR modules which are patched via Multi-mode OM3 Fiber cabling with distances ranging from 50m to 200m.
We have attempted the following troubleshooting such as replacing the fiber patch cables, sockets on fiber panel, replaced the SFPs both ends, re-patched the line card, disabled the DTP, remove speed no-negotiate from POCH, rebooted the entire switch stack but the issue still persists.

Is there anyone here have faced this kind of issue before or this is related to a Cisco bug?

This doesn't affect other switches with the similar setup and same SFP modules.

Logs:

--------

Aug 6 05:38:14.582: %ETC-5-CANNOT_BUNDLE2: Twe1/1/0/14 is not compatible with Twe2/1/0/14 and will be suspended (speed of Twe1/1/0/14 is 10G, Twe2/1/0/14 is 25G)
Aug 6 05:38:14.719: %ETC-5-CANNOT_BUNDLE2: Twe1/1/0/14 is not compatible with Twe2/1/0/14 and will be suspended (speed of Twe1/1/0/14 is 10G, Twe2/1/0/14 is 25G)
Aug 6 05:38:18.583: %ETC-5-CANNOT_BUNDLE2: Twe1/1/0/14 is not compatible with Twe2/1/0/14 and will be suspended (speed of Twe1/1/0/14 is 10G, Twe2/1/0/14 is 25G)

Cisco_9000_switch#show int description | i Mmbr to Switch1
Twe1/1/0/14 down down Mmbr to Switch1
Twe2/1/0/14 up up Mmbr to Switch1

Cisco_9000_switch#show run int twe1/1/0/14
Building configuration...

Current configuration : 543 bytes
!
interface TwentyFiveGigE1/1/0/14
description Mmbr to Switch1
switchport access vlan 999
switchport trunk native vlan 999
switchport trunk allowed vlan 10,12,13,15-19,21,24,34-36,41-44,47,51,55,58,96
switchport trunk allowed vlan add 102,113,119,136,144,152,200-299,324-332
switchport trunk allowed vlan add 400-921,925-935,938-980,999,1000
logging event trunk-status
logging event bundle-status
load-interval 30
speed 25000
udld port
channel-group 24 mode desirable
service-policy output CORE-10G-40G-EGRESS-POLICY
end

Cisco_9000_switch#show run int twe2/1/0/14
Building configuration...

Current configuration : 543 bytes
!
interface TwentyFiveGigE2/1/0/14
description Mmbr to Switch1
switchport access vlan 999
switchport trunk native vlan 999
switchport trunk allowed vlan 10,12,13,15-19,21,24,34-36,41-44,47,51,55,58,96
switchport trunk allowed vlan add 102,113,119,136,144,152,200-299,324-332
switchport trunk allowed vlan add 400-921,925-935,938-980,999,1000
logging event trunk-status
logging event bundle-status
load-interval 30
speed 25000
udld port
channel-group 24 mode desirable
service-policy output CORE-10G-40G-EGRESS-POLICY
end

Cisco_9000_switch#show run int po24
Building configuration...

Current configuration : 416 bytes
!
interface Port-channel24
description PoCh to Switch1
switchport access vlan 999
switchport trunk native vlan 999
switchport trunk allowed vlan 10,12,13,15-19,21,24,34-36,41-44,47,51,55,58,96
switchport trunk allowed vlan add 102,113,119,136,144,152,200-299,324-332
switchport trunk allowed vlan add 400-921,925-935,938-980,999,1000
load-interval 30
speed nonegotiate
spanning-tree guard root
end

Cisco_9000_switch#show int twe1/1/0/14 capabilities
TwentyFiveGigE1/1/0/14
Model: C9600-LC-48YL
Type: SFP-10/25GBase-CSR
Speed: 10000,25000
Duplex: full
FEC: Auto
Trunk encap. type: 802.1Q
Trunk mode: on,off,desirable,nonegotiate
Channel: yes
Broadcast suppression: percentage(0-100)
Unicast suppression: percentage(0-100)
Multicast suppression: percentage(0-100)
Flowcontrol: rx-(off,on,desired),tx-(off,on,desired)
Fast Start: yes
QoS scheduling: rx-(not configurable on per port basis),
tx-(2p6q3t)
CoS rewrite: yes
ToS rewrite: yes
UDLD: yes
Inline power: no
SPAN: source/destination
PortSecure: yes
Dot1x: yes
Diagnostic Monitoring: yes
Breakout Support: not applicable

Cisco_9000_switch#show int twe2/1/0/14 capabilities
TwentyFiveGigE2/1/0/14
Model: C9600-LC-48YL
Type: SFP-10/25GBase-LR
Speed: 10000,25000
Duplex: full
FEC: Auto
Trunk encap. type: 802.1Q
Trunk mode: on,off,desirable,nonegotiate
Channel: yes
Broadcast suppression: percentage(0-100)
Unicast suppression: percentage(0-100)
Multicast suppression: percentage(0-100)
Flowcontrol: rx-(off,on,desired),tx-(off,on,desired)
Fast Start: yes
QoS scheduling: rx-(not configurable on per port basis),
tx-(2p6q3t)
CoS rewrite: yes
ToS rewrite: yes
UDLD: yes
Inline power: no
SPAN: source/destination
PortSecure: yes
Dot1x: yes
Diagnostic Monitoring: yes
Breakout Support: not applicable

[Elliot Dierksen]

It isn't a bug, it is intentional that interfaces of different speeds cannot be added to a port channel group. My suggestion would be to remove the port channel and let spanning tree do the redundancy between these physical links. See if one of the ports comes up with a different speed, or has interface flaps. You can look up the spec sheets on your optics, but 200m seems like that might be too long of a run for multi-mode fiber at the higher speeds.

Edit: The spec sheet on those LR optics only references single mode fiber. Cisco 25GBASE SFP28 Modules Data Sheet You may have to use different optics or different cable plant.

[Mano11]

Thanks Elliot, but both interfaces are set to speed 25000 so it shouldn't receive the logs alarms unless the speeds value are different.

Do you think both interfaces need to use the same type of SFP modules. That's the only difference i could see. I have a working port-channel interfaces using the different SFP modules.

[Elliot Dierksen]

They may be set to that, but perhaps aren't able to negotiate that speed. Try temporarily breaking the port channel so you can get better information. On both sides, do a "no channel-group 24 " on each of the physical interfaces. Then watch the logs to see if you have up/down events. Also do "show interface" on the physical interfaces to see the speed at which they are operating.

[Mano11]

Thanks for the suggestion Elliot, getting the customer for approval in order to break POCH and monitor further.