Solved: Re: Packet loss after adding interface to port-channel

robert · ‎09-12-2025

Hi

C9500 and C9300 running 17.12.5

I have a working port-channel between the 2 switches with 1 active interface.
When i add and enable a second interface the channel goes down and shortly after up, but then i have extremly high packet loss and loose management connection to the "other" switch.

Below is my config. I have looked at "sh interfaces counters errors" and "sh interfaces transceiver detail" and there is no errors or high values on the fiber sfp ports.

The log does not show anyting except the interface goes up and down, when i enable/disable it.

The interface which causes this is twe1/0/30 on SW1 and twe1/1/4 on SW2.

Any good ideas?

SW1
interface Port-channel33
switchport mode trunk

interface TwentyFiveGigE1/0/30
switchport mode trunk
channel-group 33 mode active

interface TwentyFiveGigE2/0/5
switchport mode trunk
channel-group 33 mode active

SW2
interface Port-channel3
description UPLINKS
switchport mode trunk
storm-control broadcast level 10.00 5.00
storm-control multicast level 10.00 5.00
ip dhcp snooping trust

interface TwentyFiveGigE1/1/1
switchport mode trunk
storm-control broadcast level 10.00 5.00
storm-control multicast level 10.00 5.00
channel-group 3 mode active
ip dhcp snooping trust

interface TwentyFiveGigE1/1/4
switchport mode trunk
storm-control broadcast level 10.00 5.00
storm-control multicast level 10.00 5.00
channel-group 3 mode active
ip dhcp snooping trust

*Sep 12 17:20:41.174: %LINK-5-UPDOWN: Interface TwentyFiveGigE1/0/30, changed state to up
*Sep 12 17:20:45.284: %LINEPROTO-5-UPDOWN: Line protocol on Interface TwentyFiveGigE1/0/30, changed state to up
*Sep 12 17:21:06.634: %LINK-5-CHANGED: Interface TwentyFiveGigE1/0/30, changed state to administratively down
*Sep 12 17:21:07.634: %LINEPROTO-5-UPDOWN: Line protocol on Interface TwentyFiveGigE1/0/30, changed state to down

096427: Sep 12 15:23:37.459: %LINK-3-UPDOWN: Interface TwentyFiveGigE1/1/4, changed state to up
096428: Sep 12 15:23:41.523: %LINEPROTO-5-UPDOWN: Line protocol on Interface TwentyFiveGigE1/1/4, changed state to up
096429: Sep 12 15:24:01.921: %LINEPROTO-5-UPDOWN: Line protocol on Interface TwentyFiveGigE1/1/4, changed state to down
096430: Sep 12 15:24:02.927: %LINK-3-UPDOWN: Interface TwentyFiveGigE1/1/4, changed state to down

Regards
Robert

balaji.bandi · ‎09-13-2025

i would suggest some tips here for best practice.

1. Always only allow required VLAN in the port-channel to make it work as expected (both the side), make sure spanning tree will be configured where you need root bridge for VLAN.

2. if this is trusted port avoid using storm-control - this may trigger different issue.

3. what is cause of interface go down, you can see error disable reason.

4. Do all the configuration global configuration before you make the port-channel enable(before hand) - later you can add any interface to port-channel is easy and no other side effects take place.

on PO 3 you see lot of input errors :

268115 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

BB

***** Rate All Helpful Responses *****

How to Ask The Cisco Community for Help

View solution in original post

robert · ‎09-17-2025

I ended up replacing the fiber patch cable on port twe1/0/30 and now it works.

View solution in original post

Mark Elsen · ‎09-12-2025

- @robert Any reason for the different port channel numbers on the switches ( 3 versus 33) ?
Use show etherchannel summary and show lacp neighbor on both switches.
Confirm that both interfaces are bundling correctly and that LACP sees the same system ID and port-channel.

The storm control configuration is also asymmetric (SW1 does not have it) ; for starters I would remove it on both
and try again,

M.

-- Let everything happen to you
   Beauty and terror
      Just keep going
     No feeling is final
Reiner Maria Rilke (1899)

robert · ‎09-13-2025

Channel group 33 neighbors
LACP port Admin Oper Port Port
Port Flags Priority Dev ID Age key Key Number State
Twe1/0/30 SP 32768 20db.ea50.9d80 493s 0x0 0x3 0x135 0x34
Twe2/0/5 SA 32768 20db.ea50.9d80 18s 0x0 0x3 0x132 0x3D

Number of channel-groups in use: 38
Number of aggregators: 38

Group Port-channel Protocol Ports
------+-------------+-----------+-----------------------------------------------
10 Po10(SU) LACP Twe1/0/19(P) Twe2/0/44(P)
11 Po11(SU) LACP Twe1/0/1(P) Twe2/0/26(P)
12 Po12(SU) LACP Twe1/0/2(P) Twe2/0/25(P)
13 Po13(SU) LACP Twe1/0/26(D) Twe2/0/1(P)
14 Po14(SU) LACP Twe1/0/25(D) Twe2/0/2(P)
15 Po15(SD) -
21 Po21(SU) LACP Twe1/0/3(P) Twe2/0/28(P)
22 Po22(SU) LACP Twe1/0/4(D) Twe2/0/27(P)
23 Po23(SU) LACP Twe1/0/28(D) Twe2/0/3(P)
24 Po24(SU) LACP Twe1/0/27(s) Twe2/0/4(P)
30 Po30(SD) -
31 Po31(SU) LACP Twe1/0/5(P) Twe2/0/30(P)
32 Po32(SU) LACP Twe1/0/6(P) Twe2/0/29(P)
33 Po33(SU) LACP Twe1/0/30(D) Twe2/0/5(P)
34 Po34(SU) LACP Twe1/0/29(P) Twe2/0/6(P)
35 Po35(SU) LACP Twe1/0/15(D) Twe2/0/40(P)
41 Po41(SD) LACP Twe1/0/7(D) Twe2/0/32(D)
42 Po42(SD) LACP Twe1/0/8(D) Twe2/0/31(D)
43 Po43(SD) LACP Twe1/0/32(D) Twe2/0/7(D)
44 Po44(SD) LACP Twe1/0/31(D) Twe2/0/8(D)
51 Po51(SU) LACP Twe1/0/9(P) Twe2/0/34(P)
52 Po52(SU) LACP Twe1/0/10(D) Twe2/0/33(P)
53 Po53(SU) LACP Twe1/0/34(P) Twe2/0/9(P)
54 Po54(SD) LACP Twe1/0/33(D) Twe2/0/10(D)
61 Po61(SU) LACP Twe1/0/11(P) Twe2/0/36(P)
62 Po62(SU) LACP Twe1/0/12(P) Twe2/0/35(s)
63 Po63(SU) LACP Twe1/0/36(D) Twe2/0/11(P)
64 Po64(SD) LACP Twe1/0/35(D) Twe2/0/12(D)
71 Po71(SU) LACP Twe1/0/13(P) Twe2/0/38(P)
72 Po72(SU) LACP Twe1/0/14(P) Twe2/0/37(P)
73 Po73(SD) LACP Twe1/0/38(D) Twe2/0/13(D)
74 Po74(SD) LACP Twe1/0/37(D) Twe2/0/14(D)
90 Po90(SU) LACP Twe1/0/23(P) Twe2/0/24(P)
91 Po91(SD) -
100 Po100(SU) LACP Twe1/0/47(P) Twe2/0/47(P)
109 Po109(SD) LACP Twe1/0/46(D)
110 Po110(SU) LACP Twe1/0/45(P)
111 Po111(SU) LACP Twe2/0/45(P)

Channel group 3 neighbors
LACP port Admin Oper Port Port
Port Flags Priority Dev ID Age key Key Number State
Twe1/1/1 SA 32768 20db.ea9f.6fa0 19s 0x0 0x21 0x506 0x3D
Twe1/1/4 SP 32768 20db.ea9f.6fa0 562s 0x0 0x21 0x31F 0x34

Number of channel-groups in use: 2
Number of aggregators: 2

Group Port-channel Protocol Ports
------+-------------+-----------+-----------------------------------------------
1 Po1(SD) -
3 Po3(SU) LACP Twe1/1/1(P) Twe1/1/4(D)

Torbjørn · ‎09-12-2025

I think I would try to configure UDLD to see if it triggers on this.

Can you post the output of show interface from both sides as well?

Happy to help! Please mark as helpful/solution if applicable.
Get in touch: https://torbjorn.dev

robert · ‎09-13-2025

sh int po3
Port-channel3 is up, line protocol is up (connected)
Hardware is EtherChannel, address is 20db.ea50.9db1 (bia 20db.ea50.9db1)
Description: UPLINKS
MTU 1500 bytes, BW 10000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 10Gb/s, link type is auto, media type is N/A
input flow-control is on, output flow-control is unsupported
Members in this channel: Twe1/1/1
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:13:56, output hang never
Last clearing of "show interface" counters never
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 292000 bits/sec, 337 packets/sec
5 minute output rate 311000 bits/sec, 365 packets/sec
368504895 packets input, 93195908455 bytes, 0 no buffer
Received 159862045 broadcasts (22051891 multicasts)
0 runts, 0 giants, 0 throttles
268115 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 22051891 multicast, 0 pause input
0 input packets with dribble condition detected
153140059 packets output, 26328392741 bytes, 0 underruns
Output 6920806 broadcasts (379064 multicasts)
0 output errors, 0 collisions, 1 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

#sh interfaces po33
Port-channel33 is up, line protocol is up (connected)
Hardware is EtherChannel, address is 20db.ea9f.6fa5 (bia 20db.ea9f.6fa5)
Description: ** H3.X3 **
MTU 1500 bytes, BW 10000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 10Gb/s, link type is auto, media type is N/A
input flow-control is on, output flow-control is unsupported
Members in this channel: Twe2/0/5
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:16:41, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 170
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 316000 bits/sec, 355 packets/sec
5 minute output rate 295000 bits/sec, 339 packets/sec
86761569 packets input, 13958034425 bytes, 0 no buffer
Received 450629 broadcasts (143607 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 143607 multicast, 0 pause input
0 input packets with dribble condition detected
133551500 packets output, 27050732408 bytes, 0 underruns
Output 33798102 broadcasts (6969784 multicasts)
0 output errors, 0 collisions, 3 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

robert · ‎09-13-2025

enable udld on the int two1/0/30 and twe2/0/5 on Po33 did not show any uni links when i have the twe1/0/30 interface enabled.

Total number of bidirectional entries displayed: 2
TH-DK-MID2-DIST-01#sh udld neighbors
Port Device Name Device ID Port ID Neighbor State
---- ----------- --------- ------- --------------
Twe1/0/30 20DBEA509D80 1 Twe1/1/4 Bidirectional
Twe2/0/5 20DBEA509D80 1 Twe1/1/1 Bidirectional

MHM Cisco World · ‎09-12-2025

Normal' stp detect loop and down link before lacp add new port to PO

What you need is only shut/no shut PO

And all will be OK.

MHM

robert · ‎09-13-2025

Did a shut on Po33, enabled twe1/0/30 and did a no shut on Po33.

Same problem, extreme packet loss.

MHM Cisco World · ‎09-13-2025

debug lacp [event|packet|fsm|misc]

debug condition <condition>

Run above debug

Note:- use conditions to only include interface (port member not port channel) have issue

Share debug here

Note:- run debug in both SW

MHM

balaji.bandi · ‎09-13-2025

i would suggest some tips here for best practice.

1. Always only allow required VLAN in the port-channel to make it work as expected (both the side), make sure spanning tree will be configured where you need root bridge for VLAN.

2. if this is trusted port avoid using storm-control - this may trigger different issue.

3. what is cause of interface go down, you can see error disable reason.

4. Do all the configuration global configuration before you make the port-channel enable(before hand) - later you can add any interface to port-channel is easy and no other side effects take place.

on PO 3 you see lot of input errors :

268115 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

BB

***** Rate All Helpful Responses *****

How to Ask The Cisco Community for Help

robert · ‎09-17-2025

I ended up replacing the fiber patch cable on port twe1/0/30 and now it works.

MHM Cisco World · ‎09-17-2025

Change fiber meaning there was uni directional' but your udld show bidirectional.

What meaning of uni' one side send lacp other not receive or not send lacp.

Anyway glad issue solved

MHM

balaji.bandi · ‎09-17-2025

268115 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

this infact give you clue that cable gone wrong.

BB

***** Rate All Helpful Responses *****

How to Ask The Cisco Community for Help

robert · ‎09-17-2025

Thank you