SG350 issues with LACP ports going randomly down/up for no reason

[Nico.]

Hello,

I have a really big issue with port channel members going down for no reason and coming back up again. When this happens to both ports, the po goes down offcourse (2 members per port channel).
Problem is this happens to 3 of the 4 SG350x series switches installed like this atm, so not isolated to one or two switches. (all on same firmware version)

Switches configured in star configuration with LACP uplink on 2 ports.
Situation: SG350X-48MP uplink to Cisco WS-C4500X-16 (core switch, stacked)

Firmware version SG350x = 2.5.5.47
Firmware version 4500x = 15.0(1r)SG(15)

Uplink 1: 10Gbit DAC cable - SFP-H10GB-CU1M (ports 1/0/4 - 1/1/8)
Uplink 2: 2 SFP+ moduels 10Gbit Single mode - SFP-10GLR-31 (ports 1/0/3 - 2/1/8)

Config on sw 4500x:

interface TenGigabitEthernet1/1/8
description UPL_1
switchport mode trunk
lacp rate fast
channel-group 8 mode active
end
interface TenGigabitEthernet2/1/8
description UPL_2
switchport mode trunk
lacp rate fast
channel-group 8 mode active
end
interface Port-channel8
description UPL
switchport
switchport mode trunk
spanning-tree link-type point-to-point
end

Config on SG350:
interface TenGigabitEthernet1/0/3
no eee enable
description UPL-SW-CORE01p2.x
lacp timeout short
channel-group 1 mode auto
no eee lldp enable
!
interface TenGigabitEthernet1/0/4
no eee enable
description UPL-SW-CORE01p1.x
lacp timeout short
channel-group 1 mode auto
no eee lldp enable
!
interface Port-Channel1
description UPL-SW-CORE-01
spanning-tree link-type point-to-point
switchport mode trunk
!
no eee enable
macro auto disabled

sho int po
Load balancing: src-dst-mac.

Gathering information...
Channel Ports
------- -----
Po1 Active: te1/0/3-4

Config on 4500x:
show int t1/1/8 etherchannel
Port state = Up Mstr Assoc In-Bndl
Channel group = 8 Mode = Active Gcchange = -
Port-channel = Po8 GC = - Pseudo port-channel = Po8
Port index = 0 Load = 0x00 Protocol = LACP

Flags: S - Device is sending Slow LACPDUs F - Device is sending fast LACPDUs.
A - Device is in active mode. P - Device is in passive mode.

Local information:
LACP port Admin Oper Port Port
Port Flags State Priority Key Key Number State
Te1/1/8 FA bndl 32768 0x8 0x8 0x8 0x3F

Partner's information:

LACP port Admin Oper Port Port
Port Flags Priority Dev ID Age key Key Number State
Te1/1/8 FA 1 045f.b95e.2721 0s 0x0 0x3E8 0x33 0x3F

Age of the port in the current state: 3d:18h:02m:25s

show int t2/1/8 etherchannel
Port state = Up Mstr Assoc In-Bndl
Channel group = 8 Mode = Active Gcchange = -
Port-channel = Po8 GC = - Pseudo port-channel = Po8
Port index = 1 Load = 0x00 Protocol = LACP

Flags: S - Device is sending Slow LACPDUs F - Device is sending fast LACPDUs.
A - Device is in active mode. P - Device is in passive mode.

Local information:
LACP port Admin Oper Port Port
Port Flags State Priority Key Key Number State
Te2/1/8 FA bndl 32768 0x8 0x8 0x18 0x3F

Partner's information:

LACP port Admin Oper Port Port
Port Flags Priority Dev ID Age key Key Number State
Te2/1/8 FA 1 045f.b95e.2721 0s 0x0 0x3E8 0x34 0x3F

Age of the port in the current state: 3d:18h:03m:11s

sho etherchannel sum
Flags: D - down P - bundled in port-channel
I - stand-alone s - suspended
H - Hot-standby (LACP only)
R - Layer3 S - Layer2
U - in use N - not in use, no aggregation
f - failed to allocate aggregator

M - not in use, minimum links not met
m - not in use, port not aggregated due to minimum links not met
u - unsuitable for bundling
w - waiting to be aggregated
d - default port

A - formed by Auto LAG

Number of channel-groups in use: 9
Number of aggregators: 9

Group Port-channel Protocol Ports
------+-------------+-----------+-----------------------------------------------
8 Po8(SU) LACP Te1/1/8(P) Te2/1/8(P)

Logs from SG350, as you can see the ports go down and up in seconds:

2147483211 2021-Apr-26 06:01:11 Informational %TRUNK-I-PORTADDED: Port te1/0/4 added to Po1
2147483212 2021-Apr-26 06:01:10 Warning %TRUNK-W-PORTREMOVED: Port te1/0/4 removed from Po1
2147483213 2021-Apr-26 06:01:10 Informational %TRUNK-I-PORTADDED: Port te1/0/3 added to Po1
2147483214 2021-Apr-26 06:01:10 Warning %TRUNK-W-PORTREMOVED: Port te1/0/3 removed from Po1
2147483215 2021-Apr-26 05:01:14 Informational %TRUNK-I-PORTADDED: Port te1/0/3 added to Po1
2147483216 2021-Apr-26 05:01:14 Warning %TRUNK-W-PORTREMOVED: Port te1/0/3 removed from Po1
2147483217 2021-Apr-26 04:01:15 Informational %TRUNK-I-PORTADDED: Port te1/0/3 added to Po1
2147483218 2021-Apr-26 04:01:14 Warning %TRUNK-W-PORTREMOVED: Port te1/0/3 removed from Po1
2147483219 2021-Apr-26 03:01:11 Warning %STP-W-PORTSTATUS: Po1: STP status Forwarding
2147483220 2021-Apr-26 03:01:10 Informational %TRUNK-I-PORTADDED: Port te1/0/3 added to Po1
2147483221 2021-Apr-26 03:01:10 Informational %LINK-I-Up: Po1
2147483222 2021-Apr-26 03:01:10 Informational %TRUNK-I-PORTADDED: Port te1/0/4 added to Po1
2147483223 2021-Apr-26 03:01:10 Warning %LINK-W-Down: Po1
2147483224 2021-Apr-26 03:01:10 Warning %TRUNK-W-PORTREMOVED: Port te1/0/3 removed from Po1
2147483225 2021-Apr-26 03:01:10 Warning %TRUNK-W-PORTREMOVED: Port te1/0/4 removed from Po1
2147483226 2021-Apr-26 02:01:10 Informational %TRUNK-I-PORTADDED: Port te1/0/3 added to Po1
2147483227 2021-Apr-26 02:01:10 Warning %TRUNK-W-PORTREMOVED: Port te1/0/3 removed from Po1
2147483228 2021-Apr-25 23:01:14 Informational %TRUNK-I-PORTADDED: Port te1/0/3 added to Po1
2147483229 2021-Apr-25 23:01:14 Warning %TRUNK-W-PORTREMOVED: Port te1/0/3 removed from Po1

Logs from another SG350 switch with same issue (same firmware):
2147447001 2021-Apr-26 06:02:28 Warning %LINK-W-Down: Po1
2147447002 2021-Apr-26 06:02:28 Warning %TRUNK-W-PORTREMOVED: Port te1/0/3 removed from Po1
2147447003 2021-Apr-26 06:02:28 Warning %TRUNK-W-PORTREMOVED: Port te1/0/4 removed from Po1
2147447004 2021-Apr-26 02:02:28 Informational %TRUNK-I-PORTADDED: Port te1/0/3 added to Po1
2147447005 2021-Apr-26 02:02:28 Warning %TRUNK-W-PORTREMOVED: Port te1/0/3 removed from Po1
2147447006 2021-Apr-26 00:02:29 Warning %STP-W-PORTSTATUS: Po1: STP status Forwarding

Hopefully someone has an idea on what could be causing this issue.

[marce1000]

                - What do the logs at the other end of the lacp-connection have to say ?

 M.

[Nico.]

Unfortunately logs on the 4500x don't show anything on the recent failures, changed setting to debug but no info yet.

In history I do find this but don't have logs on the SG350 from March anymore to do a direct compare:

Mar 12 12:23:08.781: %EC-5-BUNDLE: Interface Te1/1/8 joined port-channel Po8
Mar 12 12:23:08.894: %EC-5-BUNDLE: STANDBY:Interface Te1/1/8 joined port-channel Po8
Mar 12 12:32:25.193: %EC-5-UNBUNDLE: Interface Te1/1/8 left the port-channel Po8
Mar 12 12:32:25.196: %EC-5-UNBUNDLE: STANDBY:Interface Te1/1/8 left the port-channel Po8
Mar 12 12:34:38.039: %EC-5-BUNDLE: Interface Te1/1/8 joined port-channel Po8
Mar 12 12:34:38.155: %EC-5-BUNDLE: STANDBY:Interface Te1/1/8 joined port-channel Po8
Mar 12 12:42:36.974: %EC-5-UNBUNDLE: Interface Te1/1/8 left the port-channel Po8
Mar 12 12:42:36.977: %EC-5-UNBUNDLE: STANDBY:Interface Te1/1/8 left the port-channel Po8
Mar 12 12:44:29.765: %EC-5-BUNDLE: Interface Te1/1/8 joined port-channel Po8
Mar 12 12:44:29.864: %EC-5-BUNDLE: STANDBY:Interface Te1/1/8 joined port-channel Po8
Mar 12 12:49:45.226: %EC-5-UNBUNDLE: Interface Te1/1/8 left the port-channel Po8
Mar 12 12:49:45.229: %EC-5-UNBUNDLE: STANDBY:Interface Te1/1/8 left the port-channel Po8
Mar 12 12:50:58.853: %EC-5-BUNDLE: Interface Te1/1/8 joined port-channel Po8
Mar 12 12:50:58.972: %EC-5-BUNDLE: STANDBY:Interface Te1/1/8 joined port-channel Po8

[marce1000]

    - Look at interface counters , at all ends, look for errors , and or start by looking for link-level (-connection) issues.

 M.

[balaji.bandi]

how frequently this occuring  ? is this occuring after any upgrade ? any side ?

1. check the Physical Cable

2. check any STP priorities )in case)

can you post show interface g x/x bot the side switches.

check some configuration of SG side :

https://www.cisco.com/c/en/us/support/docs/smb/switches/cisco-550x-series-stackable-managed-switches/smb5848-configuring-lag-settings-on-a-switch-through-cli.html

[Nico.]

Thx already for the helpful tips. Gathered some more info as requested.

Physical connections have all been checked, cables and sfp modules replaced, optical connections cleaned.

4500x:
#show int te 1/1/8
TenGigabitEthernet1/1/8 is up, line protocol is up (connected)
Hardware is Ten Gigabit Ethernet Port, address is 005d.735e.f447 (bia 005d.735e.f447)
Description: UPL_SW-HBDC-102-te1-3
MTU 1500 bytes, BW 10000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 10Gb/s, link type is auto, media type is 10GBase-CU 1M
input flow-control is on, output flow-control is on
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output never, output hang never
Last clearing of "show interface" counters never
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 9
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 3221000 bits/sec, 421 packets/sec
5 minute output rate 4117000 bits/sec, 998 packets/sec
434416533 packets input, 269871400020 bytes, 0 no buffer
Received 6838781 broadcasts (6825504 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 input packets with dribble condition detected
1009922892 packets output, 589688411084 bytes, 0 underruns
0 output errors, 0 collisions, 3 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out
#show int te 2/1/8
TenGigabitEthernet2/1/8 is up, line protocol is up (connected)
Hardware is Ten Gigabit Ethernet Port, address is 005d.735e.d347 (bia 005d.735e.d347)
Description: UPL_SW-HBDC-102-te1-4
MTU 1500 bytes, BW 10000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 10Gb/s, link type is auto, media type is 10GBase-LR
input flow-control is on, output flow-control is on
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output never, output hang never
Last clearing of "show interface" counters never
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 3887000 bits/sec, 572 packets/sec
5 minute output rate 49000 bits/sec, 36 packets/sec
298558622 packets input, 178996053545 bytes, 0 no buffer
Received 5997595 broadcasts (5983223 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 input packets with dribble condition detected
53102813 packets output, 14269209834 bytes, 0 underruns
0 output errors, 0 collisions, 3 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out

#sho int te 1/1/8 counters

Port InBytes InUcastPkts InMcastPkts InBcastPkts
Te1/1/8 268947895153 426639318 6823405 13192

Port OutBytes OutUcastPkts OutMcastPkts OutBcastPkts
Te1/1/8 588497944043 779567588 175121424 53081898
#sho int te 2/1/8 counters

Port InBytes InUcastPkts InMcastPkts InBcastPkts
Te2/1/8 178068662348 291455063 5980239 14309

Port OutBytes OutUcastPkts OutMcastPkts OutBcastPkts
Te2/1/8 14260558291 20071619 20481333 12497874

#show int po1 counters

Port InBytes InUcastPkts InMcastPkts InBcastPkts
Po1 899676226523 1243591105 156683874 47978888

Port OutBytes OutUcastPkts OutMcastPkts OutBcastPkts
Po1 1159266739191 5429837276 72673144 96181989

#sho spanning-tree

VLAN0001
Spanning tree enabled protocol rstp
Root ID Priority 0
Address 6cc2.17b1.d620
Cost 24
Port 15 (TenGigabitEthernet1/1/15)
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 32769 (priority 32768 sys-id-ext 1)
Address 0008.e3ff.fdb8
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Aging Time 300 sec

Interface Role Sts Cost Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
Te1/1/15 Root FWD 4 128.15 P2p
Te2/1/15 Desg FWD 4 128.1295 P2p
Te2/1/16 Desg FWD 4 128.1296 P2p
Po4 Desg FWD 1 128.2564 P2p
Po5 Desg FWD 1 128.2565 P2p
Po6 Desg FWD 2 128.2566 P2p
Po7 Desg FWD 1 128.2567 P2p
Po8 Desg FWD 1 128.2568 P2p
Po9 Desg FWD 2 128.2569 P2p
Po10 Desg FWD 2 128.2570 P2p

_____________________________________________________________________________________________

SG350x:

#sho int counters te 1/0/3

Port InUcastPkts InMcastPkts InBcastPkts InOctets
---------------- ------------ ------------ ------------ ------------
te1/0/3 23795331 1058191 318348 15966761391

Port OutUcastPkts OutMcastPkts OutBcastPkts OutOctets
---------------- ------------ ------------ ------------ ------------
te1/0/3 6028532 30316 889 4944994500

FCS Errors: 0
Single Collision Frames: 0
Multiple Collision Frames: 0
SQE Test Errors: 0
Deferred Transmissions: 0
Late Collisions: 0
Excessive Collisions: 0
Carrier Sense Errors: 0
Oversize Packets: 0
Internal MAC Rx Errors: 0
Symbol Errors: 0
Received Pause Frames: 0
Transmitted Pause Frames: 0

#sho int counters te 1/0/4

Port InUcastPkts InMcastPkts InBcastPkts InOctets
---------------- ------------ ------------ ------------ ------------
te1/0/4 157255 210458 108483 95764898

Port OutUcastPkts OutMcastPkts OutBcastPkts OutOctets
---------------- ------------ ------------ ------------ ------------
te1/0/4 14797908 45490 255 12564028254

FCS Errors: 0
Single Collision Frames: 0
Multiple Collision Frames: 0
SQE Test Errors: 0
Deferred Transmissions: 0
Late Collisions: 0
Excessive Collisions: 0
Carrier Sense Errors: 0
Oversize Packets: 0
Internal MAC Rx Errors: 0
Symbol Errors: 0
Received Pause Frames: 0
Transmitted Pause Frames: 0

#show int counters po 1

Port InUcastPkts InMcastPkts InBcastPkts InOctets
---------------- ------------ ------------ ------------ ------------
Po1 25749706 1341373 454285 17111483476

Port OutUcastPkts OutMcastPkts OutBcastPkts OutOctets
---------------- ------------ ------------ ------------ ------------
Po1 22572195 79992 1267 19096278807

FCS Errors: 0
Single Collision Frames: 0
Multiple Collision Frames: 0
SQE Test Errors: 0
Deferred Transmissions: 0
Late Collisions: 0
Excessive Collisions: 0
Carrier Sense Errors: 0
Oversize Packets: 0
Internal MAC Rx Errors: 0
Symbol Errors: 0
Received Pause Frames: 0
Transmitted Pause Frames: 0

#sho int status te 1/0/3
Flow Link Back Mdix
Port Type Duplex Speed Neg ctrl State Pressure Mode
-------- ------------ ------ ----- -------- ---- ----------- -------- -------
te1/0/3 10G-Fiber Full 10000 Disabled Off Up Disabled Off
#sho int status te 1/0/4
Flow Link Back Mdix
Port Type Duplex Speed Neg ctrl State Pressure Mode
-------- ------------ ------ ----- -------- ---- ----------- -------- -------
te1/0/4 10G-Fiber Full 10000 Disabled Off Up Disabled Off

#show spanning-tree
Spanning tree enabled mode: RSTP
Default port cost method: long
Loopback guard: Disabled

Root ID Priority: 0
Address: 6c:c2:17:b1:d6:20
Cost: 1024
Port: Po1
Hello Time: 2 sec Max Age: 20 sec Forward Delay: 15 sec
Bridge ID Priority: 32768
Address: 04:5f:b9:5e:27:21
Hello Time: 2 sec Max Age: 20 sec Forward Delay: 15 sec

Number of topology changes: 69 last change occurred: 00:43:48 ago
Times: hold: 1, topology change: 35, notification: 2
hello: 2, max age: 20, forward delay: 15

Interfaces
Name State Prio.Nbr Cost Sts Role PortFast Type
--------- -------- --------- -------- ------ ---- -------- -----------------
xxx
Po1 enabled 128.1000 1000 Frw Root No P2P (RSTP)
Po2 enabled 128.1001 20000 Dsbl Dsbl No -

[balaji.bandi]

Full-duplex, 10Gb/s, link type is auto, media type is 10GBase-CU 1M

If you using Twinax cable does the cable support matrix checked with SG series switch ?  Try remove this cable and see any stability ?

#show int te 2/1/8
Full-duplex, 10Gb/s, link type is auto, media type is 10GBase-LR   -  This single mode Fibre ? - what cable you using ?

Number of topology changes: 69 last change occurred: 00:43:48 ago

Why this topology changes so frequently ?

[Nico.]

Have replaced the DAC cable with fiber and SFP+ modules to check if this does indeed make a difference.

Checked compatiblity matrix and should be ok: https://tmgmatrix.cisco.com/?si=sg350x

And getting no errors on the interfaces and logging only mentioning it going down and back up again without any other reason, just like it is physically removed. (which it is not)

Fiber is single mode OS2 cable.

Topology change seems to occur when a member port of the po interface goes down, but this is not always the case.

When I remove a port from the port channel and add it again I do get a first message when the port is just coming up that it differs from po1 capabilities for some reason.

   switch-sg350x(config-if)#channel-group 1 mode auto
   26-Apr-2021 16:47:13 %2SWTRUNK-I-TRNKPORTPARAM: auto-negotiation/adv. capabilities of port te1/0/4 differ from auto-   negotiation/adv. capabilities of Po1
   switch-sg350x(config-if)#26-Apr-2021 16:47:13 %LINK-I-Up: te1/0/4
   26-Apr-2021 16:47:15 %2SWTRUNK-I-TRNKPORTPARAM: auto-negotiation/adv. capabilities of port te1/0/4 differ from auto-   negotiation/adv. capabilities of Po1

   switch-sg350x(config-if)#no 26-Apr-2021 16:47:23 %TRUNK-I-PORTADDED: Port te1/0/4 added to Po1

[balaji.bandi]

Suggestion is  - remove the DAC cable, run with only 1 Fibre cable and advise what is the outcome, is this stable ?

te1/0/4  - is this connected to DAC Cable ?

[Nico.]

te 1/0/4 = single mode fiber

With te 1/0/3 connection fiber instead of DAC, problem remained:

2147483135 2021-Apr-27 10:01:11 Informational %TRUNK-I-PORTADDED: Port te1/0/3 added to Po1
2147483136 2021-Apr-27 10:01:11 Warning %TRUNK-W-PORTREMOVED: Port te1/0/3 removed from Po1
2147483137 2021-Apr-27 06:01:12 Informational %TRUNK-I-PORTADDED: Port te1/0/4 added to Po1
2147483138 2021-Apr-27 06:01:12 Warning %TRUNK-W-PORTREMOVED: Port te1/0/4 removed from Po1
2147483139 2021-Apr-27 06:01:12 Informational %TRUNK-I-PORTADDED: Port te1/0/3 added to Po1
2147483140 2021-Apr-27 06:01:12 Warning %TRUNK-W-PORTREMOVED: Port te1/0/3 removed from Po1
2147483141 2021-Apr-27 03:01:09 Informational %TRUNK-I-PORTADDED: Port te1/0/4 added to Po1
2147483142 2021-Apr-27 03:01:09 Warning %TRUNK-W-PORTREMOVED: Port te1/0/4 removed from Po1
2147483143 2021-Apr-27 01:01:11 Informational %TRUNK-I-PORTADDED: Port te1/0/3 added to Po1
2147483144 2021-Apr-27 01:01:10 Warning %TRUNK-W-PORTREMOVED: Port te1/0/3 removed from Po1
2147483145 2021-Apr-26 17:01:11 Informational %TRUNK-I-PORTADDED: Port te1/0/3 added to Po1
2147483146 2021-Apr-26 17:01:11 Warning %TRUNK-W-PORTREMOVED: Port te1/0/3 removed from Po1

Have now removed the te 1/0/3 connection so only one port is active (te 1/0/4) in the po interface.

[balaji.bandi · Accepted solution]

Can you post both side interface output : Cat 4K and SG)

show int te x/x

show ethchannel summary

show interface port-channel X

Look also below thread for reference :

https://community.cisco.com/t5/small-business-switches/sg550-sg300-lacp-portchannel-interfaces-joining-and-leaving-quot/td-p/3733564

[Nico.]

Checked the thread and hopefully flashing to the older firmware will resolve the issue, would be a bit strange that this is still not resolved, but if it works it works.

Flashed the switch now to firmware 2.3.5.63 and reconnected the other fiber. So po is now up with 2 member ports atm, will check if I have any errors tomorrow morning.

Output of requested int:

4500x#show int te 1/1/8
TenGigabitEthernet1/1/8 is down, line protocol is down (notconnect)
Hardware is Ten Gigabit Ethernet Port, address is 005d.735e.f447 (bia 005d.735e.f447)
Description: UPL_SW-HBDC-102-te1-3
MTU 1500 bytes, BW 10000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, Auto-speed, link type is auto, media type is 10GBase-LR
input flow-control is off, output flow-control is off
ARP type: ARPA, ARP Timeout 04:00:00
Last input 03:12:14, output never, output hang never
Last clearing of "show interface" counters never
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 11
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
454660991 packets input, 286255167645 bytes, 0 no buffer
Received 6945972 broadcasts (6931666 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 input packets with dribble condition detected
1063134970 packets output, 625951129706 bytes, 0 underruns
0 output errors, 0 collisions, 3 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out

4500x#show int te 2/1/8
TenGigabitEthernet2/1/8 is up, line protocol is up (connected)
Hardware is Ten Gigabit Ethernet Port, address is 005d.735e.d347 (bia 005d.735e.d347)
Description: UPL_SW-HBDC-102-te1-4
MTU 1500 bytes, BW 10000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 10Gb/s, link type is auto, media type is 10GBase-LR
input flow-control is on, output flow-control is on
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output never, output hang never
Last clearing of "show interface" counters never
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 3475000 bits/sec, 600 packets/sec
5 minute output rate 2222000 bits/sec, 660 packets/sec
332163365 packets input, 208296655460 bytes, 0 no buffer
Received 6118696 broadcasts (6102971 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 input packets with dribble condition detected
67776310 packets output, 23286222007 bytes, 0 underruns
0 output errors, 0 collisions, 3 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out

4500x#show etherchannel sum
Flags: D - down P - bundled in port-channel
I - stand-alone s - suspended
H - Hot-standby (LACP only)
R - Layer3 S - Layer2
U - in use N - not in use, no aggregation
f - failed to allocate aggregator

M - not in use, minimum links not met
m - not in use, port not aggregated due to minimum links not met
u - unsuitable for bundling
w - waiting to be aggregated
d - default port

A - formed by Auto LAG

Number of channel-groups in use: 9
Number of aggregators: 9

Group Port-channel Protocol Ports
------+-------------+-----------+-----------------------------------------------
1 Po1(SU) - Te1/1/1(P) Te1/1/2(P)
2 Po2(SU) - Te2/1/1(P) Te2/1/2(P)
4 Po4(SU) LACP Te1/1/4(P) Te2/1/4(P)
5 Po5(SU) LACP Te1/1/5(P) Te2/1/5(P)
6 Po6(SU) LACP Te1/1/6(D) Te2/1/6(P)
7 Po7(SU) LACP Te1/1/7(P) Te2/1/7(P)
8 Po8(SU) LACP Te1/1/8(D) Te2/1/8(P)
9 Po9(SU) LACP Te1/1/13(D) Te2/1/13(P)
10 Po10(SU) LACP Te1/1/14(D) Te2/1/14(P)

4500x#sho int port-channel 8
Port-channel8 is up, line protocol is up (connected)
Hardware is EtherChannel, address is 005d.735e.d347 (bia 005d.735e.d347)
Description: UPL_SW-HB-DC-102
MTU 1500 bytes, BW 10000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 10Gb/s, media type is N/A
input flow-control is on, output flow-control is unsupported
Members in this channel: Te2/1/8
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:07:11, output never, output hang never
Last clearing of "show interface" counters never
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 3260000 bits/sec, 578 packets/sec
5 minute output rate 2093000 bits/sec, 640 packets/sec
786852204 packets input, 494567209872 bytes, 0 no buffer
Received 13063653 broadcasts (13033607 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 input packets with dribble condition detected
1130944173 packets output, 649250208837 bytes, 0 underruns
0 output errors, 0 collisions, 2 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
sg350x#show interfaces advertise te 1/0/3

Port: te1/0/3
Type: 10G-Fiber
Link state: Up
Auto negotiation: Disabled
---------------------------------------------------
sg350x#sho interfaces switchport te 1/0/3
Gathering information....

S-VLAN Ethernet Type: 0x8100 (802.1q)
Name: te1/0/3
Switchport: enable
Administrative Mode: access (Inactive: lag membership)
Operational Mode: up
Access Mode VLAN: 1
Access Multicast TV VLAN: none
Trunking Native Mode VLAN: 1
Trunking VLANs: 1,10,20,30,40,4041-4094 (Inactive)
General PVID: 1
General VLANs: none
General Egress Tagged VLANs: none
General Forbidden VLANs: none
General Ingress Filtering: enabled
General Acceptable Frame Type: all
General GVRP status: disabled
Customer Mode VLAN: none
Customer Multicast TV VLANs: none
Private-vlan promiscuous-association primary VLAN: none
Private-vlan promiscuous-association Secondary VLANs: none
Private-vlan host-association primary VLAN: none
Private-vlan host-association Secondary VLAN: none

VLAN Mapping Tunnel - no resources

VLAN Mapping One-To-One - no resources

Classification rules:

Classification type Group ID VLAN ID
------------------- -------- -------

---------------------------------------------------
sg350x#show int counters te 1/0/3

Port InUcastPkts InMcastPkts InBcastPkts InOctets
---------------- ------------ ------------ ------------ ------------
te1/0/3 1911461 97670 29270 976870205

Port OutUcastPkts OutMcastPkts OutBcastPkts OutOctets
---------------- ------------ ------------ ------------ ------------
te1/0/3 1280303 2850 36 1092291476

FCS Errors: 0
Multiple Collision Frames: 0
SQE Test Errors: 0
Deferred Transmissions: 0
Carrier Sense Errors: 0
Oversize Packets: 0
Internal MAC Rx Errors: 0
Symbol Errors: 0
Received Pause Frames: 0
Transmitted Pause Frames: 0
---------------------------------------------------
sg350x#sho interfaces switchport te 1/0/4
Gathering information....

S-VLAN Ethernet Type: 0x8100 (802.1q)
Name: te1/0/4
Switchport: enable
Administrative Mode: access (Inactive: lag membership)
Operational Mode: up
Access Mode VLAN: 1
Access Multicast TV VLAN: none
Trunking Native Mode VLAN: 1
Trunking VLANs: 1,10,20,30,40,4041-4094 (Inactive)
General PVID: 1
General VLANs: none
General Egress Tagged VLANs: none
General Forbidden VLANs: none
General Ingress Filtering: enabled
General Acceptable Frame Type: all
General GVRP status: disabled
Customer Mode VLAN: none
Customer Multicast TV VLANs: none
Private-vlan promiscuous-association primary VLAN: none
Private-vlan promiscuous-association Secondary VLANs: none
Private-vlan host-association primary VLAN: none
Private-vlan host-association Secondary VLAN: none

VLAN Mapping Tunnel - no resources

VLAN Mapping One-To-One - no resources

Classification rules:

Classification type Group ID VLAN ID
------------------- -------- -------
---------------------------------------------------
sg350x#show int counters te 1/0/4

Port InUcastPkts InMcastPkts InBcastPkts InOctets
---------------- ------------ ------------ ------------ ------------
te1/0/4 4118389 211493 71515 1988731393

Port OutUcastPkts OutMcastPkts OutBcastPkts OutOctets
---------------- ------------ ------------ ------------ ------------
te1/0/4 4951697 12220 572 4330428672

FCS Errors: 0
Multiple Collision Frames: 0
SQE Test Errors: 0
Deferred Transmissions: 0
Carrier Sense Errors: 0
Oversize Packets: 0
Internal MAC Rx Errors: 0
Symbol Errors: 0
Received Pause Frames: 0
Transmitted Pause Frames: 0
---------------------------------------------------
sg350x#sho int port-channel 1

Load balancing: src-dst-mac.

Gathering information...
Channel Ports
------- -----
Po1 Active: te1/0/3-4

[balaji.bandi]

glad and that good news so far, please do monitor and report back your outcome.

[Nico.]

At the moment, no more issues on this switch with ports going down and up. So firmware downgrade seems to have helped.

Will perform this step on the other switches I have as well and follow up for a week or 2 to see if the problem is actually resolved.

Thx for the help.

[Nico.]

Have had no more issues on the portchannel interfaces on SG350x and SG550x switches that were downgraded to v2.3.5.63.

This firmware version cannot be used on the CBS350 and SX550x switches however, these are now on latest firmware version CBS350 v3.0.069 and SX550x v2.5.87.

For the CBS350 there is no problem with the PO interfaces, so seems to be ok. For SX550x I still need to test PO interfaces.