Packet loss/increased latency when 1 link of Port channel is down

Teambrutewilson1 · ‎09-09-2021

I Have 2 unstacked Nexus 9300 switches in a datacentre that are using a port channel group that has a VPC on it.

Both of the connections from these switches run through separate Cloud VXC's and connect to a pair of Catalyst 9200's that are stacked with a port channel in another Datacentre.

I occasionally see the interface from the first 9300 go down and then come up within 5 mins and after that happens there is significant packet loss and increased latency.

Currently to resolve the issue I down up that interface and the connection becomes stable.

Any ideas on what could be causing the packet loss and latency from the side of the connection that the 9300's are on?

Georg Pauwen · ‎09-09-2021

Hello,

the default load balancing algorithm on your port channel is src-dst-mac (source-destination MAC). You could try and change that to e.g. src-mac or dst-mac, and check if that makes a difference. The real issue is obviously why the one Cloud VXC port channel member goes down. It is always a good idea to get with the provider asap.

For the time being, an EEM script such as the one below at least lets you automate the process of bouncing the interface manually (the actual entry in your log might be different, so change that to whatever you see in the log):

event manager applet BOUNCE_INTF
event syslog pattern "%LINEPROTO-5-UPDOWN: Line protocol on Interface Ethernet1/4, changed state to down"
action 1.0 cli command "enable"
action 2.0 cli command "config t"
action 3.0 cli command "interface Ethernet 1/4"
action 4.0 cli command "shut"
action 5.0 cli command "no shut"

Giuseppe Larosa · ‎09-10-2021

Hello @Teambrutewilson1 ,

the problem can be caused by the L2 transport services used to interconnect each member link between the two datacenters.

Check in the logs of all involved devices if when the link on first Nexus goes down this info is propagated to the corresponding Cat9200 port or that port is still up .

if that is the case and you are using unconditional bundling I would suggest you to move to LACP mode active so that the lack of LACP frames received on C9200 side will cause the Cat9200 to put the port in suspended state and to resume later.

Hope to help

Giuseppe

Teambrutewilson1 · ‎09-14-2021

Hi @Giuseppe Larosa
thank you for replying.

On the 9200 the 2 ports are in LACP active mode:

Port: Te1/1/1
------------

Port state = Up Mstr Assoc In-Bndl
Channel group = 10 Mode = Active Gcchange = -
Port-channel = Po10 GC = - Pseudo port-channel = Po10
Port index = 0 Load = 0x00 Protocol = LACP

Flags: S - Device is sending Slow LACPDUs F - Device is sending fast LACPDUs.
A - Device is in active mode. P - Device is in passive mode.

Local information:
LACP port Admin Oper Port Port
Port Flags State Priority Key Key Number State
Te1/1/1 SA bndl 32768 0xA 0xA 0x132 0x3D

Partner's information:

LACP port Admin Oper Port Port
Port Flags Priority Dev ID Age key Key Number State
Te1/1/1 SA 32768 0023.04ee.be01 17s 0x0 0x800A 0x161 0x3D

Port: Te2/1/1
------------

Port state = Up Mstr Assoc In-Bndl
Channel group = 10 Mode = Active Gcchange = -
Port-channel = Po10 GC = - Pseudo port-channel = Po10
Port index = 0 Load = 0x00 Protocol = LACP

Flags: S - Device is sending Slow LACPDUs F - Device is sending fast LACPDUs.
A - Device is in active mode. P - Device is in passive mode.

Local information:
LACP port Admin Oper Port Port
Port Flags State Priority Key Key Number State
Te2/1/1 SA bndl 32768 0xA 0xA 0x232 0x3D

Partner's information:

LACP port Admin Oper Port Port
Port Flags Priority Dev ID Age key Key Number State
Te2/1/1 SA 32768 0023.04ee.be01 13s 0x0 0x800A 0x161 0x3D

attached is a visual of the connectivity setup

in the last occurance the 2nd core switch ports went down first:

2021 Sep 10 07:25:34 CL1SWCORE02 %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel10: Ethernet1/97 is down
2021 Sep 10 07:25:34 CL1SWCORE02 %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel10: first operational port changed from Ethernet1/97 to none
2021 Sep 10 07:25:34 CL1SWCORE02 %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel10 is down (No operational members)
2021 Sep 10 07:25:34 CL1SWCORE02 %ETHPORT-5-IF_DOWN_CFG_CHANGE: Interface Ethernet1/97 is down(Config change)
2021 Sep 10 07:25:34 CL1SWCORE02 %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel10 is down (No operational members)
2021 Sep 10 07:25:34 CL1SWCORE02 %ETHPORT-5-IF_DOWN_ADMIN_DOWN: Interface Ethernet1/97 is down (Administratively down)
2021 Sep 10 07:25:35 CL1SWCORE02 %ETHPORT-5-IF_ADMIN_UP: Interface Ethernet1/97 is admin up .
2021 Sep 10 07:25:38 CL1SWCORE02 %VSHD-5-VSHD_SYSLOG_CONFIG_I: Configured from vty by admin on 10.237.121.199@pts/1
2021 Sep 10 07:25:38 CL1SWCORE02 %ETHPORT-5-SPEED: Interface Ethernet1/97, operational speed changed to 10 Gbps
2021 Sep 10 07:25:38 CL1SWCORE02 %ETHPORT-5-IF_DUPLEX: Interface Ethernet1/97, operational duplex mode changed to Full
2021 Sep 10 07:25:38 CL1SWCORE02 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet1/97, operational Receive Flow Control state changed to off
2021 Sep 10 07:25:38 CL1SWCORE02 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet1/97, operational Transmit Flow Control state changed to off
2021 Sep 10 07:25:38 CL1SWCORE02 %ETHPORT-5-SPEED: Interface port-channel10, operational speed changed to 10 Gbps
2021 Sep 10 07:25:38 CL1SWCORE02 %ETHPORT-5-IF_DUPLEX: Interface port-channel10, operational duplex mode changed to Full
2021 Sep 10 07:25:38 CL1SWCORE02 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface port-channel10, operational Receive Flow Control state changed to off
2021 Sep 10 07:25:38 CL1SWCORE02 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface port-channel10, operational Transmit Flow Control state changed to off
2021 Sep 10 07:25:45 CL1SWCORE02 %ETH_PORT_CHANNEL-5-PORT_UP: port-channel10: Ethernet1/97 is up
2021 Sep 10 07:25:45 CL1SWCORE02 %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel10: first operational port changed from none to Ethernet1/97
2021 Sep 10 07:25:45 CL1SWCORE02 %ETHPORT-5-IF_UP: Interface Ethernet1/97 is up in mode trunk
2021 Sep 10 07:25:45 CL1SWCORE02 %ETHPORT-5-IF_UP: Interface port-channel10 is up in mode trunk
2021 Sep 10 07:29:00 CL1SWCORE02 %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel10: Ethernet1/97 is down
2021 Sep 10 07:29:00 CL1SWCORE02 %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel10: first operational port changed from Ethernet1/97 to none
2021 Sep 10 07:29:00 CL1SWCORE02 %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel10 is down (No operational members)
2021 Sep 10 07:29:00 CL1SWCORE02 %ETHPORT-5-IF_DOWN_CFG_CHANGE: Interface Ethernet1/97 is down(Config change)
2021 Sep 10 07:29:00 CL1SWCORE02 %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel10 is down (No operational members)
2021 Sep 10 07:29:00 CL1SWCORE02 %ETHPORT-5-IF_DOWN_ADMIN_DOWN: Interface Ethernet1/97 is down (Administratively down)
2021 Sep 10 07:30:37 CL1SWCORE02 %ETHPORT-5-IF_ADMIN_UP: Interface Ethernet1/97 is admin up .
2021 Sep 10 07:30:40 CL1SWCORE02 %ETHPORT-5-SPEED: Interface Ethernet1/97, operational speed changed to 10 Gbps
2021 Sep 10 07:30:40 CL1SWCORE02 %ETHPORT-5-IF_DUPLEX: Interface Ethernet1/97, operational duplex mode changed to Full
2021 Sep 10 07:30:40 CL1SWCORE02 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet1/97, operational Receive Flow Control state changed to off
2021 Sep 10 07:30:40 CL1SWCORE02 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet1/97, operational Transmit Flow Control state changed to off
2021 Sep 10 07:30:40 CL1SWCORE02 %ETHPORT-5-SPEED: Interface port-channel10, operational speed changed to 10 Gbps
2021 Sep 10 07:30:40 CL1SWCORE02 %ETHPORT-5-IF_DUPLEX: Interface port-channel10, operational duplex mode changed to Full
2021 Sep 10 07:30:40 CL1SWCORE02 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface port-channel10, operational Receive Flow Control state changed to off
2021 Sep 10 07:30:40 CL1SWCORE02 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface port-channel10, operational Transmit Flow Control state changed to off
2021 Sep 10 07:30:41 CL1SWCORE02 %VSHD-5-VSHD_SYSLOG_CONFIG_I: Configured from vty by admin on 10.237.121.199@pts/1
2021 Sep 10 07:30:46 CL1SWCORE02 %ETH_PORT_CHANNEL-5-PORT_UP: port-channel10: Ethernet1/97 is up
2021 Sep 10 07:30:46 CL1SWCORE02 %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel10: first operational port changed from none to Ethernet1/97
2021 Sep 10 07:30:46 CL1SWCORE02 %ETHPORT-5-IF_UP: Interface Ethernet1/97 is up in mode trunk
2021 Sep 10 07:30:46 CL1SWCORE02 %ETHPORT-5-IF_UP: Interface port-channel10 is up in mode trunk

then the 1st core switch after

Giuseppe Larosa · ‎09-14-2021

Hello @Teambrutewilson1 ,

>> 2021 Sep 10 07:25:34 CL1SWCORE02 %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel10: Ethernet1/97 is down
2021 Sep 10 07:25:34 CL1SWCORE02 %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel10: first operational port changed from Ethernet1/97 to none

Then you shut down the port and re-enable the port and the port becomes up and the port-channel is operationa again

>> 2021 Sep 10 07:25:45 CL1SWCORE02 %ETH_PORT_CHANNEL-5-PORT_UP: port-channel10: Ethernet1/97 is up
2021 Sep 10 07:25:45 CL1SWCORE02 %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel10: first operational port changed from none to Ethernet1/97

Are you using an EEM script ? how it is possible that you were able to shut and unshut in so few seconds ?

You have LACP on member links.

Have you configured the port-channel 10 for vPC ?

int port-channel 10

switchport mode trunk

vpc 10

and have you done this on both Nexus using the same vPC value ?

What are the logs on the Catalyst 9200 side , there are nany events reported there for member links Te1/1/1 and Te2/1/1 ?

Hope to help

Giuseppe