Re: interface getting down N9300-4503 vPC

jen2181918 · ‎06-20-2020

I have two cisco Nexus 93108TC Chassis Im having problem with the traffic going in and out the interface, please let me know what I missed.

Thank you in advance.

From the logging:

zhongxinjifang9300-1 %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-

channel2: first operational port changed from Ethernet1/49 to none

2020 Jun 21 07:03:11 zhongxinjifang9300-1 %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel2: Ethernet1/49 is down

2020 Jun 21 07:03:11 zhongxinjifang9300-1 %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBER

S_DOWN: Interface port-channel2 is down (No operational members)

2020 Jun 21 07:03:11 zhongxinjifang9300-1 %ETHPORT-5-IF_BANDWIDTH_CHANGE: Interface port-channel2,bandwidth changed to 100000 Kbit

2020 Jun 21 07:03:11 zhongxinjifang9300-1 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/49 is down (Link failure)

2020 Jun 21 07:03:12 zhongxinjifang9300-1 %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBER

S_DOWN: Interface port-channel2 is down (No operational members)

2020 Jun 21 07:03:12 zhongxinjifang9300-1 last message repeated 1 time

2020 Jun 21 07:03:12 zhongxinjifang9300-1 %ETHPORT-5-SPEED: Interface Ethernet1/

49, operational speed changed to 1 Gbps

2020 Jun 21 07:03:12 zhongxinjifang9300-1 %ETHPORT-5-IF_DUPLEX: Interface Ethern

et1/49, operational duplex mode changed to Full

2020 Jun 21 07:03:12 zhongxinjifang9300-1 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interfa

ce Ethernet1/49, operational Receive Flow Control state changed to off

2020 Jun 21 07:03:12 zhongxinjifang9300-1 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interfa

ce Ethernet1/49, operational Transmit Flow Control state changed to off

2020 Jun 21 07:03:12 zhongxinjifang9300-1 %ETHPORT-5-SPEED: Interface port-chann

el2, operational speed changed to 1 Gbps

2020 Jun 21 07:03:12 zhongxinjifang9300-1 %ETHPORT-5-IF_DUPLEX: Interface port-c

hannel2, operational duplex mode changed to Full

2020 Jun 21 07:03:12 zhongxinjifang9300-1 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interfa

ce port-channel2, operational Receive Flow Control state changed to off

2020 Jun 21 07:03:12 zhongxinjifang9300-1 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interfa

ce port-channel2, operational Transmit Flow Control state changed to off

2020 Jun 21 07:03:13 zhongxinjifang9300-1 %ETH_PORT_CHANNEL-5-PORT_UP: port-chan

nel2: Ethernet1/49 is up

2020 Jun 21 07:03:13 zhongxinjifang9300-1 %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-

channel2: first operational port changed from none to Ethernet1/49

2020 Jun 21 07:03:13 zhongxinjifang9300-1 %ETHPORT-5-IF_BANDWIDTH_CHANGE: Interf

ace port-channel2,bandwidth changed to 1000000 Kbit

2020 Jun 21 07:03:13 zhongxinjifang9300-1 %ETHPORT-5-IF_UP: Interface Ethernet1/

49 is up in mode trunk

2020 Jun 21 07:03:13 zhongxinjifang9300-1 %ETHPORT-5-IF_UP: Interface port-chann

el2 is up in mode trunk

2020 Jun 21 07:03:13 zhongxinjifang9300-1 %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-

channel2: first operational port changed from Ethernet1/49 to none

2020 Jun 21 07:03:13 zhongxinjifang9300-1 %ETH_PORT_CHANNEL-5-PORT_DOWN: port-ch

annel2: Ethernet1/49 is down

2020 Jun 21 07:03:13 zhongxinjifang9300-1 %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBER

S_DOWN: Interface port-channel2 is down (No operational members)

2020 Jun 21 07:03:13 zhongxinjifang9300-1 %ETHPORT-5-IF_BANDWIDTH_CHANGE: Interf

ace port-channel2,bandwidth changed to 100000 Kbit

2020 Jun 21 07:03:13 zhongxinjifang9300-1 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Inter

face Ethernet1/49 is down (Link failure)

2020 Jun 21 07:03:13 zhongxinjifang9300-1 %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBER

S_DOWN: Interface port-channel2 is down (No operational members)

2020 Jun 21 07:03:14 zhongxinjifang9300-1 last message repeated 1 time

2020 Jun 21 07:03:14 zhongxinjifang9300-1 %ETHPORT-5-SPEED: Interface Ethernet1/

49, operational speed changed to 1 Gbps

2020 Jun 21 07:03:14 zhongxinjifang9300-1 %ETHPORT-5-IF_DUPLEX: Interface Ethern

et1/49, operational duplex mode changed to Full

2020 Jun 21 07:03:14 zhongxinjifang9300-1 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interfa

zhongxinjifang9300-1 %TAHUSD-SLOT1-4-BUFFER_THRESHOLD_EXCEE

DED: Module 1 Instance 0 Pool-group buffer 90 percent threshold is exceeded!

marce1000 · ‎06-21-2020

- Check the physical-health of the particular connection.

M.

-- Each morning when I wake up and look into the mirror I always say ' Why am I so brilliant ? '
When the mirror will then always repond to me with ' The only thing that exceeds your brilliance is your beauty! '

jen2181918 · ‎06-21-2020

Thank you for your reply。

I want to ask the meaning of this log：

2020 Jun 21 07:03:50zhongxinjifang9300-1 %TAHUSD-SLOT1-4-BUFFER_THRESHOLD_EXCEE

DED: Module 1 Instance 0 Pool-group buffer 90 percent threshold is exceeded!

The down message that appears sometimes few minutes， few hours，or half a day, or one day.

2020 Jun 18 17:34:19 zhongxinjifang9300-1 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Inter

face Ethernet1/49 is down (Link failure)

2020 Jun 18 17:34:19 zhongxinjifang9300-1 %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBER

S_DOWN: Interface port-channel2 is down (No operational members)

2020 Jun 18 17:34:19 zhongxinjifang9300-1 last message repeated 1 time

2020 Jun 18 17:34:19 zhongxinjifang9300-1 %ETHPORT-5-SPEED: Interface Ethernet1/

49, operational speed changed to 1 Gbps

2020 Jun 18 17:34:19 zhongxinjifang9300-1 %ETHPORT-5-IF_DUPLEX: Interface Ethern

et1/49, operational duplex mode changed to Full

2020 Jun 18 17:34:19 zhongxinjifang9300-1 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interfa

ce Ethernet1/49, operational Receive Flow Control state changed to off

2020 Jun 18 17:34:19 zhongxinjifang9300-1 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interfa

ce Ethernet1/49, operational Transmit Flow Control state changed to off

2020 Jun 18 17:34:19 zhongxinjifang9300-1 %ETHPORT-5-SPEED: Interface port-chann

el2, operational speed changed to 1 Gbps

2020 Jun 18 17:34:19 zhongxinjifang9300-1 %ETHPORT-5-IF_DUPLEX: Interface port-c

hannel2, operational duplex mode changed to Full

2020 Jun 18 17:34:19 zhongxinjifang9300-1 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interfa

ce port-channel2, operational Receive Flow Control state changed to off

2020 Jun 18 17:34:19 zhongxinjifang9300-1 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interfa

ce port-channel2, operational Transmit Flow Control state changed to off

2020 Jun 18 17:34:19 zhongxinjifang9300-1 %ETH_PORT_CHANNEL-5-PORT_UP: port-chan

nel2: Ethernet1/49 is up

2020 Jun 18 17:34:19 zhongxinjifang9300-1 %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-

channel2: first operational port changed from none to Ethernet1/49

2020 Jun 18 17:34:19 zhongxinjifang9300-1 %ETHPORT-5-IF_BANDWIDTH_CHANGE: Interf

ace port-channel2,bandwidth changed to 1000000 Kbit

2020 Jun 18 17:34:19 zhongxinjifang9300-1 %ETHPORT-5-IF_UP: Interface Ethernet1/

49 is up in mode trunk

2020 Jun 18 17:34:19 zhongxinjifang9300-1 %ETHPORT-5-IF_UP: Interface port-chann

el2 is up in mode trunk

2020 Jun 19 06:08:38 zhongxinjifang9300-1 %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-

channel2: first operational port changed from Ethernet1/49 to none

2020 Jun 19 06:08:38 zhongxinjifang9300-1 %ETH_PORT_CHANNEL-5-PORT_DOWN: port-ch

annel2: Ethernet1/49 is down

2020 Jun 19 06:08:39 zhongxinjifang9300-1 %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBER

S_DOWN: Interface port-channel2 is down (No operational members)

2020 Jun 19 06:08:39 zhongxinjifang9300-1 %ETHPORT-5-IF_BANDWIDTH_CHANGE: Interf

ace port-channel2,bandwidth changed to 100000 Kbit

2020 Jun 19 06:08:39 zhongxinjifang9300-1 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Inter

face Ethernet1/49 is down (Link failure)

2020 Jun 19 06:08:39 zhongxinjifang9300-1 %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBER

S_DOWN: Interface port-channel2 is down (No operational members)

2020 Jun 19 06:08:39 zhongxinjifang9300-1 last message repeated 1 time

2020 Jun 19 06:08:39 zhongxinjifang9300-1 %ETHPORT-5-SPEED: Interface Ethernet1/

49, operational speed changed to 1 Gbps

2020 Jun 19 06:08:39 zhongxinjifang9300-1 %ETHPORT-5-IF_DUPLEX: Interface Ethern

et1/49, operational duplex mode changed to Full

2020 Jun 19 06:08:39 zhongxinjifang9300-1 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interfa

ce Ethernet1/49, operational Receive Flow Control state changed to off

2020 Jun 19 06:08:39 zhongxinjifang9300-1 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interfa

ce Ethernet1/49, operational Transmit Flow Control state changed to off

2020 Jun 19 06:08:40 zhongxinjifang9300-1 %ETHPORT-5-SPEED: Interface port-chann

el2, operational speed changed to 1 Gbps

2020 Jun 19 06:08:40 zhongxinjifang9300-1 %ETHPORT-5-IF_DUPLEX: Interface port-c

hannel2, operational duplex mode changed to Full

2020 Jun 19 06:08:40 zhongxinjifang9300-1 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interfa

ce port-channel2, operational Receive Flow Control state changed to off

2020 Jun 19 06:08:40 zhongxinjifang9300-1 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interfa

ce port-channel2, operational Transmit Flow Control state changed to off

2020 Jun 19 06:08:40 zhongxinjifang9300-1 %ETH_PORT_CHANNEL-5-PORT_UP: port-chan

nel2: Ethernet1/49 is up

2020 Jun 19 06:08:40 zhongxinjifang9300-1 %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-

channel2: first operational port changed from none to Ethernet1/49

2020 Jun 19 06:08:40 zhongxinjifang9300-1 %ETHPORT-5-IF_BANDWIDTH_CHANGE: Interf

ace port-channel2,bandwidth changed to 1000000 Kbit

2020 Jun 19 06:08:40 zhongxinjifang9300-1 %ETHPORT-5-IF_UP: Interface Ethernet1/

49 is up in mode trunk

2020 Jun 19 06:08:40 zhongxinjifang9300-1 %ETHPORT-5-IF_UP: Interface port-chann

el2 is up in mode trunk

2020 Jun 19 06:08:40 zhongxinjifang9300-1 %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-

channel2: first operational port changed from Ethernet1/49 to none

2020 Jun 19 06:08:40 zhongxinjifang9300-1 %ETH_PORT_CHANNEL-5-PORT_DOWN: port-ch

annel2: Ethernet1/49 is down

2020 Jun 19 06:08:40 zhongxinjifang9300-1 %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBER

S_DOWN: Interface port-channel2 is down (No operational members)

2020 Jun 19 06:08:40 zhongxinjifang9300-1 %ETHPORT-5-IF_BANDWIDTH_CHANGE: Interf

ace port-channel2,bandwidth changed to 100000 Kbit

2020 Jun 19 06:08:40 zhongxinjifang9300-1 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Inter

face Ethernet1/49 is down (Link failure)

2020 Jun 19 06:08:40 zhongxinjifang9300-1 %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBER

S_DOWN: Interface port-channel2 is down (No operational members)

2020 Jun 19 06:08:41 zhongxinjifang9300-1 last message repeated 1 time

2020 Jun 19 06:08:41 zhongxinjifang9300-1 %ETHPORT-5-SPEED: Interface Ethernet1/

49, operational speed changed to 1 Gbps

2020 Jun 19 06:08:41 zhongxinjifang9300-1 %ETHPORT-5-IF_DUPLEX: Interface Ethern

et1/49, operational duplex mode changed to Full

2020 Jun 19 06:08:41 zhongxinjifang9300-1 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interfa

ce Ethernet1/49, operational Receive Flow Control state changed to off

2020 Jun 19 06:08:41 zhongxinjifang9300-1 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interfa

ce Ethernet1/49, operational Transmit Flow Control state changed to off

2020 Jun 19 06:08:41 zhongxinjifang9300-1 %ETHPORT-5-SPEED: Interface port-chann

el2, operational speed changed to 1 Gbps

2020 Jun 19 06:08:41 zhongxinjifang9300-1 %ETHPORT-5-IF_DUPLEX: Interface port-c

hannel2, operational duplex mode changed to Full

2020 Jun 19 06:08:41 zhongxinjifang9300-1 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interfa

ce port-channel2, operational Receive Flow Control state changed to off

2020 Jun 19 06:08:41 zhongxinjifang9300-1 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interfa

ce port-channel2, operational Transmit Flow Control state changed to off

2020 Jun 19 06:08:41 zhongxinjifang9300-1 %ETH_PORT_CHANNEL-5-PORT_UP: port-chan

nel2: Ethernet1/49 is up

2020 Jun 19 06:08:41 zhongxinjifang9300-1 %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-

channel2: first operational port changed from none to Ethernet1/49

2020 Jun 19 06:08:41 zhongxinjifang9300-1 %ETHPORT-5-IF_BANDWIDTH_CHANGE: Interf

ace port-channel2,bandwidth changed to 1000000 Kbit

2020 Jun 19 06:08:41 zhongxinjifang9300-1 %ETHPORT-5-IF_UP: Interface Ethernet1/

49 is up in mode trunk

2020 Jun 19 06:08:42 zhongxinjifang9300-1 %ETHPORT-5-IF_UP: Interface port-chann

el2 is up in mode trunk

marce1000 · ‎06-21-2020

- It could mean , that you 'only' have a physical connection problem on the link. Disconnect and re-connect at both sides (for instance).

M.

-- Each morning when I wake up and look into the mirror I always say ' Why am I so brilliant ? '
When the mirror will then always repond to me with ' The only thing that exceeds your brilliance is your beauty! '

Sergiu.Daniluk · ‎06-22-2020

Hi @jen2181918

The message: "Module 1 Instance 0 Pool-group buffer 90 percent threshold is exceeded" means that the buffer is reaching the limit of 100% utilization. This happens basically due to a congestion on the interface which is either caused by over utilization of the egress port (1x 10G ingress, 1x 1G egress OR 2x 1G ingress, 1x 1G egress), either by micro bursts of traffic which is eating up the buffers.

I would recommend you monitor the interface utilization and see if it reaches the limits. if yes, then you will need more bandwidth for the egress (port-channeling) or you can apply QoS and prioritize the important traffic. Note, the last option will only avoid traffic drops for higher priority classes.

The link flaps, looks like a separate issue - you will have to perform a L1 check:

+ verify/replace the SFPs

+ verify/replace the cable/fiber

+ verify that the cable/connectors/SFP are properly inserted

+ if you changed all of the above, start moving the SFPs/cables to another switch port OR another NIC.

Stay safe,

Sergiu