High Utilization and link flaps

[katerina.dardoufa]

Hello all,

we have an environment where two N3K are connected to N7K via 4x1Gbps links forming a Port-channel (attached file). We noticed that when the utilization on the Port-channel connecting to N7K-2 goes over 86%, link 2 goes down. After this comes up, link 4 goes down and comes back up. Any ideas why this is happening? Is it due to bursts?

[Sergiu.Daniluk]

Hi @katerina.dardoufa 

What do you see in 'show logging log' for the time of the problem? What is the reason of link going down? 

Regards,

Sergiu

[katerina.dardoufa]

Hello Sergiu,

the logs only show changes in the link status (up/down). No interface errors or other stuff that would explain the link flaps.

[Sergiu.Daniluk · Accepted solution]

Hi @katerina.dardoufa 

Does it show "Interface X/Y is down (Link failure)" ? If yes, this means that the port does not receive signal from remote side.

You can use this to verify which side is bringing the port down.

If you do not see it on neither N3K or N7K, then some other component brings down the port (like LACP). 

You can use "show system internal ethpm event-history interface ethernet X/Y" to verify from ETHPM perspective (ethernet port manager the software component that interacts with the hardware and upper software component). The outputs might be cryptic, but are intuitive. Look for timeline and "Triggered event + Next state".

Best regards,

Sergiu

[katerina.dardoufa]

Hello Sergiu,

The interface x/y down (link failure) is shown on both sides (N3K-N7K). Today the incident happened again with a link utilization coming up to 94%. As you already observed, I guess that the high utilization prevents the ports from receiving signals from the remote side. I suppose there is nothing we can do, until we increase the interface bandwidth. Any ideas?

I will look into the "show system internal ethpm event-history" as you suggested, but a quick view does not show anything weird.

25) FSM:<Ethernet3/8> Transition at 418290 usecs after Sat May 2 00:33:21 2020
Previous state: [ETH_PORT_FSM_ST_WAIT_BRINGUP]
Triggered event: [ETH_PORT_FSM_EV_PORT_IS_BUNDLE_MEMBER]
Next state: [ETH_PORT_FSM_ST_WAIT_BUNDLE_MEMBER_BRINGUP]

26) FSM:<Ethernet3/8> Transition at 418575 usecs after Sat May 2 00:33:21 2020
Previous state: [ETH_PORT_FSM_ST_WAIT_BUNDLE_MEMBER_BRINGUP]
Triggered event: [ETH_PORT_FSM_EV_BRINGUP_BUNDLE_MEMBER]
Next state: [FSM_ST_NO_CHANGE]

Kind regards,

Katerina

[Sergiu.Daniluk]

Can you share the full output of the ethpm event history + the time when the interfaces went down and up?

Regards,

Sergiu

[katerina.dardoufa]

Hi Sergiu,

attached is the command output from both devices (N3K & N7K).

Kind regards,

Katerina

[paul driver]

Hello
Can you post the output of the following:
sh interface  x/x 
sh interface cap 
sh interface brief  

sh port-channel traffic
sh port-channel usage
sh port-channel database interface port-channel 102
sh port-channel summary 102

sh port-channel load-balance
sh vpc brief
sh vpc consistency-parameters global
sh vpc consistency-parameters vlans

[katerina.dardoufa]

Hello Paul,

attached is the info from the side of the N3K.

The links in question are 1/54/1 on NK-1 and 1/53/1.

Thank you in advance,

Katerina