vPC member port went down causing impact to production

[Nitesh_A]

Hello Experts,

I have 2 switches in vPC domain. switch1 is primary and switch2 is secondary. eth1/47-48 are keepalive links in port-channel. And eth2/23-24 are peer links in port-channel. We had one activity scheduled last night to replace faulty QSFP from eth2/23 from switch1. Below are the steps we followed which caused outage where all member ports went down and came back up. It would be great if someone shed light and help us understand what caused this issue.

1. On switch1, eth2/23 shutdown

2. Unplugged cable from 2/23 and replace failure QSFP (and cable if required)

3.monitor status of port post replacement and plug cable.

4. Bring up the port by "no shutdown" on eth2/23.

Below are the logs from switch1 console.

Thankyou

Nitesh

[Mark Elsen · Accepted solution]

   - @Nitesh_A   From the first screenshot : VPC-2-TRACK_INTFS_DOWN.....
                         You may have tracked objects (interfaces or vlans) configured , VPC will then bring down
                         supporting interfaces when one of these goes down, to prevent asymmetric traffic (e.g.)

                         Check the running configuration with commands such as :
                                           show track
                                           show running-config track

  M.

[Jens Albrecht · Accepted solution]

Hello @Nitesh_A,

based on your screenshots the reason is based on your config:

VPC-TRACK-2-STORM-CFG_CHANGE: Interface Ethernet217/2 is down(config change)
VPC-TRACK2-STORM_FAIL_DOMAIN_1: vPC tracked interface down, suspending all vPCs and keepalives

You configured the peer-link with 2 interfaces for redundancy but some mis-configured track is taking down all vPC member ports and even the keepalive link.

If you provide the config we should be able to spot the reason for this behavior and how to fix this.

HTH!

[julian.bendix]

Everything resolved?

[Nitesh_A]

@Jens Albrecht @julian.bendix @Mark Elsen  Thankyou so much for your valuable feedback. It appears that second peer link in port-channel has low light cable problem. Even though vPC should remain up because of keep-alive. However there was misconfiguration in vPC Boolean operation which caused this issue.