Solved: Re: High volume vpc interface resets

geoffFx · ‎03-16-2021

Hi,

I have a dual homed nexus 5k & 2k fex configured, even though everything is working I have noticed 2 issues;

sh int br is only showing one of the fex's (ie Eth101/1/1 & etc..)
High volume of interface reset for int po 102 and almost no reset for int po 101 with both 5k's

Any kind assistance would be very much appreciated.

Below are dumps from system;

First 5k;

SR-N5K1# sh fex
FEX FEX FEX FEX Fex
Number Description State Model Serial
------------------------------------------------------------------------
101 FEX0101 Online N2K-C2248TP-1GE SSI163401RR
102 FEX0102 Offline N2K-C2248TP-E-1GE FOX1936GDD1

Second 5k;

SR-N5K2# sh fex
FEX FEX FEX FEX Fex
Number Description State Model Serial
------------------------------------------------------------------------
101 FEX0101 Online N2K-C2248TP-E-1GE FOX1936GDD1
102 FEX0102 Offline N2K-C2248TP-1GE SSI163401RR

First 5k;

SR-N5K1# sh vpc br
Legend:
(*) - local vPC is down, forwarding via vPC peer-link

vPC domain id : 10
Peer status : peer adjacency formed ok
vPC keep-alive status : peer is alive
Configuration consistency status : success
Per-vlan consistency status : success
Type-2 consistency status : success
vPC role : primary
Number of vPCs configured : 70
Peer Gateway : Disabled
Dual-active excluded VLANs : -
Graceful Consistency Check : Enabled
Operational Layer3 Peer-router : Disabled
Auto-recovery status : Enabled (timeout = 240 seconds)

vPC Peer-link status
---------------------------------------------------------------------
id Port Status Active vlans
-- ---- ------ --------------------------------------------------
1 Po15 up 1,20,60,100-150

vPC status
----------------------------------------------------------------------------
id Port Status Consistency Reason Active vlans
------ ----------- ------ ----------- -------------------------- -----------
17 Po17 up success success 20
18 Po18 up success success 20
19 Po19 up success success 20
20 Po20 up success success 20
21 Po21 up success success 20
22 Po22 up success success 20
27 Po27 up success success 60,100-150
32 Po32 down* success success -
101 Po101 up success success -
102 Po102 up success success -

Second 5k;

SR-N5K2# sh vpc br
Legend:
(*) - local vPC is down, forwarding via vPC peer-link

vPC domain id : 10
Peer status : peer adjacency formed ok
vPC keep-alive status : peer is alive
Configuration consistency status : success
Per-vlan consistency status : success
Type-2 consistency status : success
vPC role : secondary
Number of vPCs configured : 70
Peer Gateway : Disabled
Dual-active excluded VLANs : -
Graceful Consistency Check : Enabled
Operational Layer3 Peer-router : Disabled
Auto-recovery status : Enabled (timeout = 240 seconds)

vPC Peer-link status
---------------------------------------------------------------------
id Port Status Active vlans
-- ---- ------ --------------------------------------------------
1 Po15 up 1,20,60,100-150

vPC status
----------------------------------------------------------------------------
id Port Status Consistency Reason Active vlans
------ ----------- ------ ----------- -------------------------- -----------
17 Po17 up success success 20
18 Po18 up success success 20
19 Po19 up success success 20
20 Po20 up success success 20
21 Po21 up success success 20
22 Po22 up success success 20
27 Po27 up success success 60,100-150
32 Po32 down* success success -
101 Po101 up success success -
102 Po102 up success success -

First 5k;

SR-N5K1# sh int po 101
port-channel101 is up

vPC Status: Up, vPC number: 101
Hardware: Port-Channel, address: 547f.ee49.bc34 (bia 547f.ee49.bc34)
MTU 1500 bytes, BW 20000000 Kbit,, BW 20000000 Kbit, DLY 10 usec
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, medium is broadcast
Port mode is fex-fabric
full-duplex, 10 Gb/s
Input flow-control is off, output flow-control is off
Switchport monitor is off
EtherType is 0x8100
Members in this channel: Eth1/13, Eth1/14
Last clearing of "show interface" counters never
144 interface resets
30 seconds input rate 619952 bits/sec, 187 packets/sec
30 seconds output rate 311096 bits/sec, 205 packets/sec
Load-Interval #2: 5 minute (300 seconds)
input rate 523.41 Kbps, 172 pps; output rate 290.59 Kbps, 143 pps
RX
80967355 unicast packets 92116 multicast packets 7597 broadcast packets
81067068 input packets 74136115930 bytes
32709256 jumbo packets 0 storm suppression bytes
0 runts 0 giants 0 CRC 0 no buffer
0 input error 0 short frame 0 overrun 0 underrun 0 ignored
0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop
0 input with dribble 0 input discard
0 Rx pause
TX
149143203 unicast packets 96265 multicast packets 65491 broadcast packets
149304959 output packets 178466375158 bytes
111702715 jumbo packets
0 output error 0 collision 0 deferred 0 late collision
0 lost carrier 0 no carrier 0 babble 0 output discard
0 Tx pause
144 interface resets

SR-N5K1# sh int po 102
port-channel102 is up

vPC Status: Up, vPC number: 102
Hardware: Port-Channel, address: 547f.ee49.bc36 (bia 547f.ee49.bc36)
MTU 1500 bytes, BW 20000000 Kbit,, BW 20000000 Kbit, DLY 10 usec
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, medium is broadcast
Port mode is fex-fabric
full-duplex, 10 Gb/s
Input flow-control is off, output flow-control is off
Switchport monitor is off
EtherType is 0x8100
Members in this channel: Eth1/15, Eth1/16
Last clearing of "show interface" counters never
7850 interface resets
30 seconds input rate 5832 bits/sec, 0 packets/sec
30 seconds output rate 3144 bits/sec, 0 packets/sec
Load-Interval #2: 5 minute (300 seconds)
input rate 6.66 Kbps, 0 pps; output rate 2.47 Kbps, 0 pps
RX
42066 unicast packets 112509 multicast packets 190 broadcast packets
154765 input packets 172630922 bytes
241 jumbo packets 0 storm suppression bytes
0 runts 0 giants 0 CRC 0 no buffer
0 input error 0 short frame 0 overrun 0 underrun 0 ignored
0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop
0 input with dribble 0 input discard
0 Rx pause
TX
11622 unicast packets 96483 multicast packets 195 broadcast packets
108300 output packets 64035306 bytes
241 jumbo packets
0 output error 0 collision 0 deferred 0 late collision
0 lost carrier 0 no carrier 0 babble 0 output discard
0 Tx pause
7850 interface resets

Second 5k;

SR-N5K2# sh int po 101
port-channel101 is up

vPC Status: Up, vPC number: 101
Hardware: Port-Channel, address: 547f.ee47.c214 (bia 547f.ee47.c214)
MTU 1500 bytes, BW 20000000 Kbit,, BW 20000000 Kbit, DLY 10 usec
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, medium is broadcast
Port mode is fex-fabric
full-duplex, 10 Gb/s
Input flow-control is off, output flow-control is off
Switchport monitor is off
EtherType is 0x8100
Members in this channel: Eth1/13, Eth1/14
Last clearing of "show interface" counters never
210 interface resets
30 seconds input rate 881744 bits/sec, 583 packets/sec
30 seconds output rate 843544 bits/sec, 633 packets/sec
Load-Interval #2: 5 minute (300 seconds)
input rate 747.69 Kbps, 235 pps; output rate 622.43 Kbps, 251 pps
RX
93334197 unicast packets 89295 multicast packets 56037 broadcast packets
93479529 input packets 88999080479 bytes
42131511 jumbo packets 0 storm suppression bytes
0 runts 0 giants 0 CRC 0 no buffer
0 input error 0 short frame 0 overrun 0 underrun 0 ignored
0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop
0 input with dribble 0 input discard
0 Rx pause
TX
87606847 unicast packets 90073 multicast packets 56048 broadcast packets
87752968 output packets 74315774652 bytes
42126334 jumbo packets
0 output error 0 collision 0 deferred 0 late collision
0 lost carrier 0 no carrier 0 babble 0 output discard
0 Tx pause
210 interface resets

SR-N5K2# sh int po 102
port-channel102 is up

vPC Status: Up, vPC number: 102
Hardware: Port-Channel, address: 547f.ee47.c216 (bia 547f.ee47.c216)
MTU 1500 bytes, BW 20000000 Kbit,, BW 20000000 Kbit, DLY 10 usec
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, medium is broadcast
Port mode is fex-fabric
full-duplex, 10 Gb/s
Input flow-control is off, output flow-control is off
Switchport monitor is off
EtherType is 0x8100
Members in this channel: Eth1/15, Eth1/16
Last clearing of "show interface" counters never
7714 interface resets
30 seconds input rate 6192 bits/sec, 0 packets/sec
30 seconds output rate 3624 bits/sec, 0 packets/sec
Load-Interval #2: 5 minute (300 seconds)
input rate 5.34 Kbps, 0 pps; output rate 2.54 Kbps, 0 pps
RX
0 unicast packets 108441 multicast packets 0 broadcast packets
108441 input packets 114509102 bytes
0 jumbo packets 0 storm suppression bytes
0 runts 0 giants 0 CRC 0 no buffer
0 input error 0 short frame 0 overrun 0 underrun 0 ignored
0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop
0 input with dribble 0 input discard
0 Rx pause
TX
0 unicast packets 92552 multicast packets 0 broadcast packets
92552 output packets 59629232 bytes
0 jumbo packets
0 output error 0 collision 0 deferred 0 late collision
0 lost carrier 0 no carrier 0 babble 0 output discard
0 Tx pause
7714 interface resets

Consistency checks;

SR-N5K1# show vpc consistency-parameters interface port-channel 101

Legend:
Type 1 : vPC will be suspended in case of mismatch

Name Type Local Value Peer Value
------------- ---- ---------------------- -----------------------
Fex id 1 101 101
Shut Lan 1 No No
STP Port Type 1 Default Default
STP Port Guard 1 None None
STP MST Simulate PVST 1 Default Default
mode 1 on on
Members 1 Ethernet1/13 Ethernet1/13
1
Speed 1 10 Gb/s 10 Gb/s
Duplex 1 full full
Port Mode 1 fex-fabric fex-fabric
MTU 1 1500 1500
Admin port mode 1 unknown unknown
vPC card type 1 Empty Empty
Allowed VLANs -
Local suspended VLANs - - -
SR-N5K1# show vpc consistency-parameters interface port-channel 102

Legend:
Type 1 : vPC will be suspended in case of mismatch

Name Type Local Value Peer Value
------------- ---- ---------------------- -----------------------
Fex id 1 102 102
Shut Lan 1 No No
STP Port Type 1 Default Default
STP Port Guard 1 None None
STP MST Simulate PVST 1 Default Default
mode 1 on on
Members 1 Ethernet1/15 Ethernet1/15
1
Speed 1 10 Gb/s 10 Gb/s
Duplex 1 full full
Port Mode 1 fex-fabric fex-fabric
MTU 1 1500 1500
Admin port mode 1 unknown unknown
vPC card type 1 Empty Empty
Allowed VLANs -
Local suspended VLANs - - -

melchib · ‎03-18-2021

Take a look at your FEX's - it looks like you have them cabled incorrectly. If you swap the cables for FEX 101 and 102 on SR-N5K2 and you should be golden. I'm actually surprised the FEX's are operating.

First 5k;

SR-N5K1# sh fex
FEX FEX FEX FEX Fex
Number Description State Model Serial
------------------------------------------------------------------------
101 FEX0101 Online N2K-C2248TP-1GE SSI163401RR
102 FEX0102 Offline N2K-C2248TP-E-1GE FOX1936GDD1

Second 5k;

SR-N5K2# sh fex
FEX FEX FEX FEX Fex
Number Description State Model Serial
------------------------------------------------------------------------
101 FEX0101 Online N2K-C2248TP-E-1GE FOX1936GDD1 <<------------------------should be FEX102
102 FEX0102 Offline N2K-C2248TP-1GE SSI163401RR <<------------------------should be FEX101

View solution in original post

melchib · ‎03-18-2021

Take a look at your FEX's - it looks like you have them cabled incorrectly. If you swap the cables for FEX 101 and 102 on SR-N5K2 and you should be golden. I'm actually surprised the FEX's are operating.

First 5k;

SR-N5K1# sh fex
FEX FEX FEX FEX Fex
Number Description State Model Serial
------------------------------------------------------------------------
101 FEX0101 Online N2K-C2248TP-1GE SSI163401RR
102 FEX0102 Offline N2K-C2248TP-E-1GE FOX1936GDD1

Second 5k;

SR-N5K2# sh fex
FEX FEX FEX FEX Fex
Number Description State Model Serial
------------------------------------------------------------------------
101 FEX0101 Online N2K-C2248TP-E-1GE FOX1936GDD1 <<------------------------should be FEX102
102 FEX0102 Offline N2K-C2248TP-1GE SSI163401RR <<------------------------should be FEX101

geoffFx · ‎03-18-2021

Many thanks melchib.....I can see my mistake now, second pair of eyes is always good.

I am equally surprised its be working although I have notice weird stuff which is why I started troubleshooting, have identically setup in another DC which is what was confusing me.

Will update cabling later tonight and get back to you.

geoffFx · ‎03-19-2021

Swapping cables resolved the issue although one was required to reboot both switch and reconfigure both switches again.