Nexus 5672UP input/output errors increasing

[zizou6500]

Hi Community,

I'm running into an issue with a 5k switch for several days, I'm getting input/output errors on the uplink ports.

There is mainly 1 uplink port going to the edge device + 1 uplink port going to the peer 5k switch + 4 uplinks going to the FEX devices (N2k).

I already started with changing cable, SFP, and moving ports. However, this didn't help at all.

Can anyone help in this case, please?

[balaji.bandi]

what nexus code running, can you post the show interface ethx/x (the interface having errors) - also post other device same output.

also worth refer below thread :

https://community.cisco.com/t5/routing/output-errors-on-port-nexus-5020/td-p/3339391

[zizou6500]

Hi @balaji.bandi 

Thank you for your reply.

Software
BIOS: version 2.1.7
Power Sequencer Firmware:
Module 1: v4.0
Module 1: v4.0
Fabric Power Sequencer Firmware: Module 1: version v4.0
Microcontroller Firmware: version v0.0.0.15
QSFP Microcontroller Firmware:
Module 1: v2.0.0.0
CXP Microcontroller Firmware:
Module not detected
kickstart: version 7.2(1)N1(1)
system: version 7.2(1)N1(1)
BIOS compile time: 06/16/2016
kickstart image file is: bootflash:///n6000-uk9-kickstart.7.2.1.N1.1.bin
kickstart compile time: 10/14/2015 19:00:00 [10/15/2015 06:02:08]
system image file is: bootflash:///n6000-uk9.7.2.1.N1.1.bin
system compile time: 10/14/2015 19:00:00 [10/15/2015 06:02:52]

=============

# sh int eth1/41
Ethernet1/41 is up
Dedicated Interface

Belongs to Po107
MTU 1500 bytes, BW 10000000 Kbit,, BW 10000000 Kbit, DLY 10 usec
reliability 245/255, txload 1/255, rxload 1/255
Encapsulation ARPA, medium is broadcast
Port mode is fex-fabric
full-duplex, 10 Gb/s, media type is 10G
Beacon is turned off
Input flow-control is off, output flow-control is off
Rate mode is dedicated
Switchport monitor is off
EtherType is 0x8100
Last link flapped 00:18:44
Last clearing of "show interface" counters 00:45:41
1 interface resets
30 seconds input rate 528416 bits/sec, 148 packets/sec
30 seconds output rate 383688 bits/sec, 276 packets/sec
Load-Interval #2: 5 minute (300 seconds)
input rate 544.43 Kbps, 125 pps; output rate 495.91 Kbps, 261 pps
RX
385920 unicast packets 12890 multicast packets 187 broadcast packets
453051 input packets 180239096 bytes
34097 jumbo packets 0 storm suppression bytes
0 runts 0 giants 54054 CRC 0 no buffer
54054 input error 0 short frame 0 overrun 0 underrun 0 ignored
0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop
0 input with dribble 0 input discard
0 Rx pause
TX
420972 unicast packets 498015 multicast packets 13116 broadcast packets
937163 output packets 207331121 bytes
23355 jumbo packets
2003 output error 0 collision 0 deferred 0 late collision
0 lost carrier 0 no carrier 0 babble 0 output discard
0 Tx pause

[balaji.bandi]

Eth1/41 is part of port-channel connected to FEX

is this nexus in vPC ? is the FEX is dual-homed uplink?

what is the other interface part of port-channel ?

Can you post same information from other interface ? does it show same errors /

Can you post - 

show interface  Eth1/41 counters detailed all

[zizou6500]

Hi @balaji.bandi @MHM Cisco World 

The issue seems to be related to the Fex device, please see the below errors:

2022 Oct 25 06:47:04.959 hkghke-5k-02 %VMM-2-VMM_SERVICE_ERR: VDC1: Service SAP 175 for slot 40 (FEX chassis 107) returned error 0x40290126 (Communication with port client failed.) in if_bind sequence
2022 Oct 25 06:47:04.959 hkghke-5k-02 %IM-3-IM_RESP_ERROR: Component MTS_SAP_VMM opcode:MTS_OPC_IM_IF_VDC_BIND in vdc:1 returned error:Communication with port client failed.
2022 Oct 25 06:47:04.960 hkghke-5k-02 %IM-3-IM_SEQ_ERROR: Error (Communication with port client failed.) while communicating with component <Internal Error> opcode:N/A (for:RID_MODULE: 40)

[balaji.bandi]

Eth1/41 is part of port-channel connected to FEX

is this nexus in vPC ? is the FEX is dual-homed uplink?

what is the other interface part of port-channel ?

Can you post same information from other interface ? does it show same errors /

Can you post - 

show interface  Eth1/41 counters detailed all

Rather jumping, i expect to address the questions asked before.

also what FEX Model is this ? 

can you post  - show fex 107

Also worth looking at some Bugs below :

https://bst.cisco.com/bugsearch/bug/CSCuw48505

[zizou6500]

Hi Balaji,

I shutdown the physical port with Fex 107, as it was impacting the production environment so I can't provide all details.

However please see the below:

# sh int counter err | egrep "Port|--|\B [1-9]" | egrep -v "\ 0\ *--\ *0\ *0\ *0\ *0"
--------------------------------------------------------------------------------
Port          Align-Err    FCS-Err   Xmit-Err    Rcv-Err  UnderSize OutDiscards
--------------------------------------------------------------------------------
Eth1/1                0          0    1795208          0          0           0
Eth1/2                0          0     114651          0          0           0
Eth1/9                0          0     190495          0          0           0
Eth1/17               1         12          0         13          0           0
Eth1/33               0          0     101461          0          0           0
Eth1/41               0      91130          0      91130          0           0
Po17                  1         12          0         13          0           0
Po101                 0          0    1795208          0          0           0
Po103                 0          0     114651          0          0           0
Po105                 0          0     190495          0          0           0
Po107                 0      91141          0      91141          0           0
Po109                 0          0     101461          0          0           0

The Fex model is Extender Model: N2K-C2232PP-10GE, this is a capture from the Fex log( this is when we tried to change the Fex Uplink port):

10/24/2022 20:35:38.686629: Module Online Sequence
10/24/2022 20:36:32.870033: Aborting lc insert sequence
10/24/2022 20:36:32.870100: Deleting route to FEX
10/24/2022 20:36:32.878377: Module disconnected
10/24/2022 20:36:32.879500: Module insert failed
10/24/2022 20:36:32.880375: Module Offline Sequence
10/24/2022 20:36:32.992234: Module Offline

The Fex was flapping continuously. Please note that this Fex is a single-homed Uplink.

[MHM Cisco World]

https://community.cisco.com/t5/switching/nexus-5672-output-to-fex-errors/td-p/3221074

you have issue with jumbo frame in NSK, 
check if the frame is come from jumbo to 1500 mtu port 

[MHM Cisco World]

can we see show interface, 
when need to see error