Cisco Nexus 9000 with FEX modules random module disconnect

sdventures · ‎09-24-2024

Hello everyone,

Simple topology:
Two Nexus 9000 93180YC-EX connected via VPC have 6 fex modules connected to them — 3 fex modules to each.

All of a sudden out of nowhere one morning the modules started to disconnect randomly and later all ports on the switch went down.
I have attached logs from both devices. The switches had to be restarted to restore the service.

What else can we look for?

MHM Cisco World · ‎09-24-2024

There is no attachment

MHM

balaji.bandi · ‎09-24-2024

Two Nexus 9000 93180YC-EX connected via VPC have 6 fex modules connected to them

what is the FEX Module models connected and what nexus OS code running on it.

how are they connected - what mode fiber (SM or MM) - what SFP Module you used to connected these FEX FET-10G or SFP (10G-SR or LR )

Have you checked Layer 1 (physically anything wrong with the cables ? on the first place )

Did any recent upgrades on the Nexus code ?

check some troubleshooting methods :

https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/93x/troubleshooting/guide/b-cisco-nexus-9000-nx-os-troubleshooting-guide-93x/b-cisco-nexus-9000-nx-os-troubleshooting-guide-93x_chapter_010001.html

BB

***** Rate All Helpful Responses *****

How to Ask The Cisco Community for Help

sdventures · ‎09-25-2024

sorry, here are the attachments

fexes are N2K-C2348UPQ-10GE

as for code:

Software
BIOS: version 07.69
NXOS: version 10.2(6) [Maintenance Release]
BIOS compile time: 04/07/2021
NXOS image file is: bootflash:///nxos64-cs.10.2.6.M.bin
NXOS compile time: 8/31/2023 12:00:00 [08/29/2023 21:07:36]

Hardware
cisco Nexus9000 C93180YC-EX chassis
Intel(R) Xeon(R) CPU @ 1.80GHz with 24617872 kB of memory.

balaji.bandi · ‎09-25-2024

as i see the logs high level that does not show any indication of the issue, until you are hitting any bugs here.

have you checked Physical cables, and replaced SFP Modules ?

is the FEX dual homed, do you have port configuration where this is connected on nexus 9K

have you upgraded nexus os recently ? since i see you are latest code ?

BB

***** Rate All Helpful Responses *****

How to Ask The Cisco Community for Help

sdventures · ‎09-28-2024

Nothing was changed. We did the last upgrade 6 months ago from 7 to 10.2.6M. We're thinking about upgrading to 10.3.5M and also upgrading the EPLD package. Would this make sense?

Here is the log:

show install all status
This is the log of last installation.

Verifying image bootflash:/nxos64-cs.10.2.6.M.bin for boot variable "nxos".
-- SUCCESS

Verifying image type.
-- SUCCESS

Preparing "nxos" version info using image bootflash:/nxos64-cs.10.2.6.M.bin.
-- SUCCESS

Preparing "bios" version info using image bootflash:/nxos64-cs.10.2.6.M.bin.
-- SUCCESS

Performing module support checks.
-- SUCCESS

Notifying services about system upgrade.
-- SUCCESS

Compatibility check is done:
Module bootable Impact Install-type Reason
------ -------- -------------- ------------ ------
1 yes disruptive reset Incompatible image for ISSU
101 yes disruptive reset Incompatible image for ISSU
102 yes disruptive reset Incompatible image for ISSU
103 yes disruptive reset Incompatible image for ISSU

Images will be upgraded according to following table:
Module Image Running-Version(pri:alt) New-Version Upg-Required
------ ---------- ---------------------------------------- -------------------- ------------
1 nxos 7.0(3)I7(6) 10.2(6) yes
1 bios v07.56(06/08/2016):v07.56(06/08/2016) v07.69(04/07/2021) yes
101 fex4 7.0(3)I7(6) 10.2(6) yes
102 fex4 7.0(3)I7(6) 10.2(6) yes
103 fex4 7.0(3)I7(6) 10.2(6) yes

Switch will be reloaded for disruptive upgrade.

Install is in progress, please wait.

Performing runtime checks.
-- SUCCESS

Setting boot variables.
-- SUCCESS

Performing configuration copy.
-- SUCCESS

Module 1: Refreshing compact flash and upgrading bios/loader/bootrom.
Warning: please do not remove or power off the module at this time.
-- SUCCESS