Nexus C9396PX port freeze issue when 1G module is plugged

[ss1]

Hi,

I have recently faced a strange issue on our Nexus chassis.

Software 
 BIOS: version 07.65 
 NXOS: version 7.0(3)I7(3) 
 BIOS compile time:  09/04/2018 
 NXOS image file is: bootflash:///nxos.7.0.3.I7.3.bin 
 NXOS compile time:  2/12/2018 13:00:00 [02/12/2018 21:13:48] 

Hardware 
 cisco Nexus9000 C9396PX Chassis  
 Intel(R) Core(TM) i3- CPU @ 2.50GHz with 16400992 kB of memory. 
 Processor Board ID SAL1925GZYH

There is some port freeze behavior which prevents ports previously used as 1G to be re-used at 10G at a later stage until switch reload. See what I did:
1. Configured a trunk port without any VLANs - switchport mode trunk / switchport trunk allowed vlan none
2. Plugged a 10Gbase-LR module and looped it. The link went UP (as expected)
3. Disconnected the 10Gbase-LR and plugged a 1000base-LH module and looped it. The link went UP (I'm not sure why it never asked me to configure "speed 1000" as in Nexus 3000 series)
4. Disconnected the 1000base-LH module and connected the 10Gbase-LR again. The link does NOT go UP no matter of the running configuration on the port.
At this stage it only links at 1G speed with 1G modules and refuses to negotiate at 10G. The only solution that is known to me is switch reload. The same issue may be reproduced if the same steps are repeated. I reproduced it three times. 

Nothing suspicious in the logs:

2019 Jun  3 10:56:24 Nexus-C9396PX %ETHPORT-5-IF_HARDWARE: Interface Ethernet1/46, hardware type changed to 10G
2019 Jun  3 10:56:46 Nexus-C9396PX %ETHPORT-5-SPEED: Interface Ethernet1/46, operational speed changed to 10 Gbps
2019 Jun  3 10:56:46 Nexus-C9396PX %ETHPORT-5-IF_DUPLEX: Interface Ethernet1/46, operational duplex mode changed to Full
2019 Jun  3 10:56:46 Nexus-C9396PX %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet1/46, operational Receive Flow Control state changed to off
2019 Jun  3 10:56:46 Nexus-C9396PX %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet1/46, operational Transmit Flow Control state changed to off
2019 Jun  3 10:56:47 Nexus-C9396PX %ETHPORT-5-IF_UP: Interface Ethernet1/46 is up in mode trunk
2019 Jun  3 10:56:49 Nexus-C9396PX %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/46 is down (Link failure)
2019 Jun  3 10:56:52 Nexus-C9396PX %ETHPORT-5-IF_HARDWARE: Interface Ethernet1/46, hardware type changed to No-Transceiver
2019 Jun  3 10:56:52 Nexus-C9396PX %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet1/46 is down (None)

2019 Jun  3 10:57:19 Nexus-C9396PX %ETHPORT-5-IF_HARDWARE: Interface Ethernet1/46, hardware type changed to 1G
2019 Jun  3 10:57:19 Nexus-C9396PX %ETHPORT-5-SPEED: Interface Ethernet1/46, operational speed changed to 1 Gbps
2019 Jun  3 10:57:19 Nexus-C9396PX %ETHPORT-5-IF_DUPLEX: Interface Ethernet1/46, operational duplex mode changed to Full
2019 Jun  3 10:57:19 Nexus-C9396PX %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet1/46, operational Receive Flow Control state changed to off
2019 Jun  3 10:57:19 Nexus-C9396PX %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet1/46, operational Transmit Flow Control state changed to off
2019 Jun  3 10:57:19 Nexus-C9396PX %ETHPORT-5-IF_UP: Interface Ethernet1/46 is up in mode trunk
2019 Jun  3 10:57:20 Nexus-C9396PX %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/46 is down (Link failure)
2019 Jun  3 10:57:20 Nexus-C9396PX %ETHPORT-5-IF_HARDWARE: Interface Ethernet1/46, hardware type changed to No-Transceiver

2019 Jun  3 10:57:21 Nexus-C9396PX %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet1/46 is down (None)
2019 Jun  3 10:57:39 Nexus-C9396PX %ETHPORT-5-IF_HARDWARE: Interface Ethernet1/46, hardware type changed to 10G

# show running-config interface ethernet 1/46 all 

!Command: show running-config interface Ethernet1/46 all
!Time: Mon Jun  3 11:34:09 2019

version 7.0(3)I7(3)

interface Ethernet1/46
  description loop
  lacp port-priority 32768 
  lacp rate normal
  cdp enable
  switchport
  switchport mode trunk
  no switchport monitor
  no switchport dot1q ethertype
  no switchport priority extend
  switchport access vlan 1
  switchport trunk native vlan 1
  switchport trunk allow-multi-tag
  no switchport vlan mapping enable 
  switchport trunk allowed vlan none
  priority-flow-control mode auto
  priority-flow-control watch-dog-interval off
  spanning-tree port-priority 128
  spanning-tree cost auto
  spanning-tree link-type auto
  spanning-tree port type normal
  no spanning-tree bpduguard
  no spanning-tree bpdufilter
  buffer-boost
  flowcontrol receive off
  flowcontrol send off
  delay 1
  snmp trap link-status
  mtu 9198
  link debounce time 100
  no beacon
  speed auto
  duplex auto
  fec auto
  dfe-tuning-delay 100
  negotiate auto
  logging event port link-status default
  logging event port trunk-status default
  storm-control broadcast level 100.00
  storm-control multicast level 100.00
  storm-control unicast level 100.00
  no storm-control action
  bandwidth 10000000
  no bandwidth inherit
  load-interval counter 1 30
  load-interval counter 2 300
  no load-interval counter 3
  no switchport block multicast 
  no switchport block unicast 
  no switchport mac-learn disable
  no shutdown lan
  no switchport autostate exclude
  no shutdown

Any help would be appreciated very much. 

Thank you.

[Mark Malone]

Hi
First thing i would do is change the software could be hitting some strange caveat, you can check the release guide below for any similar caveat issues but it may not be known too, theres a newer recommended software version for that switch nxos.7.0.3.I7.6.bin , whats the far end device your connecting to is it direct or through a patch panel ?

If that doesn't resolve the issue i would open a TAC case you may have a hardware issue

https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/7-x/release/notes/70373_nxos_rn.html

[ss1]

Thank you for your reply. Unfortunately the far end device is the same switch - it's the same SFP looped, however, it's not getting up even if connected to another switch. 

I have just noted that the EPLD image we are running is a bit older. Would you recommend an upgrade and a new tryout?

We are running n9000-epld.7.0.3.I1.2.img dating back to May 2015.

[ss1]

Hi

It seems that the issue was resolved after I upgraded NX-OS from 7.0.3.I7.3 to 7.0.3.I7.6. No EPLD upgrade was necessary.

I'm not sure if anybody else faced the same issue with 9396PX but I hope it would be now stable.

Thank you very much.

[Andrea Testino · Accepted solution]

Sounds like you were hitting  CSCvm49821 or CSCvj04677and have now resolved it by upgrading to NX-OS 7.0(3)I7(6). The DDTS was corrected in I7(4) and later.

Hope that helps!