Nexus ports down

[shaikh.zaid22]

We have a fortigate firewall which has 2 ports connected to cisco catalyst switch as PO.

2 ports connected to 2 different cisco nexus 9300 sw

2 ports to cisco 9200 switch.

Now, we got a new firewall, as soon as we unplugged the cables and connected to new firewall unit we are seeing ports connected to Nexus port 48 on both switch is Down.

However, other ports connected to catalyst and access sws are up and running.

I tried toggling the Nx sw ports and restarted the firewall as well but of no use.

So, just wanted to know as i have experienced in past sometimes Nx sw ports required to re-configure.

Note: Nexus ports are 25G ports.

Need suggestions

[M02@rt37]

Hello @shaikh.zaid22 

Possibble mismatch between your new firewall and the nexus 9300.

You must ensure that both sides of the 25G link agree on FEC settings...

[shaikh.zaid22]

Can you share any article FEC settings..

Also, FYI.. other 2 ports connected to Catalyst sw is also 25G which is UP. The ones which are on Nexus are down.

[M02@rt37]

@shaikh.zaid22 

Cisco c9200 and nexus 9300 switches support different FEC behaviors...especialy on 25G interfaces, and this can absolutely affect link negotiation with devices like firewalls.

Check datasheets...

https://www.cisco.com/c/en/us/products/collateral/switches/nexus-9000-series-switches/datasheet-c78-742284.html

[marce1000]

    - Check logs     on the nexus switches when connection attempts are made ,

  M.

[shaikh.zaid22]

You mean on show logging on nexus?

Since it is multimode fiber connection having both devices in the same rack, i can see SFP lights and fiber lights coming-in in. 

[marce1000]

     @shaikh.zaid22                >....You mean on show logging on nexus?
                                Yes, try  to use similar commands on the firewall too when connection attempts are made,
                                                          (checking statuses of links and investigating logs)

  M.

[Rich R]

Hardware and software specifics are relevant - you should always provide this info.
Check out:
https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/7-x/interfaces/configuration/guide/b_Cisco_Nexus_9000_Series_NX-OS_Interfaces_Configuration_Guide_7x/b_Cisco_Nexus_9000_Series_NX-OS_Interfaces_Configuration_Guide_7x_chapter_010....

https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/93x/interfaces/configuration/guide/b-cisco-nexus-9000-nx-os-interfaces-configuration-guide-93x/b-cisco-nexus-9000-nx-os-interfaces-configuration-guide-93x_chapter_01110.html#id_78...

https://community.fortinet.com/t5/FortiGate/Troubleshooting-Tip-25-Gigabit-Ethernet-connection-between/ta-p/189603

[shaikh.zaid22]

@Rich R 

Thanks for the response and useful articles. Going by the article and verifying my configurations below are the comments:

Firstly, i can see in my cisco nexus port48 configuration i can see "fec off" is applied, unfortunately i dont understand what it means, since it was configured by the build team.

Secondly, based on the second cisco article i can see it is mentioned under 25G Auto negotiation on copper based 25g interface which is what exactly in my case as well requires to configure negotiation of 25000 auto negotiate in the interface config. SO this i will try and see how it goes.

Finally, the fortigate article reers to manuall configure speed 25000 on both fortigate and nexus to work properly.

I will try and update you all.

If any other recommendations please feel free to write.   

[marce1000]

  - @shaikh.zaid22 "FEC off" on a fiber port connection means that Forward Error Correction (FEC) is disabled. FEC is a technique used in optical networking to detect and correct errors in data transmission without requiring retransmission , it's advisable or probably required that the connections (ports) at both ends use the same FEC setting

                             As far as auto negotiation is concerned , for fiber it is advised to set the speed fixed for the port according to the capabilities of the SFP (of course port at both ends must support the same speed)

  M.

[shaikh.zaid22]

@marce1000 Thanks for the response, i will try it on Monday and update

[marce1000 · Accepted solution]

   - @shaikh.zaid22                 >.... i will try it on Monday and update
                                 Ok, doing some research for the time  being without charge (LOL!)
                                 You may find these two links interesting, concerning the use of FEC on a fortigate :
                                              https://community.fortinet.com/t5/FortiGate/Technical-Tip-Changes-in-Forward-Error-Correct-FEC-settings/ta-p/231356
                                              https://docs.fortinet.com/document/fortigate/7.0.0/new-features/740127/allow-only-supported-fec-implementations-on-10g-25g-40g-and-100g-interfaces-7-0-4

                                  Probably best ,  for starters to try disabling fec at both ends, if possible.

  M.

[shaikh.zaid22]

@marce1000 thanks marce..

btw, i was also thinking why did the other 25G ports on fortigate connected to a cisco catalyst sw does not have this affect. It seems this is only specific to FEC feature on nexus. if i consider this analogy then, the issue i  not on FGT side.

[marce1000]

 - @shaikh.zaid22   That may be but sync the fec settings on fortigate and nexus anyway , before making the
                               connection,

  M.

[M02@rt37]

Hello @shaikh.zaid22 

This is a known interoperability issue with Nexus 9300 and 25G links, especially when using DAC cables or certain optics. Nexus switches require a matching FEC configuration on 25G interfaces. Some platform (like Fortigate or Catalyst) may autonegotiate or tolerate no-FEC, while nexus might expect RS-FEC or FC-FEC explicitly set...like the source I provided to you explain...