Nexus 93108TC-FX3P 10 Gigabit Cat6a with Broadcom BCM57454

[kerstin-534]

We migrated some Dell servers with Broadcom BCM57454 Server NIC connected to 93108TC-FX to Nexus 93108TC-FX3P. Same configuration in NXOS is in place with NXOS version the same on both switches with 10.3.6M.

Some 10 Gigabit Cat6a connections does not come up after migration, cable length is 2 meters.

The problem is that on server side link comes up with 10Gbit and and on switch side link is down. Hardcoding speed to 1 Gigabit helps temporary for links to come up.

Can this a compatibility problem between Nexus 93108TC-FX and Broadcom NIC ?

[AshSe]

Hey @kerstin-534 

The issue you're describing could indeed be related to compatibility or configuration differences between the Broadcom BCM57454 NIC and the Nexus 93108TC-FX3P switch. However, it is not necessarily a direct compatibility issue but could be due to auto-negotiation, signal integrity, or hardware-specific behavior.

Below are some of the actions you can take to troubleshoot the issue:

1. Verify and update the NIC firmware and driver.
2. Check and replace the Cat6a cable if necessary.
3. Ensure auto-negotiation is enabled on both ends, or hardcode both ends to 10 Gbps.
4. Compare port configurations between the Nexus 93108TC-FX and 93108TC-FX3P.
5. Check switch logs and Cisco's Bug Search Tool for any related issues.
6. If the issue persists, escalate to Cisco TAC and/or Broadcom support for further assistance.

Hope This Helps!!!

AshSe

Forum Tips: 

1. Insert photos/images inline - don't attach.
2. Always mark helpful and correct answers, it helps others find what they need.
3. For a prompt reply, kindly tag @name. An email will be automatically sent to the member.

[kerstin-534]

TAC identified an internal defect affecting interoperability between the Nexus N9K-C93108TC-FX3P and the BCM57454 NICs.

The fix for this issue is available in 10.5(2) version.

[lucas-gillingham]

Hello, kerstin-543,

I am having this exact issue with N9K-C93108TC-FX3P and BCM57454 NICs. Can you please identify the bug ID that was associated with this issue? I have updated my Nexus switches to 10.5(2) as per your note but we are still experiencing this problem. 

Lucas

[kerstin-534]

Hello Lucas,

there was no specific information for BUG ID from TAC, because it is internal.

What we have seen after upgrade to 10.5.2F behaviour did not change. Instead server reboot was necessary too. Seems that Broadcom NIC in servers need to be reinitalized after switch upgrade.

Kerstin

[lucas-gillingham]

Hi Kerstin, 

Thank you for sharing this information. We observed the same behaviour - a cold boot of our ESX hosts seems to bring the interfaces back up. However the ports did come online in the past when cold booting the hosts. If the issue seems permanently resolved I will send an update. Thank you, Kerstin! 

Lucas