Link between C9300-NM-2Q & 9500-48Y4C using QSFP-H40G-CU5M not up

[shills1]

Hi experts,

I have a 9300-48T with C9300-NM-2Q= and a 9500-48Y4C and am linking them with a QSFP-H40G-CU5M=. The interface won't come up. The Compatability matrix states that they should work (unless I'm a muppet and have read it wrong) as they have the min IOS and auto negotiation is turned off (although there is no option to turn off). See config below. Any help very much appreciated as I am stumped

9500-48Y4C Config

interface HundredGigE1/0/52
switchport trunk native vlan 2
switchport mode trunk
logging event trunk-status
logging event spanning-tree
auto qos trust dscp
storm-control broadcast level 5.00
storm-control multicast level 5.00
spanning-tree portfast disable
spanning-tree guard root
service-policy input AutoQos-4.0-Trust-Dscp-Input-Policy
service-policy output AutoQos-4.0-Output-Policy
ip dhcp snooping trust

9300-48T Config

interface FortyGigabitEthernet1/1/1
switchport trunk native vlan 2
switchport mode trunk
logging event trunk-status
logging event spanning-tree
auto qos trust dscp
spanning-tree portfast disable
spanning-tree guard loop
service-policy input AutoQos-4.0-Trust-Dscp-Input-Policy
service-policy output AutoQos-4.0-Output-Policy
ip dhcp snooping trust

Many thanks

Steve

[M02@rt37]

Hello @shills1 

A FEC (Forward Error Correction) issue could indeed be the cause of your link not coming up, especially with high-speed interfaces like 40G and 100G.

FEC is often required to maintain data integrity over long-distance links and can be enabled or disabled on both sides of the connection. If there’s a mismatch in FEC settings between your switches, the link may fail to establish. You should verify that both the 9300-48T and 9500-48Y4C switches are configured with the same FEC type, either FEC rs (required for certain types of media) or no-fec.

So, please check the FEC status using the show interface xx/xx transceiver command to ensure consistency between both ends. If there’s a mismatch in FEC settings, this could prevent the link from coming up, so you may need to explicitly configure FEC on both switches.

[shills1]

Hi,

I've stripped back the config as suggested. and the interface has not come up see no mention of the FEC in the command that you mentioned, any additional suggestions greatly appreciated. See output of the commands below, the only difference I can see is the operational speed any ideas how to change this?? 

fch-9500-48Y4C#sh int hu1/0/52 trans deta sw 1

Diagnostic Monitoring is not implemented.

fch-9500-48Y4C#sh int hu1/0/52 trans deta sw 2

Diagnostic Monitoring is not implemented.

fch-9500-48Y4C#sh int hu1/0/52 trans prop ?

switch Switch

fch-9500-48Y4C#sh int hu1/0/52 trans prop swi 1

Name : Hu1/0/52

Administrative Speed: 40000

Administrative Duplex: full

Administrative Auto-MDIX: on

Administrative Power Inline: N/A

Operational Speed: auto

Operational Duplex: auto

Operational Auto-MDIX: on

Media Type: QSFP 40G CU5M

fch-9500-48Y4C#sh int hu1/0/52 trans prop swi 2

Name : Hu1/0/52

Administrative Speed: 40000

Administrative Duplex: full

Administrative Auto-MDIX: on

Administrative Power Inline: N/A

Operational Speed: auto

Operational Duplex: auto

Operational Auto-MDIX: on

Media Type: QSFP 40G CU5M

fch-9300-48T#sh int for1/1/1 trans det

Diagnostic Monitoring is not implemented.

fch-9300-48T#sh int for1/1/1 trans prop ?

| Output modifiers

<cr> <cr>

fch-9300-48T#sh int for1/1/1 trans prop

Name : Fo1/1/1

Administrative Speed: 40000

Administrative Duplex: full

Administrative Auto-MDIX: on

Administrative Power Inline: N/A

Operational Speed: 40000

Operational Duplex: auto

Operational Auto-MDIX: on

Media Type: QSFP 40G CU5M

fch-9300-48T#sh int for1/1/2 trans prop

Name : Fo1/1/2

Administrative Speed: 40000

Administrative Duplex: full

Administrative Auto-MDIX: on

Administrative Power Inline: N/A

Operational Speed: 40000

Operational Duplex: auto

Operational Auto-MDIX: on

Media Type: QSFP 40G CU5M

fch-9300-48T#sh int for1/1/2 trans det

Diagnostic Monitoring is not implemented.

fch-9300-48T#

[marce1000]

                   - Check logs on both devices just after trying to make a connection
                   - Initially I would use a bare configuration ; meaning remove all configuration statements on the interfaces 
                      in the running config to be able to just test basic connectivity
                   - Check software version on the switches and use the latest (advisory)

 M/

[shills1]

Hi,

I've stripped back the config as suggested. and the interface has not come up, any additional suggestions greatly appreciated.

Both switches are running 17.12.03, the 9500 has network and DNA Advantge and the 9300 has network and DNA essentials. There is nothing showing in the logs and both interfaces recongnise the SFP. I have 2 cables between the switches and neither of them are coming up so it is not a cable problem.

Any help gratefully received.

[Joshqun Ismayilov]

What is the interface status? Both side is UP/UP?


#show interfaces HundredGigE1/0/52 transceiver detail
#show interfaces FortyGigabitEthernet1/1/1 transceiver detail

[shills1]

See response to M02@rt37 for details.

Both sides are showing as FortyGigabitEthernet1/1/2 is down, line protocol is down (notconnect).

Any help you can give is gratefully received.

[shills1]

When I enter the suggested commands it says that diagnostic Monitoring is not implemented on both switches (see below), any ideas.??

fch-9500-48Y4C#sh int hu1/0/52 transceiver detai

Diagnostic Monitoring is not implemented.

fch-9300-48T#sh int fo1/1/1 transceiver

Diagnostic Monitoring is not implemented.

I have just stumbled across this command, is this saying that the SFP is not supported??

fch-9300-48T#sh int trans supported-list

Transceiver Type             Cisco p/n min version supporting DOM

------------------ -------------------------

QSFP_H40G_CU             NONE

[Joshqun Ismayilov]

Also, you can check the license 

show license right-to-use | include network-advantage
show license status

and 

show inventory | include QSFP
show interfaces transceiver supported-list

[shills1]

I think the license may be the issue, one side is advantage the other is essentials. Sould they both be advantage??

The inventory is showing the QSFPs in both routers and the sho int trans supported-list is shown below, it's the same for both switches:

fch-9300-48T#sh int trans supported-list

Transceiver Type             Cisco p/n min version supporting DOM

------------------ -------------------------

QSFP_H40G_CU             NONE

[shills1]

The show licese right-to-use command is not recognised.

ch-9500-48Y4C#sh inve | inc QSFP

NAME: "HundredGigE1/0/49", DESCR: "QSFP 40GE CU5M"

PID: QSFP-H40G-CU5M , VID: V03 , SN: APF281102R3-A

NAME: "HundredGigE1/0/50", DESCR: "QSFP 40GE CU5M"

PID: QSFP-H40G-CU5M , VID: V03 , SN: APF28120DH3-A

NAME: "HundredGigE1/0/52", DESCR: "QSFP 40GE CU5M"

PID: QSFP-H40G-CU5M , VID: V03 , SN: APF280104ED-A

NAME: "HundredGigE2/0/49", DESCR: "QSFP 40GE CU5M"

PID: QSFP-H40G-CU5M , VID: V03 , SN: APF281102R3-B

NAME: "HundredGigE2/0/50", DESCR: "QSFP 40GE CU5M"

PID: QSFP-H40G-CU5M , VID: V03 , SN: APF28120DH3-B

NAME: "HundredGigE2/0/52", DESCR: "QSFP 40GE CU5M"

PID: QSFP-H40G-CU5M , VID: V03 , SN: APF280104E0-A

fch-9500-48Y4C#

Is this saying that the SFP is not supported??

fch-9300-48T#sh int trans supported-list

Transceiver Type             Cisco p/n min version supporting DOM

------------------ -------------------------

QSFP_H40G_CU             NONE

Both switches are running 17.12.03, the 9500 has network and DNA Advantge and the 9300 has network and DNA essentials. There is nothing showing in the logs and both interfaces recongnise the SFP. I have 2 cables between the switches and neither of them are coming up so it is not a cable problem.

Any help gratefully received as I am stumped. All config has been removed from interfaces and they are still down/down

[vishalbhandari]

@shills1 According to me it looks like the issue might be due to a speed mismatch. The QSFP-H40G-CU5M is a 40G direct attach cable, but you’re connecting it between a 100G port on the 9500 and a 40G port on the 9300. The 9500’s 100G port needs to be configured to operate at 40G. Try adding the command speed nonegotiate or, if available, speed 40000 under the 9500 interface. Also, ensure the breakout settings aren’t enabled on the 9500 port, as this could cause issues. If this doesn’t work, try swapping the cable to rule out a physical issue.

[shills1]

Thanks for your response. Unfortunatley none of your suggestions have fixed the issue. 

[shills1]

The only difference I can see is the operational speed any ideas how to change this??  Also how do I check the breakout settings. Any help gratefully received as I'm at a loss

fch-9500-48Y4C#sh int hu1/0/52 trans deta sw 1

Diagnostic Monitoring is not implemented.

fch-9500-48Y4C#sh int hu1/0/52 trans deta sw 2

Diagnostic Monitoring is not implemented.

fch-9500-48Y4C#sh int hu1/0/52 trans prop ?

switch Switch

fch-9500-48Y4C#sh int hu1/0/52 trans prop swi 1

Name : Hu1/0/52

Administrative Speed: 40000

Administrative Duplex: full

Administrative Auto-MDIX: on

Administrative Power Inline: N/A

Operational Speed: auto

Operational Duplex: auto

Operational Auto-MDIX: on

Media Type: QSFP 40G CU5M

fch-9500-48Y4C#sh int hu1/0/52 trans prop swi 2

Name : Hu1/0/52

Administrative Speed: 40000

Administrative Duplex: full

Administrative Auto-MDIX: on

Administrative Power Inline: N/A

Operational Speed: auto

Operational Duplex: auto

Operational Auto-MDIX: on

Media Type: QSFP 40G CU5M

fch-9300-48T#sh int for1/1/1 trans det

Diagnostic Monitoring is not implemented.

fch-9300-48T#sh int for1/1/1 trans prop ?

| Output modifiers

<cr> <cr>

fch-9300-48T#sh int for1/1/1 trans prop

Name : Fo1/1/1

Administrative Speed: 40000

Administrative Duplex: full

Administrative Auto-MDIX: on

Administrative Power Inline: N/A

Operational Speed: 40000

Operational Duplex: auto

Operational Auto-MDIX: on

Media Type: QSFP 40G CU5M

fch-9300-48T#sh int for1/1/2 trans prop

Name : Fo1/1/2

Administrative Speed: 40000

Administrative Duplex: full

Administrative Auto-MDIX: on

Administrative Power Inline: N/A

Operational Speed: 40000

Operational Duplex: auto

Operational Auto-MDIX: on

Media Type: QSFP 40G CU5M