Nexus 5548 to Catalyst 9300 - fiber - not connect

[BL7202]

Hi Everyone, I cannot seem to connect an idf switch (Catalyst 9300) to our main MDF Nexus 5548.

I have verified light at the IDF.  I have for example light on side A of fiber going into switch, verified light on Side B coming out of switch/transceiver.  I have verified the same at the Nexus switch as well.

Do I have a configuration problem or a hardware problem?  I have tried to play with the speed command and on the Catalyst I receive configuration does not match the port capability error.

On the Nexus I can set speed 10000 but if i try auto i get:

NEXUS5548(config-if)# speed auto
ERROR: Ethernet1/31: Configuration does not match the port capability.

Hardware:
Catalyst 9300 with C9300-NM-2Y with SFP-10G-SR-X
Nexus 5548 and SFP-10G-SR-X

CATALYST9300#sh interfaces twentyFiveGigE 1/1/1
TwentyFiveGigE1/1/1 is down, line protocol is down (notconnect)
Hardware is Twenty Five Gigabit Ethernet, address is ccb6.c85e.ca3f (bia ccb6.c85e.ca3f)
MTU 1500 bytes, BW 25000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive not set
Full-duplex, 10Gb/s, link type is auto, media type is SFP-10GBase-SR
input flow-control is on, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output never, output hang never
Last clearing of "show interface" counters never
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec

High Alarm High Warn Low Warn Low Alarm
Temperature Threshold Threshold Threshold Threshold
Port (Celsius) (Celsius) (Celsius) (Celsius) (Celsius)
--------- ----------------- ---------- --------- --------- ---------
Twe1/1/1 27.8 90.0 85.0 -5.0 -10.0

High Alarm High Warn Low Warn Low Alarm
Voltage Threshold Threshold Threshold Threshold
Port (Volts) (Volts) (Volts) (Volts) (Volts)
--------- ----------------- ---------- --------- --------- ---------
Twe1/1/1 3.29 3.63 3.46 3.13 2.97

High Alarm High Warn Low Warn Low Alarm
Current Threshold Threshold Threshold Threshold
Port Lane (milliamperes) (mA) (mA) (mA) (mA)
--------- ---- --------------- ---------- --------- --------- ---------
Twe1/1/1 N/A 7.3 10.5 10.5 2.5 2.5

Optical High Alarm High Warn Low Warn Low Alarm
Transmit Power Threshold Threshold Threshold Threshold
Port Lane (dBm) (dBm) (dBm) (dBm) (dBm)
--------- ---- --------------- ---------- --------- --------- ---------
Twe1/1/1 N/A -2.2 1.70 -1.30 -7.30 -11.31

Optical High Alarm High Warn Low Warn Low Alarm
Receive Power Threshold Threshold Threshold Threshold
Port Lane (dBm) (dBm) (dBm) (dBm) (dBm)
--------- ---- --------------- ---------- --------- --------- ---------
Twe1/1/1 N/A -10.3 2.00 -1.00 -9.91 -13.91

interface TwentyFiveGigE1/1/1
switchport access vlan 10
switchport trunk allowed vlan 10,20,61,62
switchport mode trunk

NEXUS5548# sh interface ethernet 1/31
Ethernet1/31 is down (Link not connected)
Dedicated Interface

Hardware: 1000/10000 Ethernet, address: 0005.73fe.98c6 (bia 0005.73fe.98c6)
Description: TEST-UPLINK
MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, medium is broadcast
Port mode is trunk
auto-duplex, 10 Gb/s, media type is 10G
Beacon is turned off
Input flow-control is off, output flow-control is off
Rate mode is dedicated
Switchport monitor is off
EtherType is 0x8100
Last link flapped never
Last clearing of "show interface" counters 23:44:24
0 interface resets
30 seconds input rate 0 bits/sec, 0 packets/sec
30 seconds output rate 0 bits/sec, 0 packets/sec
Load-Interval #2: 5 minute (300 seconds)
input rate 0 bps, 0 pps; output rate 0 bps, 0 pps
RX
0 unicast packets 0 multicast packets 0 broadcast packets
0 input packets 0 bytes
0 jumbo packets 0 storm suppression bytes
0 runts 0 giants 0 CRC 0 no buffer
0 input error 0 short frame 0 overrun 0 underrun 0 ignored
0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop
0 input with dribble 0 input discard
0 Rx pause
TX
0 unicast packets 0 multicast packets 0 broadcast packets
0 output packets 0 bytes
0 jumbo packets
0 output error 0 collision 0 deferred 0 late collision
0 lost carrier 0 no carrier 0 babble 0 output discard
0 Tx pause

Ethernet1/31
transceiver is present
type is 10Gbase-SR
name is CISCO-AVAGO
part number is SFBR-709ASMZ-CS1
revision is G4.1
serial number is AVD19337137
nominal bitrate is 10300 MBit/sec
Link length supported for 50/125um OM2 fiber is 82 m
Link length supported for 62.5/125um fiber is 26 m
Link length supported for 50/125um OM3 fiber is 300 m
cisco id is --
cisco extended id number is 4

SFP Detail Diagnostics Information (internal calibration)
----------------------------------------------------------------------------
Current Alarms Warnings
Measurement High Low High Low
----------------------------------------------------------------------------
Temperature 34.22 C 90.00 C -10.00 C 85.00 C -5.00 C
Voltage 3.25 V 3.63 V 2.97 V 3.46 V 3.13 V
Current 7.19 mA 10.50 mA 2.50 mA 10.50 mA 2.50 mA
Tx Power -2.31 dBm 1.69 dBm -11.30 dBm -1.30 dBm -7.30 dBm
Rx Power -11.48 dBm - 1.99 dBm -13.97 dBm -1.00 dBm -9.91 dBm
----------------------------------------------------------------------------
Note: ++ high-alarm; + high-warning; -- low-alarm; - low-warning

interface Ethernet1/31
description TEST-UPLINK
switchport mode trunk
spanning-tree port type normal

[marce1000]

 - For starters , make sure that all SFP's are compatible per device model : Cisco Optics-to-Device Compatibility Matrix
    + Check logs on both devices when connection attempts are made.

    Also check this post https://community.cisco.com/t5/switching/cisco-nexus-5k-configuration-does-not-match-the-port-capability/m-p/2328958/highlight/true#M272031

 M.

[BL7202]

Hi marce, according to this, I am okay:

Also when i run the 'speed 10000' command that works just fine.  Doesn't this also show it is good:

sh interface ethernet 1/31
Ethernet1/31 is down (Link not connected)
Dedicated Interface

Hardware: 1000/10000 Ethernet, address: 0005.73fe.98c6 (bia 0005.73fe.98c6)
Description: TEST-UPLINK
MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, medium is broadcast
Port mode is trunk
auto-duplex, 10 Gb/s, media type is 10G

[marce1000]

 - For the nexus I can't find SFP-10G-SR-X in https://tmgmatrix.cisco.com/?npid=617

  M.

[Reza Sharifi]

Hi,

Are you using 3rd party SFPs or Cisco? Also, on the Nexus, no need for auto-negotiation. Just set the speed to 10000.

HTH

[BL7202]

Hi,

Yup transceivers are brand new Cisco SFP-10G-SR-X 
On the Nexus, I can set speed 10000. I just found it odd that 'auto' threw the error.

On the 9300, with the C9300-NM-2Y, I am wondering if that bay doesn't like/cannot accept the 10gig SFP, even though vender said yes... As I cannot set any 'speed' command on that.  they all fail 

[Reza Sharifi]

Can you provide the exact model number for the 9300?

The 10Gig module should work since it shows the correct media type inserted (media type is SFP-10GBase-SR)

Also, can you make sure your fiber is MM? If yes, you may want to swap the strands and test again.

HTH

[BL7202]

Model Number : C9300-48U.   I replaced the C9300-NM-2Y with a C9300-NM-8X.

*Oct 31 17:32:32.356: %PLATFORM_PM-6-FRULINK_INSERTED: 8x10G uplink module inserted in the switch 1 slot 1
*Oct 31 17:32:34.657: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te1/1/1

I fell a bit better about this, but sitll notconnect:     Te1/1/1 notconnect trunk full 10G SFP-10GBase-SR

interface TenGigabitEthernet1/1/1
switchport access vlan 10
switchport trunk allowed vlan 10,20,42,61,62
switchport mode trunk
switchport nonegotiate  <--- added this and rebooted, still no joy.  

Will the fiber patch cables say MM on them somewhere?  Someone else purchased these.. I'll try to verify if i can find out where they came from or if there's a marking on them.

Also, I should NOT flip these, correct?
Side A on fiber cable has light, Pluges into what I marked as A on transceiver

Side B on fibe cable has no light, plugs into what I marked as B on transceiver. B on transceiver has light 

[BL7202]

This above, from my frist post, proves the Nexus side is okay, right?  

Ethernet1/31
transceiver is present
type is 10Gbase-SR
name is CISCO-AVAGO
part number is SFBR-709ASMZ-CS1
revision is G4.1
serial number is AVD19337137
nominal bitrate is 10300 MBit/sec
Link length supported for 50/125um OM2 fiber is 82 m
Link length supported for 62.5/125um fiber is 26 m
Link length supported for 50/125um OM3 fiber is 300 m
cisco id is --
cisco extended id number is 4

SFP Detail Diagnostics Information (internal calibration)
----------------------------------------------------------------------------
Current Alarms Warnings
Measurement High Low High Low
----------------------------------------------------------------------------
Temperature 34.22 C 90.00 C -10.00 C 85.00 C -5.00 C
Voltage 3.25 V 3.63 V 2.97 V 3.46 V 3.13 V
Current 7.19 mA 10.50 mA 2.50 mA 10.50 mA 2.50 mA
Tx Power -2.31 dBm 1.69 dBm -11.30 dBm -1.30 dBm -7.30 dBm
Rx Power -11.48 dBm - 1.99 dBm -13.97 dBm -1.00 dBm -9.91 dBm
----------------------------------------------------------------------------
Note: ++ high-alarm; + high-warning; -- low-alarm; - low-warning

[Reza Sharifi]

In the picture you posted, I see your finger on a yellow fiber patch. Usually yellow is single mode fiber and aqua is MM.

Also, flipping the fibers do not hurt anything. So, you should just try it.

HTH

[BL7202]

Thank you!  I believe we have found our issue!   I am reviewing our order for the fiber that was ran between closets, and the patch cables etc, they are all single mode fiber... Looks like we need to buy 2 Cisco SFP+10g-LR-X, correct  For single mode SFP

[Reza Sharifi]

I am reviewing our order for the fiber that was ran between closets, and the patch cables etc, they are all single mode fiber...

Correct! If this is the case, you need single mode SFPs. If the distance between the closets is less than 500 meters, you only need MM fiber, but what is done is done now. SM SFPs are much more expensive than MM.

HTH