10G Not Working between Nexus 7K and 3850

[ccomkim]

Help needed!

We have 7004 Nexus switch and 3850 switch(C3850-24S-S) with 10GE Network Module(C3850-NM-2-10G) installed. 

The Nexus 7004 has 1/10 Gbps Ethernet Module, N7K-F248XP-25E.

We bought 2 SFP-10G-LR from OSI. And they both show up from the "sh int status" command.

Eth3/10 10GLink notconnec 5 full 10G 10Gbase-LR

We are trying to connect the two switches with 10Gb Single mode Fibre link, however, the link does not come up. I've checked the fibre cable health with laser checker, and SFP with loopback adapter, all verified healthy. However, when connecting them together, the link always shows "notconnect".

N7K# show int eth3/10
Ethernet3/10 is down (Link not connected)
admin state is up, Dedicated Interface
Hardware: 1000/10000 Ethernet, address: e4c7.2206.8f49 (bia e4c7.2206.8f49)
Description: Link to 3850 Te1/0/3
MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, medium is p2p
Port mode is access
full-duplex, 10 Gb/s, media type is 10G
Beacon is turned off
Auto-Negotiation is turned off
Input flow-control is off, output flow-control is off
Auto-mdix is turned on
Rate mode is dedicated
Switchport monitor is off
EtherType is 0x8100
EEE (efficient-ethernet) : n/a
Last link flapped 00:35:44
Last clearing of "show interface" counters 01:36:20
4 interface resets
Load-Interval #1: 30 seconds
30 seconds input rate 0 bits/sec, 0 packets/sec
30 seconds output rate 0 bits/sec, 0 packets/sec
input rate 0 bps, 0 pps; output rate 0 bps, 0 pps
Load-Interval #2: 5 minute (300 seconds)
300 seconds input rate 0 bits/sec, 0 packets/sec
300 seconds output rate 0 bits/sec, 0 packets/sec
input rate 0 bps, 0 pps; output rate 0 bps, 0 pps
RX
96 unicast packets 280 multicast packets 0 broadcast packets
280 input packets 24973 bytes
0 jumbo packets 0 storm suppression packets
0 runts 0 giants 0 CRC/FCS 0 no buffer
0 input error 0 short frame 0 overrun 0 underrun 0 ignored
0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop
0 input with dribble 0 input discard
0 Rx pause
TX
96 unicast packets 288 multicast packets 0 broadcast packets
280 output packets 24973 bytes
0 jumbo packets
0 output error 0 collision 0 deferred 0 late collision
0 lost carrier 0 no carrier 0 babble 0 output discard
0 Tx pause

N7K#

N7K# show run int eth3/10

!Command: show running-config interface Ethernet3/10
!Time: Tue Jul 9 15:34:45 2019

version 6.2(20)

interface Ethernet3/10
description Link to 3850 Te1/0/3
switchport
switchport access vlan 5
speed 10000
duplex full
no shutdown

N7K#

The log doesn't say much on the Nexus 7K:

2019 Jul 9 14:54:58 TKVANDCSC01 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet3/10 is down (Link failure)

Speed and duplex are set to 10000/Full Manual. 

On the 3850:
3850#sh int te1/1/
TenGigabitEthernet1/1/3 is down, line protocol is down (notconnect)
Hardware is Ten Gigabit Ethernet, address is 00bf.77dd.879f (bia 00bf.77dd.879f)
Description: Link to N7K
MTU 1500 bytes, BW 10000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive not set
Full-duplex, 10Gb/s, link type is force-up, media type is SFP-10GBase-LR
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output 01:56:18, output hang never
Last clearing of "show interface" counters never
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
1112 packets input, 88279 bytes, 0 no buffer
Received 1112 broadcasts (1112 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 1112 multicast, 0 pause input
0 input packets with dribble condition detected
1112 packets output, 88279 bytes, 0 underruns
0 output errors, 0 collisions, 3 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

3850#sh run int te1/1/3
Building configuration...

Current configuration : 164 bytes
!
interface TenGigabitEthernet1/1/3
description Link to N7K
switchport access vlan 5
switchport mode access
speed nonegotiate
end

3850# sh int status | inc te1/1/3

Te1/1/3 Link to N7K notconnect 5 full 10G SFP-10GBase-LR

Please help!!

[Mark Malone]

Hi what software is the 3850 on it must be on this or higher to use that LR SFP ?

SFP-10G-LR 10 Gbps SFP+ 10km Duplex Fiber SMF LC Optic 0 to 70C Y IOS 16.6.1 IOS 16.6.1

[ccomkim]

The IOS version running on 3850 is 16.6.5.

[balaji.bandi]

If you are sure you have correct SFP installed and working and tested.

what is the distance between the switch ?

swap the physical fiber pair? - check do you see light on the fibre cable and twist the patch leads if rquired .

[ccomkim]

I have not tested the SFPs, but I am sure that they are correct SFPs for the switches. They are all new SFPs, and tested with loopback device. All verified working.

The distance between the switches is about 80 meters.

Tried swapping the fibre cables, also tested with laser tester, verified the light passing through. 

Not sure why it's not working.

[Mark Malone]

are you on the correct image for it to be supported , 16.6.1 or higher ?

tx/rx are opposite at each ends connection , so if you see the laser on left side say of the fiber line (dont look into it take a pic off your phone ) it should connect to the SFP port that has no laser seen

[ccomkim]

The IOS running under 3850 is 16.6.5. So it should be supported, I can also see the SFP from the CLI. It's LC-LC and has only one direction to the plug, so I don't think it matters.

[Mark Malone · Accepted solution]

Yes that should be fine ios , The TX and RX lasers matter for it to come up at layer 1 , if the laser on the fiber cable is connected to the SFP laser it wont come up , they must be opposite

im constantly swapping the ends in our DCs to do this to bring them up , that's why the connectors can be clipped off , definitely worth a shot if it has not been done

[balaji.bandi]

The suggestion was not about one direction, Suggestion was to flip the  fiber lead)

[ccomkim]

can't flip the fibre lead because there's a plastic plug attached to it.
https://www.fs.com/products/40203.html?currency=CAD&paid=google_shopping&gclid=CjwKCAjwvJvpBRAtEiwAjLuRPdUxA8yewe-lnP8P1F2_aZJrg5pXJ3-2iLCY9mERCZ0zDL4mvP-yMRoCEcEQAvD_BwE
We have the above cable, and you can see the blue plastic plug at the end, which makes it impossible to flap.

[Mark Malone]

Thats a standard fiber cable that clips off the two cables can pop out and flip around, thats a small plastic bracket on it right at the back where blue part starts on the cable , if your not familiar doing this test it on another cable first so not to damage the one thats already ran if have any spares

[Mark Malone]

https://www.youtube.com/watch?v=-5VH2tZEoDQ

This video shows you how to flip them with that cable, just snap the cables out and cross them over and make sure the light on the cable end does not go into the sfp port with that has the light rx/tx

[ccomkim]

Thanks for the video! I will definitely try tomorrow, and will post the update after.

[ccomkim]

Thanks! I tested it, and it worked like a charm! Thanks again for your help. :)

[balaji.bandi]

Just come acrosss same issue not same models we have resolved with 2 issue different issue at one of customer place. migrating 3850 and 3705 to new Cat 9300.

this was today issue, just before i posted the issue

Link not come up from Cat 9300 to Cat 3750 - one was twisted the cable and come up

another issue took 3 hours, same issue ( but we found 3750 IOS code issue, different 3750 with another IOS resolve the issue

Damm..some time we need to look IOS also consideration.