Can't get fiber connection to come up

[Chris Pohlad-Thomas]

We are trying to run 10GB fiber between two network stacks. The network stacks contain a combination of 3750Gs and 3750Xs. In the 3750Xs that we are trying to connect via fiber, we have the following inventory:

C3KX-NM-10G

2x SFP-10G-LRM

One of the 10G ports works fine and we have that running to a 3750X stack in a telco closet a ways away.

We are running 12.2(55)SE6 on all of the switches and the fiber cable we are using is 10GB (aqua), 50nm, OM3, LC to LC, its 15 meters long.

We have the ports on either side configured as such:

interface TenGigabitEthernet2/1/1

switchport trunk encapsulation dot1q

switchport mode trunk

speed nonegotiate

What are we missing here?

[Marvin Rhoads]

Have you checked the Tx and Rx levels at each end? Those SFPs should support DOM so you can use:

     show int te2/1/1 transceiver detail

I've not fouind it necessary to use nonegotiate on the 10 Gbps SFPs the way we need to on the older 1 Gbps ones; but that shouldn't hurt.

[Chris Pohlad-Thomas]

Ok,

This is one side:

ITU Channel not available (Wavelength not available),

Transceiver is internally calibrated.

mA: milliamperes, dBm: decibels (milliwatts), NA or N/A: not applicable.

++ : high alarm, +  : high warning, -  : low warning, -- : low alarm.

A2D readouts (if they differ), are reported in parentheses.

The threshold values are calibrated.

                              High Alarm  High Warn  Low Warn   Low Alarm

          Temperature         Threshold   Threshold  Threshold  Threshold

Port       (Celsius)          (Celsius)   (Celsius)  (Celsius)  (Celsius)

--------- ------------------  ----------  ---------  ---------  ---------

Te2/1/2     52.0                75.0        70.0         0.0       -5.0

                              High Alarm  High Warn  Low Warn   Low Alarm

           Voltage            Threshold   Threshold  Threshold  Threshold

Port       (Volts)            (Volts)     (Volts)    (Volts)    (Volts)

---------  ---------------    ----------  ---------  ---------  ---------

Te2/1/2    3.26                  3.63        3.46        3.13       2.97

           Optical            High Alarm  High Warn  Low Warn   Low Alarm

           Transmit Power     Threshold   Threshold  Threshold  Threshold

Port       (dBm)              (dBm)       (dBm)      (dBm)      (dBm)

---------  -----------------  ----------  ---------  ---------  ---------

Te2/1/2     -2.2                 3.5         0.5        -6.5      -10.5

           Optical            High Alarm  High Warn  Low Warn   Low Alarm

           Receive Power      Threshold   Threshold  Threshold  Threshold

Port       (dBm)              (dBm)       (dBm)      (dBm)      (dBm)

-------    -----------------  ----------  ---------  ---------  ---------

Te2/1/2     -0.6                 3.5         0.5        -8.0      -12.4

and this is the other:

ITU Channel not available (Wavelength not available),

Transceiver is internally calibrated.

mA: milliamperes, dBm: decibels (milliwatts), NA or N/A: not applicable.

++ : high alarm, +  : high warning, -  : low warning, -- : low alarm.

A2D readouts (if they differ), are reported in parentheses.

The threshold values are calibrated.

                              High Alarm  High Warn  Low Warn   Low Alarm

          Temperature         Threshold   Threshold  Threshold  Threshold

Port       (Celsius)          (Celsius)   (Celsius)  (Celsius)  (Celsius)

--------- ------------------  ----------  ---------  ---------  ---------

Te1/1/1     41.6                75.0        70.0         0.0       -5.0

                              High Alarm  High Warn  Low Warn   Low Alarm

           Voltage            Threshold   Threshold  Threshold  Threshold

Port       (Volts)            (Volts)     (Volts)    (Volts)    (Volts)

---------  ---------------    ----------  ---------  ---------  ---------

Te1/1/1    3.25                  3.63        3.46        3.13       2.97

           Optical            High Alarm  High Warn  Low Warn   Low Alarm

           Transmit Power     Threshold   Threshold  Threshold  Threshold

Port       (dBm)              (dBm)       (dBm)      (dBm)      (dBm)

---------  -----------------  ----------  ---------  ---------  ---------

Te1/1/1     -2.0                 3.5         0.5        -6.5      -10.5

           Optical            High Alarm  High Warn  Low Warn   Low Alarm

           Receive Power      Threshold   Threshold  Threshold  Threshold

Port       (dBm)              (dBm)       (dBm)      (dBm)      (dBm)

-------    -----------------  ----------  ---------  ---------  ---------

Te1/1/1     -1.6                 3.5         0.5        -8.0      -12.4

[Marvin Rhoads]

Hmm at first blush the levels look good.

Are those from both ends of one link? If they are something is amiss - the receive power level on Te2/1/2 is HIGHER than the transmit on Te1/1/1. We all like clean low loss connections but I've never seen a LAN connection actually show power gain!

[Chris Pohlad-Thomas]

Yes, those are from each end of a link. Now, we presently do not have the fiber hooked up. Guess we should try that first and then check the levels again?

[Marvin Rhoads]

Yes indeed - SOMETHING must be hooked up or else the Optical Receive Power should read -40 dBm (= the noise floor or equivalent to no usable optical signal received) at each end.

[Chris Pohlad-Thomas]

Once we get everything hooked back up, what should I be looking for that may indicate an issue?

[Marvin Rhoads]

The primary check is that you are receiving light and that it is within specification levels as show in in the "show int trans detail" output. Once you have tht, it's usually a set and forget item. If any problems are encountered, they should generate syslog events and/or appear in the "show interface" command for the interface(s) in question.

[Chris Pohlad-Thomas]

So, this is what I get for one side of the link:

TenGigabitEthernet1/1/1 is down, line protocol is down (notconnect)

  Hardware is Ten Gigabit Ethernet, address is e4d3.f155.731d (bia e4d3.f155.731d)

  Description: To Rack2-3750X

  MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec,

     reliability 255/255, txload 1/255, rxload 1/255

  Encapsulation ARPA, loopback not set

  Keepalive not set

  Full-duplex, 10Gb/s, link type is force-up, media type is SFP-10GBase-LRM

  input flow-control is off, output flow-control is unsupported

  ARP type: ARPA, ARP Timeout 04:00:00

  Last input never, output never, output hang never

  Last clearing of "show interface" counters never

  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0

  Queueing strategy: fifo

  Output queue: 0/40 (size/max)

  5 minute input rate 0 bits/sec, 0 packets/sec

  5 minute output rate 0 bits/sec, 0 packets/sec

     0 packets input, 0 bytes, 0 no buffer

     Received 0 broadcasts (0 multicasts)

     0 runts, 0 giants, 0 throttles

     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

     0 watchdog, 0 multicast, 0 pause input

     0 input packets with dribble condition detected

     0 packets output, 0 bytes, 0 underruns

     0 output errors, 0 collisions, 4 interface resets

     0 babbles, 0 late collision, 0 deferred

     0 lost carrier, 0 no carrier, 0 PAUSE output

     0 output buffer failures, 0 output buffers swapped out