Fiber link going down without a reason

[db1]

Hi Guys,

I have a core switch (3750-x stack) connected to an access switch (3560x) with a 10GB fiber link. I also have a backup 4x1GB port channel. The problem is that I can make the 10G link work, but it just turns off after a few hours without a clear reason, and then I have to shut the port down and turn it back on on the 3750-x side to make it work again for a few hours. This is the best case scenario, last week I had to remove config from both ports and reconfigure them again to make the link work.

Here is the config on 3750-x:

interface TenGigabitEthernet2/1/2
 switchport trunk allowed vlan 91,101,102
 switchport trunk encapsulation dot1q
 switchport trunk native vlan 999
 switchport mode trunk
 udld port aggressive
 spanning-tree portfast network
end

And on 3560-x:

interface TenGigabitEthernet1/1
 switchport trunk allowed vlan 91,101,102
 switchport trunk encapsulation dot1q
 switchport trunk native vlan 999
 switchport mode trunk
 udld port aggressive
 spanning-tree portfast network
end

The ports are not in err-disabled state, and there are no errors:

TenGigabitEthernet2/1/2 is down, line protocol is down (notconnect)
  Hardware is Ten Gigabit Ethernet, address is a46c.2aab.35b6 (bia a46c.2aab.35b6)
  Description: connected to DBAMSSW03
  MTU 9126 bytes, BW 10000000 Kbit/sec, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive not set
  Full-duplex, 10Gb/s, link type is auto, media type is SFP-10GBase-SR
  input flow-control is off, output flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 3d14h, output 3d14h, output hang never
  Last clearing of "show interface" counters 4d15h
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     535657539 packets input, 990801863713 bytes, 0 no buffer
     Received 374735 broadcasts (322986 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 322986 multicast, 0 pause input
     0 input packets with dribble condition detected
     485084056 packets output, 643747320202 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out

TenGigabitEthernet1/1 is down, line protocol is down (notconnect)
  Hardware is Ten Gigabit Ethernet, address is 44d3.ca1e.e7b5 (bia 44d3.ca1e.e7b5)
  Description: connected to DBAMSSW01
  MTU 9126 bytes, BW 10000000 Kbit/sec, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive not set
  Full-duplex, 10Gb/s, link type is auto, media type is SFP-10GBase-SR
  input flow-control is off, output flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 3d14h, output 3d14h, output hang never
  Last clearing of "show interface" counters 2w6d
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     805079008 packets input, 1006982654176 bytes, 0 no buffer
     Received 2978817 broadcasts (967136 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 967136 multicast, 0 pause input
     0 input packets with dribble condition detected
     955498446 packets output, 1702746280144 bytes, 0 underruns
     0 output errors, 0 collisions, 8 interface resets
     0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out

I have the same exact setup going to the other access switch and it works fine. I can think of a few things to troubleshoot (change the SFP modules + cable from the other switch), but I was wondering if you have any idea what else I could check / debug to see why the problem is happening?

Thanks,
Jakub

[Julio E. Moisa]

Hi 

Have you tried to replace the fiber, SFP or port?

[Mark Malone]

Its a layer 1 issue so your on the right path with the things you said to check , debug wont really help as its a l1 issue so far as its down/down , one thing you could try to is flip the fibres around in case the transmit and receive are the wrong way around , that's fixed a couple of my l1 issues in the past

[db1]

I am waiting for additional SFP modules to arrive this week, when they do I will replace both SFP's and the cable to see what will happen. I will update when I know more.

[db1]

Just wanted to say that in the end I replaced both SFP modules + cable and it's been working fine since. I did not pinpoint which element was faulty. I will have to change them back one by one and see when it fails again...

[Georg Pauwen]

Hello,

on a side note, udld aggressive may cause this behavior. In aggressive mode, the failure to echo UDLD packets, which can be caused by CPU spikes, may trigger the error disable state (this is called a false positive). UDLD normal mode might work better...