flapping link, tried everything, and still i can't find what is wrong

[izzinwa]

Got a flapping link between 2 of my MLSwitch (both are Cisco 6800-XL).
int Te7/3 on MLS_1 to int Te7/1 on MLS_2
the fiber connector on MLS_1 is SC, while the fiber connector on MLS_2 is LC
I have changed the fiber cable (Single-mode LC-to-SC) and I have also changed the transceiver/module (10G-BASE-LR on MLS_1 and SFP-10G-SR) but it is still flapping.

interestingly, only the int te7/3 shows flapping in its logs, while int Te7/1 on MLS_2 shows OSPF breaking/re-establishing (due obviously to the flapping link on the other end).

on MLS_1, i assumed int Te7/3 is prolly faulty, so i moved the connection from int Te7/3 to int Te7/2, but after moving it, the flapping still occurs.

# MLS_1 logs showing flapping
Feb 22 10:40:42.832 pst: %LINK-3-UPDOWN: Interface TenGigabitEthernet7/3, changed state to up

Feb 22 10:40:42.932 pst: %LINEPROTO-5-UPDOWN: Line protocol on Interface TenGigabitEthernet7/3, changed state to up

Feb 22 10:55:14.942 pst: %LINK-3-UPDOWN: Interface TenGigabitEthernet7/3, changed state to down

Feb 22 10:55:15.022 pst: %LINEPROTO-5-UPDOWN: Line protocol on Interface TenGigabitEthernet7/3, changed state to down

Feb 22 10:56:19.446 pst: %LINK-3-UPDOWN: Interface TenGigabitEthernet7/2, changed state to down

Feb 22 10:56:20.530 pst: %LINK-3-UPDOWN: Interface TenGigabitEthernet7/2, changed state to up

Feb 22 10:56:20.630 pst: %LINEPROTO-5-UPDOWN: Line protocol on Interface TenGigabitEthernet7/2, changed state to up

Feb 22 11:07:46.111 pst: %LINK-3-UPDOWN: Interface TenGigabitEthernet7/2, changed state to down

# MLS_2 logs showing OSPF breaking due to flapping

Feb 22 10:40:44.945 pst: %OSPFv3-5-ADJCHG: Process 10343, Nbr 192.168.1.1 on TenGigabitEthernet7/1 from LOADING to FULL, Loading Done

Feb 22 10:40:47.705 pst: %OSPF-5-ADJCHG: Process 10343, Nbr 192.168.1.1 on TenGigabitEthernet7/1 from LOADING to FULL, Loading Done

Feb 22 10:55:15.590 pst: %OSPFv3-5-ADJCHG: Process 10343, Nbr 192.168.1.1 on TenGigabitEthernet7/1 from FULL to DOWN, Neighbor Down: Interface down or detached

Feb 22 10:55:15.590 pst: %OSPF-5-ADJCHG: Process 10343, Nbr 192.168.1.1 on TenGigabitEthernet7/1 from FULL to DOWN, Neighbor Down: Interface down or detached

Feb 22 10:57:01.538 pst: %OSPF-5-ADJCHG: Process 10343, Nbr 192.168.1.1 on TenGigabitEthernet7/1 from LOADING to FULL, Loading Done

Feb 22 11:07:51.962 pst: %OSPF-5-ADJCHG: Process 10343, Nbr 192.168.1.1 on TenGigabitEthernet7/1 from LOADING to FULL, Loading Done

The flapping lasts less than a second but it is enough to bring down ospf, and it turn it is disconnecting users who have sessions going on on that link.

Anyone seen anything like this? I am stumped, I have tried everything.
Do you think the problem could be because one end has a SR transceiver (LC connector) while the other end is a LR transceiver (SC connector)?
Should I change from a Single mode to a multimode fiber cable?
really appreciate any suggestions, thanks

[izzinwa]

made a mistake in the post.
both actually use 10Gbase-LR transceivers, but just diff connector types, LC on one end and SC on the other.
connected by single mode fiber cable

[chrihussey]

Can you post the interfaces configurations and the output of "sh interface Te#/# transciever detail" for both sides?

Thank you

[izzinwa]

i have swapped ports on both switches (on the same line card)

i have swapped cables also still flapping

link is connected directly between the switches (no patch, no tap), both switches in the same rack (one above the other)

i have replaced both transceivers (LC connector 10Gbase-LR and SC connector 10Gbase-LR)

and i have even replaced the transceivers from LR (with SM fiber cables) to SR and used MM fiber cables.. still flapping

only thing left to try is add another line card to the chassis and move the link there to see if that will fix it

here is the output for both interfaces

TenGigabitEthernet7/3 is up, line protocol is up (connected)
Hardware is C6k 10000Mb 802.3, address is 3c08.f622.6a40 (bia 3c08.f622.6a40)
Description: nas-bdr-2
Internet address is 192.x.x.x/30
MTU 9216 bytes, BW 10000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 2/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 10Gb/s, media type is 10Gbase-LR
input flow-control is on, output flow-control is on
Clock mode is auto
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output never, output hang never
Last clearing of "show interface" counters never
Input queue: 0/4096/0/142 (size/max/drops/flushes); Total output drops: 51232
Queueing strategy: fifo
Output queue: 0/4096 (size/max)
5 minute input rate 1337000 bits/sec, 169 packets/sec
5 minute output rate 79849000 bits/sec, 6730 packets/sec
L2 Switched: ucast: 0 pkt, 0 bytes - mcast: 0 pkt, 0 bytes
L3 in Switched: ucast: 1785359899996 pkt, 2533184482998080 bytes - mcast: 0 pkt, 0 bytes
L3 out Switched: ucast: 1548219493064 pkt, 2477720482279743 bytes - mcast: 0 pkt, 0 bytes
Additional Counters:
IPv4 in Switched: ucast: 1774374562229 pkt, 2488403831728773 bytes
IPv6 in Switched: ucast: 10985337767 pkt, 44780651269307 bytes
IPv4 + IPv6 in Switched: mcast: 0 pkt, 0 bytes
MPLS in Switched: 0 pkt, 0 bytes
IPv4 out Switched: ucast: 1525747019346 pkt, 2453181978796048 bytes
IPv6 out Switched: ucast: 22472473718 pkt, 24538503483695 bytes
IPv4 + IPv6 out Switched: mcast: 0 pkt, 0 bytes
MPLS out Switched: 0 pkt, 0 bytes
1785442834633 packets input, 2533195821008425 bytes, 0 no buffer
Received 14968365 broadcasts (350 IP multicasts)
56 runts, 1053384483 giants, 0 throttles
6325 input errors, 3360 CRC, 2444 frame, 0 overrun, 0 ignored
0 watchdog, 0 multicast, 0 pause input
0 input packets with dribble condition detected
1548236312592 packets output, 2477716198502693 bytes, 0 underruns
0 output errors, 0 collisions, 2 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 60248 pause output
0 output buffer failures, 0 output buffers swapped out

TenGigabitEthernet7/3 is up, line protocol is up (connected)
Hardware is C6k 10000Mb 802.3, address is 580a.208a.8880 (bia 580a.208a.8880)
Description: te7/1 - nas-dmz
Internet address is 192.xx.xx.x/30
MTU 9216 bytes, BW 10000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 2/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 10Gb/s, media type is 10Gbase-LR
input flow-control is on, output flow-control is on
Clock mode is auto
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:03, output never, output hang never
Last clearing of "show interface" counters never
Input queue: 0/4096/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/4096 (size/max)
5 minute input rate 85852000 bits/sec, 7219 packets/sec
5 minute output rate 2312000 bits/sec, 244 packets/sec
L2 Switched: ucast: 0 pkt, 0 bytes - mcast: 0 pkt, 0 bytes
L3 in Switched: ucast: 33138993 pkt, 31076761299 bytes - mcast: 0 pkt, 0 bytes
L3 out Switched: ucast: 11038917 pkt, 6303172506 bytes - mcast: 0 pkt, 0 bytes
Additional Counters:
IPv4 in Switched: ucast: 0 pkt, 0 bytes
IPv6 in Switched: ucast: 33138993 pkt, 31076761299 bytes
IPv4 + IPv6 in Switched: mcast: 0 pkt, 0 bytes
MPLS in Switched: 0 pkt, 0 bytes
IPv4 out Switched: ucast: 104 pkt, 6656 bytes
IPv6 out Switched: ucast: 11038813 pkt, 6303165850 bytes
IPv4 + IPv6 out Switched: mcast: 0 pkt, 0 bytes
MPLS out Switched: 0 pkt, 0 bytes
33313086 packets input, 31163101788 bytes, 0 no buffer
Received 105471 broadcasts (8059 IP multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 0 multicast, 0 pause input
0 input packets with dribble condition detected
11117430 packets output, 6315510903 bytes, 0 underruns
0 output errors, 0 collisions, 2 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

[chrihussey]

I'm looking for the interface config and transceiver information from both switches:

show run interface t7/3

show interface t7/3 transceiver details

Thanks

[izzinwa]

MLS_1
interface TenGigabitEthernet7/3
description nas-bdr-2
mtu 9216
ip address 192.x.x.x 255.255.x.x
no ip redirects
no ip proxy-arp
ip pim sparse-mode
ip policy route-map foundstone
ip ospf mtu-ignore
ip ospf cost 10
ipv6 address 2001:xxx:xxx:xxx::/64
ipv6 enable
ipv6 nd prefix default no-advertise
ipv6 nd ra suppress
ipv6 ospf 10343 area 0
ipv6 ospf mtu-ignore
flowcontrol send on
hold-queue 4096 in
hold-queue 4096 out


MLS_1#sh int te7/3 transceiver detail
ITU Channel not available (Wavelength not available),
Transceiver is internally calibrated.
Alarms applicable when thresholds or values are not N/A.

mA: milliamperes, dBm: decibels (milliwatts), NA or N/A: not applicable.
++ : high alarm, + : high warning, - : low warning, -- : low alarm.
A2D readouts (if they differ), are reported in parentheses.
The threshold values are calibrated.

High Alarm High Warn Low Warn Low Alarm
Temperature Threshold Threshold Threshold Threshold
Port (Celsius) (Celsius) (Celsius) (Celsius) (Celsius)
---------- ------------------ ---------- --------- --------- ---------
Te7/3 31.0 74.0 70.0 0.0 -4.0

High Alarm High Warn Low Warn Low Alarm
Voltage Threshold Threshold Threshold Threshold
Port (Volts) (Volts) (Volts) (Volts) (Volts)
---------- --------------- ---------- --------- --------- ---------
Te7/3 N/A N/A N/A N/A N/A

High Alarm High Warn Low Warn Low Alarm
Current Threshold Threshold Threshold Threshold
Port (milliamperes) (mA) (mA) (mA) (mA)
---------- ----------------- ---------- --------- --------- ---------
Te7/3 32.5 -- N/A N/A N/A N/A

Optical High Alarm High Warn Low Warn Low Alarm
Transmit Power Threshold Threshold Threshold Threshold
Port (dBm) (dBm) (dBm) (dBm) (dBm)
---------- ----------------- ---------- --------- --------- ---------
Te7/3 -1.0 4.4 0.4 -8.2 -12.2

Optical High Alarm High Warn Low Warn Low Alarm
Receive Power Threshold Threshold Threshold Threshold
Port (dBm) (dBm) (dBm) (dBm) (dBm)
---------- ----------------- ---------- --------- --------- ---------
Te7/3 -1.0 4.4 0.4 -14.4 -18.4





MLS_2
interface TenGigabitEthernet7/3
description te7/1 - nas-dmz
mtu 9216
ip address 192.x.x.x 255.255.x.x
no ip redirects
no ip proxy-arp
ip pim sparse-mode
ip ospf mtu-ignore
ip ospf cost 10
ipv6 address 2001:xxxx:xxxx:xx::/64
ipv6 enable
ipv6 nd prefix default no-advertise
ipv6 nd ra suppress
no ipv6 redirects
ipv6 ospf 10343 area 0
ipv6 ospf mtu-ignore
flowcontrol send on
service-policy input buffering
hold-queue 4096 in
hold-queue 4096 out


MLS_2#sh int te7/3 transceiver det
ITU Channel not available (Wavelength not available),
Transceiver is internally calibrated.
Alarms applicable when thresholds or values are not N/A.

mA: milliamperes, dBm: decibels (milliwatts), NA or N/A: not applicable.
++ : high alarm, + : high warning, - : low warning, -- : low alarm.
A2D readouts (if they differ), are reported in parentheses.
The threshold values are calibrated.

High Alarm High Warn Low Warn Low Alarm
Temperature Threshold Threshold Threshold Threshold
Port (Celsius) (Celsius) (Celsius) (Celsius) (Celsius)
---------- ------------------ ---------- --------- --------- ---------
Te7/3 22.8 75.0 70.0 0.0 -5.0

High Alarm High Warn Low Warn Low Alarm
Voltage Threshold Threshold Threshold Threshold
Port (Volts) (Volts) (Volts) (Volts) (Volts)
---------- --------------- ---------- --------- --------- ---------
Te7/3 3.30 3.63 3.46 3.13 2.97

High Alarm High Warn Low Warn Low Alarm
Current Threshold Threshold Threshold Threshold
Port (milliamperes) (mA) (mA) (mA) (mA)
---------- ----------------- ---------- --------- --------- ---------
Te7/3 31.9 70.0 68.0 2.0 1.0

Optical High Alarm High Warn Low Warn Low Alarm
Transmit Power Threshold Threshold Threshold Threshold
Port (dBm) (dBm) (dBm) (dBm) (dBm)
---------- ----------------- ---------- --------- --------- ---------
Te7/3 -0.2 3.5 0.5 -8.2 -12.2

Optical High Alarm High Warn Low Warn Low Alarm
Receive Power Threshold Threshold Threshold Threshold
Port (dBm) (dBm) (dBm) (dBm) (dBm)
---------- ----------------- ---------- --------- --------- ---------
Te7/3 -0.7 3.5 0.5 -14.1 -18.4

[chrihussey]

Hello,

Not seeing anything out of the ordinary. The MLS_1 interface is showing some errors, runts, giant frames as well as pause outputs. It may be a good idea to clear the interface counters and monitor to better see what is happening. 

It may be a bug, but I can't find anything specific to the 6800-XL.

[izzinwa · Accepted solution]

I moved both interfaces to another slot/linecard and that fixed the issue. no more flapping.

so the issue is with the linecards that the interfaces were on, and I find it weird because they are other "up" ports/interfaces on these linecards, so why was it flapping on just this one link.

this is the weirdest thing I have ever seen.