mysterious 4500-x to 6500 fiber connection issue??

baselzind · ‎05-13-2015

im having a strange issue with a fiber connection between 4500-x and 6500 , it is a 10 gig connection with both LR sfb on both 4500 and 6500 and a sm cable , the issue is that the port on the 4500-x keep switching between connected and notconnected but the the port on the 6500 stays connected , when i check both interfaces on 4500 and 6500 it have gradually increasing crc errors and input errors on both switches , i tried replacing the cable , the sfb and the port with no avail , i tried switchport non-negotiate also tries setting it to trunk and access and still the same story , even when i connect the 4500-x with the same sfb to 2960-x sfb 10 gig as well there is no issues , also connecting the 2960-x to the 6500 shows no issues using the same 10 gig cable and sfb , please help as i ran out of solutions

Mark Malone · ‎05-13-2015

have you tried to hardcode the speed and duplex settings , does it happen on all 10gig connections between the 65 and 45 or just the same interface each time

crc and inputs together usually infer there's a layer 1 issue , how far away are these devices from each other attenuation can be a cause for this as well , whats the reliability show on the interface when connected is it 255/255

you have ruled out sfp and cable by using different equipment so i would check these next

If there sfps that support DOM you can you can do a show interface transceiver

baselzind · ‎05-13-2015

regarding the duplex i think u cant change it for sfb it is always full-duplex and yes i entered the bandwidth under the interface and it didnt work i even lowered the bandwidth , also yes i tried three ports on both switches and the same issue presist ,regarding cable im using a 1-1.5 meter back to back cable between the two devices , but i havnt checked the reliability.

Mark Malone · ‎05-13-2015

yes sorry fibre you dont have duplex ,can you post the show interface for each side

the exact type of sfps and the ios currently in use in on 45

where are the crcs mostly incrementing on 45 or 65 side

the fact the issue does not exist using same cabling and sfps with 2960 & 6500 would indicate issue is on the 4500 whether its hardware or software support problem

baselzind · ‎05-19-2015

the interface and the version of the 6500 is:

1-version >

ROM: System Bootstrap, Version 12.2(17r)S2, RELEASE SOFTWARE (fc1)

cisco WS-C6509-E (R7000) processor (revision 1.2) with 458720K/65536K bytes of memory.

Processor board ID SMC0934007G

SR71000 CPU at 600Mhz, Implementation 0x504, Rev 1.2, 512KB L2 Cache

Last reset from power-on

21 Virtual Ethernet interfaces

132 Gigabit Ethernet interfaces

32 Ten Gigabit Ethernet interfaces

1917K bytes of non-volatile configuration memory.

8192K bytes of packet buffer memory.

65536K bytes of Flash internal SIMM (Sector size 512K).

Configuration register is 0x2102

interface of the 6500 switch>

KIA-Core-Switch(config-if)#do sh int t9/16
TenGigabitEthernet9/16 is up, line protocol is up (connected)
Hardware is C6k 10000Mb 802.3, address is 6c41.6a60.0f67 (bia 6c41.6a60.0f67)
MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 10Gb/s
Transport mode LAN (10GBASE-R, 10.3125Gb/s), media type is 10Gbase-LR
input flow-control is on, output flow-control is off
Clock mode is auto
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:04, output 00:00:17, output hang never
Last clearing of "show interface" counters 02:10:38
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 215000 bits/sec, 98 packets/sec
216 packets input, 112169 bytes, 0 no buffer
Received 216 broadcasts (216 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 0 multicast, 0 pause input
0 input packets with dribble condition detected
132090 packets output, 41503844 bytes, 0 underruns
0 output errors, 0 collisions, 4 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 PAUSE output
0 output buffer failures, 0 output buffers swapped out

version and interface on the 4500:

version>

#sh ver
Cisco IOS Software, IOS-XE Software, Catalyst 4500 L3 Switch Software (cat4500e
-UNIVERSALK9-M), Version 03.07.01.E RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2015 by Cisco Systems, Inc.
Compiled Tue 28-Apr-15 12:27 by prod_rel_team

Cisco IOS-XE software, Copyright (c) 2005-2015 by cisco Systems, Inc.
All rights reserved. Certain components of Cisco IOS-XE software are
licensed under the GNU General Public License ("GPL") Version 2.0. The
software code licensed under GPL Version 2.0 is free software that comes
with ABSOLUTELY NO WARRANTY. You can redistribute and/or modify such
GPL code under the terms of GPL Version 2.0.
(http://www.gnu.org/licenses/gpl-2.0.html) For more details, see the
documentation or "License Notice" file accompanying the IOS-XE software,
or the applicable URL provided on the flyer accompanying the IOS-XE
software.

ROM: 15.0(1r)SG11
uptime is 3 days, 1 hour, 22 minutes
System returned to ROM by power-on
System image file is "bootflash:cat4500e-universalk9.SPA.03.07.01.E.152-3.E1.bin
"
Jawa Revision 2, Winter Revision 0x0.0x41

Last reload reason: power-on

interface on 4500 with the issue appearing on the interfaces>

#sh int t1/5
TenGigabitEthernet1/5 is up, line protocol is up (connected)
Hardware is Ten Gigabit Ethernet Port, address is 84b8.0201.fe44 (bia 84b8.020
1.fe44)
MTU 1500 bytes, BW 10000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 10Gb/s, link type is auto, media type is 10GBase-LR
input flow-control is on, output flow-control is on
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output never, output hang never
Last clearing of "show interface" counters never
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 183000 bits/sec, 86 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
72635 packets input, 25174291 bytes, 0 no buffer
Received 10766 broadcasts (2813 multicasts)
0 runts, 0 giants, 0 throttles
3 input errors, 3 CRC, 0 frame, 0 overrun, 0 ignored
0 input packets with dribble condition detected
167 packets output, 86673 bytes, 0 underruns
0 output errors, 0 collisions, 2 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out

4500-switch#sh int t1/5
TenGigabitEthernet1/5 is down, line protocol is down (notconnect)
Hardware is Ten Gigabit Ethernet Port, address is 84b8.0201.fe44 (bia 84b8.020
1.fe44)
MTU 1500 bytes, BW 10000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, Auto-speed, link type is auto, media type is 10GBase-LR
input flow-control is off, output flow-control is off
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output never, output hang never
Last clearing of "show interface" counters never
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 185000 bits/sec, 87 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
72966 packets input, 25274041 bytes, 0 no buffer
Received 10845 broadcasts (2828 multicasts)
0 runts, 0 giants, 0 throttles
3 input errors, 3 CRC, 0 frame, 0 overrun, 0 ignored
0 input packets with dribble condition detected
168 packets output, 87192 bytes, 0 underruns
0 output errors, 0 collisions, 2 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out

Mark Malone · ‎05-19-2015

They need to be LC-LC connections or

The ports can be configured with either SFP transceivers for 1-GB operation or SFP+ transceivers for 10-GB operation. Both transceivers use LC-type connectors (optical) or RJ-45 (copper).

ttcroziercisco · ‎05-19-2015

Our connection is an LC (4500-x) to SC (4900). We didn't have an issue for the 1st 8 months in production. Is that due to software\hardware limitations, or is that a documented requirement?

Mark Malone · ‎05-19-2015

That was taken out of a Cisco document for 4500 series switches 10GB requirements, i also came across it in another forum that they must be LC either side to achieve 10GB and SFP+ , you could check it with TAC but it looks to be hardware specific

ronbuchalski · ‎05-19-2015

markmalone2008,

I don't know where you got this information from, but we have several areas where we connect switches together using 10G, where one end is LC and the other end is SC. We have done it with multimode fiber and SR optics, as well as with singlemode fiber and LH optics. It's never been a problem.

Can you post a link to the Cisco document that says this isn't supported?

To the OP, can you post your port configurations? Also, when you say that the port(s) go down, how long do they remain down? Do they bounce and come back up?

Are you setting your ports to negotiate everything, or are you hard setting them to encapsulation (if it's an option), mode (access or trunk), turn off negotiation?

Have you tried configuring them as Layer 3 ports? If so, does the problem still occur?

-rb

ronbuchalski · ‎05-19-2015

baselzind,

Have you verified that your patch cable is certified for 10G operation?

Also, since you are using Single Mode optics, you need to be careful with optical power levels. With a short fiber optic cable, you may be overloading the optical receiver. In the long run this will burn out your SFP. In the short run, it may very well be causing the errors you are seeing.

It is very important to calculate an optical power budget for single mode connections, and to use attenuators to make sure that the receive levels are within specifications for the transceivers. Typically, attenuating it so the receive value is slightly above the middle range for the transceiver will be best.

-rb

ttcroziercisco · ‎06-03-2015

Ron,

How do you modify the optical levels?

Fyi - this case has been escalated within Cisco and hope to have a game plan in the next day or two

Mark Malone · ‎05-19-2015

I didnt say it was not supported and not the same for different switches just what i found online as the same cabling and sfps work with other switches fine and to check with TAC, this is one of the docs if your running sup7 series in 4500 series you need to use lc-lc

https://supportforums.cisco.com/discussion/11629431/catalyst-4510r-e-10gb-sfp-not-working

http://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst4500/hardware/configuration/notes/OL_23144.html#wp153042

Catalyst 4500 L3 Switch Software (cat4500e
-UNIVERSALK9-M) - See more at: https://supportforums.cisco.com/discussion/12505826/mysterious-4500-x-6500-fiber-connection-issue#sthash.UbDWJvxL.dpuf

ronbuchalski · ‎05-19-2015

markmalone2008,

Neither one of those links says that you need to use lc-lc. What they say is those particular switches have SFP optics, which by DEFAULT use LC connectors. Therefore, those switches can ONLY have LC connectors for those ports.

Conversely, SC connectors are found on X2 and XENPAK transceivers for 10G optics. So, Catalyst 6500 switch modules such as WS-X6708-10GE (X2) or WS-X6704-10GE (XENPAK) would have SC connectors for 10G. But as I said earlier, we have several connections using SC-LC, generally connecting an X2 module to a Service Provider 10G connection. In our cases, the cables are 15-20 meters long, and we still utilize attenuators to pad down the optical level on the receive side of each connection.

-rb

ttcroziercisco · ‎05-19-2015

We are setting them as trunks, negotiation was not turned off. Although, I did just turn it off as it hasn't been tried yet. We have not attempted to set it us as a Layer 3 port.

The port does down, bounces for a few minutes and typically comes back up after 12-18 minutes. It then stays up for 5 + plus and occurs again. Now, this was fine for 8 months and then it started happening every 5 hours. We even had the 4500-x switches replaced.

Currently, there is nothing on the 4500-x except a trunk link to the 4900 and its still happening.

4500-x trunk config

interface TenGigabitEthernet1/1
switchport trunk native vlan 999
switchport trunk allowed vlan 99
switchport mode trunk
switchport nonegotiate
mtu 9198

4900 trunk config

interface TenGigabitEthernet3/8
switchport trunk native vlan 999
switchport trunk allowed vlan 99
switchport mode trunk
switchport nonegotiate
mtu 9198

ronbuchalski · ‎05-19-2015

ttcroziercisco,

How long are your fiber optic cables? If they are short, what is the requirement to use singlemode rather than multimode?

I highly suggest calculating an optical power budget for your connection, and pad the levels down as necessary. Also, combined with padding down the optical levels, test the connection with a different set of SFPs on both sides. If the problem is related to high optical levels, you may have damaged the SFPs.

-rb