12-06-2013 06:27 AM
Hello everyone
crazy issue here.. everything looks like it should be working but doesnt!
a summary would be our 3750 switches will not trunk over fiber (SM or MM) to our core 6500. They work just fine over copper.
The funny thing is, ONE 3750 works over a SM fiber run.
Our goal is to have redundancy to all switches, 1x copper trunk and 1x fiber trunk. running spanning tree mode rapid-pvst. VTP mode transparent (all VLANs are created manually on all switches --- they exist.)
however, the fiber trunks on 2 switches will only pass layer 2 (CDP neighbor has full detail, mac address-table builds off the 6500) but will not ping the directly connected 6500. we are using cisco brand SFP/GBICs GLC-SX-MM &GLC-LH-SM (we are sure the correct SFP is used with correct fiber type)
debug arp / debug ip packet shows Switch B & C never actually learn the core's mac address and tie it to the IP of 153.29.45.1. all switches have a default gateway of 153.29.45.1.
SWITCH A :
interface GigabitEthernet1/0/48
description CopperTrunk-to-Core
switchport trunk encapsulation dot1q
switchport trunk native vlan 48
switchport trunk allowed vlan 37,48,172
switchport mode trunk
interface GigabitEthernet1/1/1
description FiberTrunk-to-Core
switchport trunk encapsulation dot1q
switchport trunk native vlan 48
switchport trunk allowed vlan 37,48,172
switchport mode dynamic desirable
interface Vlan48
ip address 153.29.45.67 255.255.255.192
no ip redirects
no ip proxy-arp
SWITCH B:
interface GigabitEthernet1/1/1
description FiberTrunk-to-Core
switchport trunk encapsulation dot1q
switchport trunk native vlan 47
switchport trunk allowed vlan 37,47,172
switchport mode trunk
interface GigabitEthernet2/0/48
description CopperTrunk-to-Core
switchport trunk encapsulation dot1q
switchport trunk native vlan 47
switchport trunk allowed vlan 37,47,172
interface Vlan47
ip address 153.29.45.8 255.255.255.224
no ip redirects
no ip proxy-arp
SWITCH C:
interface GigabitEthernet1/0/48
description CopperTrunk-to-Core
switchport trunk encapsulation dot1q
switchport trunk native vlan 48
switchport trunk allowed vlan 37,48,172
switchport mode trunk
interface GigabitEthernet1/1/1
description FiberTrunk-to-Core
switchport trunk encapsulation dot1q
switchport trunk native vlan 48
switchport trunk allowed vlan 37,48,172
switchport mode trunk
interface Vlan48ip address 153.29.45.81 255.255.255.192
no ip redirects
no ip proxy-arp
CORE(6500):
interface GigabitEthernet2/40
description Switch_B_Copper
switchport
switchport trunk encapsulation dot1q
switchport trunk native vlan 47
switchport trunk allowed vlan 37,47,172
switchport mode trunk
interface GigabitEthernet2/43
description Switch_A_Copper
switchport
switchport trunk encapsulation dot1q
switchport trunk native vlan 48
switchport trunk allowed vlan 37,48,172
switchport mode trunk
interface GigabitEthernet2/40
description Switch_C_Copper
switchport
switchport trunk encapsulation dot1q
switchport trunk native vlan 48
switchport trunk allowed vlan 37,48,172
switchport mode trunk
interface GigabitEthernet3/43
description Switch_A_Fiber
switchport
switchport trunk encapsulation dot1q
switchport trunk native vlan 48
switchport trunk allowed vlan 37,48,172
switchport mode trunk
interface GigabitEthernet3/40
description Switch_B_Fiber
switchport
switchport trunk encapsulation dot1q
switchport trunk native vlan 47
switchport trunk allowed vlan 36,37,47,172,500
switchport mode trunk
interface GigabitEthernet3/44
description Switch_C_Fiber
switchport
switchport trunk encapsulation dot1q
switchport trunk native vlan 48
switchport trunk allowed vlan 37,48,172
switchport mode trunk
interface Vlan47
description Internal_Management
ip address 153.29.45.1 255.255.255.224
no ip redirects
no ip proxy-arp
interface Vlan48
description Management
ip address 153.29.45.65 255.255.255.192
Summary: copper links work fine on all 3 switches. switch A & B refuse to pass layer 3 data on Fiber trunks. all were recently updated to version 12.2(53r) SE2 (c3740e-universalk9-mz.122-55.SE8.bin). Core is on 12.2 (33) SXI12 (s72033-adventerprisek9_wan-mz.122-33.SXI12.bin).
any suggestions are appreciated.
12-06-2013 07:07 AM
While I don't normally set it, I notice that your switch A working fiber trunk has:
switchport mode dynamic desirable
The non-working B and C fiber uplinks do not.
Have you tried using that command on them?
12-06-2013 07:20 AM
i have. we have been troubleshooting this 24/7 for a couple days now, so that means things change constantly.
that being said, we traveled to the site containing switch A and plugged Switch C into it's fiber link. Switch C came up on layer 3. that means there is:
1. a problem with the SFPs (unlikely)
2. damaged fiber runs
we shall get our test equipment out tomorrow and being testing strands.
will update this thread with results, but thank you for the response Marvin!
have a great day.
12-06-2013 07:22 AM
You're welcome.
You might troubleshoot the spans remotely using "show interface transceiver details". Also, turn on UDLD.
12-08-2013 10:12 PM
so, we have localized the issue to the 6500.
Switch A is currently functioning on fiber port g3/43.
I created a test switch, and programmed the fiber port. i tried several ports on mod 3 (3/27, 3/42, 3/44, etc) as well as mod 4 (4/11). i even tried 6/1, the supervisor module's SFP port.
i then plugged the test switch into g3/43, and it worked. (good layer 3 ping)
we recently upgraded the 6500 from SXI6 to SXI12. i am looking over the release notes but not seeing any bugs like this.
i doubt a "hw-module module X reset" would do anything, since i tried 3 different modules.
thoughts??
12-08-2013 10:15 PM
also the several ports i tried were all programmed exactly like 3/43, so it is not the config.
12-08-2013 11:15 PM
When you have fibre SFP in, post the output to the command "sh interface BLAH".
12-08-2013 11:39 PM
Leo: here ya go
GigabitEthernet3/43 is up, line protocol is up (connected)
Hardware is C6k 1000Mb 802.3, address is 2894.0f57.437a (bia 2894.0f57.437a)
Description: FiberTrunk-to-SwitchA
MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, media type is LH
input flow-control is off, output flow-control is off
Clock mode is auto
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:08, output 00:00:24, output hang never
Last clearing of "show interface" counters never
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 191000 bits/sec, 67 packets/sec
5 minute output rate 752000 bits/sec, 112 packets/sec
92584292 packets input, 25411211351 bytes, 0 no buffer
Received 315527 broadcasts (83158 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 0 multicast, 0 pause input
0 input packets with dribble condition detected
148923032 packets output, 114663742745 bytes, 0 underruns
0 output errors, 0 collisions, 6 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 PAUSE output
0 output buffer failures, 0 output buffers swapped out
GigabitEthernet6/1 is down, line protocol is down (notconnect)
Hardware is C6k 1000Mb 802.3, address is 001a.a22d.6984 (bia 001a.a22d.6984)
Description: FiberTestPort
MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
reliability 255/255, txload 0/255, rxload 0/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, media type is LH
input flow-control is off, output flow-control is off
Clock mode is auto
ARP type: ARPA, ARP Timeout 04:00:00
Last input 01:52:33, output 01:51:56, output hang never
Last clearing of "show interface" counters never
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
218 packets input, 26637 bytes, 0 no buffer
Received 182 broadcasts (63 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 0 multicast, 0 pause input
0 input packets with dribble condition detected
9476 packets output, 1084062 bytes, 0 underruns
0 output errors, 0 collisions, 3 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 PAUSE output
0 output buffer failures, 0 output buffers swapped out
let me know if you'd like to see anything else.
12-08-2013 11:49 PM
GigabitEthernet6/1 is down, line protocol is down (notconnect)
Hardware is C6k 1000Mb 802.3, address is 001a.a22d.6984 (bia 001a.a22d.6984)
Description: FiberTestPort
MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
reliability 255/255, txload 0/255, rxload 0/255
Your SFP could be faulty. I have NEVER seen a value like that before.
Can you try putting this SFP in a different port and different slot?
12-08-2013 11:50 PM
Wait .. there's another thing. Post a picture of the SFP. I want to see the front of the SFP where it will have the model number and the serial number.
12-08-2013 11:46 PM
there are several fiber ports in use on mod 3 and mod 4 working fine. rack switches, remote switches, etc.
any new trunks we try to add only get L2.
Mod Ports Card Type Model Serial No.
--- ----- -------------------------------------- ------------------ -----------
1 48 CEF720 48 port 10/100/1000mb Ethernet WS-X6748-GE-TX SAD082702MJ
2 48 CEF720 48 port 10/100/1000mb Ethernet WS-X6748-GE-TX SAL16042APK
3 48 CEF720 48 port 1000mb SFP WS-X6748-SFP SAL16063LT2
4 48 CEF720 48 port 1000mb SFP WS-X6748-SFP SAL16063LPT
5 2 Supervisor Engine 720 (Hot) WS-SUP720-3B SAL1127T8BG
6 2 Supervisor Engine 720 (Active) WS-SUP720-3B SAL1253F312
12-09-2013 01:33 AM
Leo: you are seeing 0 tx/rx load because it was unplugged. here is the result when its plugged in:
GigabitEthernet6/1 is up, line protocol is up (connected)
Hardware is C6k 1000Mb 802.3, address is 001a.a22d.6984 (bia 001a.a22d.6984)
Description: FiberTestPort
MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, media type is LH
input flow-control is off, output flow-control is off
Clock mode is auto
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:12, output 00:00:39, output hang never
Last clearing of "show interface" counters never
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 7000 bits/sec, 11 packets/sec
439 packets input, 54271 bytes, 0 no buffer
Received 390 broadcasts (131 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 0 multicast, 0 pause input
0 input packets with dribble condition detected
20397 packets output, 2134848 bytes, 0 underruns
0 output errors, 0 collisions, 3 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 PAUSE output
0 output buffer failures, 0 output buffers swapped out
also here are the pictures of the SFPs from both 4/43 and 6/1.
12-09-2013 01:45 AM
Hmmmmm ... Thanks for the pix. Not what I was hoping.
Can you do the following:
1. Plug one of the SFP into 5/1 and plug the other into 6/1. Test and see if it works.
2. Plug one SFP into another port other than 5 or 6. Plug another into 5/1. Test to see if this works.
3. Plug both SFP into 3750 and see if this works.
Note: Do NOT enable the SFP to go active for very long or you will permanently burn your SFP.
12-09-2013 04:09 AM
Leo, the issue is new 3750s will not pass layer 3 across any fiber trunk to the 6500. i have a test switch physically sitting in front of the 6500, with an LC-LC single-mode patch cable. it will not ping thru 5/1, 6/1, or any other port except 3/43 (existing port before the SXI12 upgrade) - all ports have exact same config as 3/43.
01-13-2014 01:45 AM
so, after troubleshooting for 7 hours with a Cisco switching expert, we finally determined the cause. Turns out, 1 of 2 supervisor modules (WS-SUP720-3B) had fabric errors, resulting in fiber ports unable to build trunks/access.
replaced the supervisor module with a new one (RMA) and everything is back to normal.
posting this fix action in case others have the same issue.
- Justin
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide