Re: Ten Gig Trunk Ports that stop passing traffic - Page 2

darinheilman · ‎09-25-2018

I am banging my head against an issue that has been ongoing on my school network. Every day (or if i'm lucky...every other day) i am seeing a switch closet go down. Its not the same one each time, and its at random times. All the closets are in a hub topography connected back to a core switch with 10 gig multimode fiber trunk links. At first i was focusing on equipment maybe going bad.....but i've swapped out tranceivers, patch cables and entire switches and the issue persists.

Some times when looking at the switch logs, i see the Ten gig port actually go DOWN.....other times i see nothing...but no traffic is going over the port. Im not seeing any spanning tree or BPDU guard errors either. Sometimes i can move the transceiver to another slot and traffic will flow again.....sometimes i need to reboot the switch (a reboot always gets the traffic flowing again, btw)

I could give specifics of equipment and configurations.....but i thought i'd reach out and ask if anyone has any ideas as to the kind of things that can cause a trunk interface just stop passing traffic....and have a reboot fix it.

Thanks in advance.

darinheilman · ‎10-10-2018

the weird thing is that the link doesn't really DROP.....it just stops passing traffic. Link light is on, sh interface (as you will see below) shows the link up. Doing a "shut"/"no shut" doesn't fix it from either end.....the only thing that brings it back is a reboot.

What happens in a reboot that would cause the link to be happy again....whereas a shut/no shut doesn't do it

sh int t2/1/2
TenGigabitEthernet2/1/2 is up, line protocol is up (connected)
Hardware is Ten Gigabit Ethernet, address is bcc4.9339.fdb6 (bia bcc4.9339.fdb                                                                             6)
Description:
MTU 1500 bytes, BW 10000000 Kbit/sec, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive not set
Full-duplex, 10Gb/s, link type is auto, media type is SFP-10GBase-LRM
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:02, output never, output hang never
Last clearing of "show interface" counters never
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 4425000 bits/sec, 587 packets/sec
5 minute output rate 5488000 bits/sec, 1054 packets/sec
     679360941 packets input, 876419381953 bytes, 0 no buffer
     Received 327682 broadcasts (95146 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 95146 multicast, 0 pause input
     0 input packets with dribble condition detected
     656196403 packets output, 208423682680 bytes, 0 underruns
     0 output errors, 0 collisions, 2 interface resets
     0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out

sh int t1/0/1
TenGigabitEthernet1/0/1 is up, line protocol is up (connected)
Hardware is Ten Gigabit Ethernet, address is f078.16be.2b33 (bia f078.16be.2b3                                                                             3)
Description:
MTU 1500 bytes, BW 10000000 Kbit/sec, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive not set
Full-duplex, 10Gb/s, link type is auto, media type is SFP-10GBase-LRM
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:18:55, output 00:00:45, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 1138000 bits/sec, 198 packets/sec
5 minute output rate 928000 bits/sec, 111 packets/sec
     3652926151 packets input, 1158150170330 bytes, 0 no buffer
     Received 130546962 broadcasts (89038802 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 89038802 multicast, 0 pause input
     0 input packets with dribble condition detected
     3985857048 packets output, 5197214445380 bytes, 0 underruns
     0 output errors, 0 collisions, 3 interface resets
     0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out

Like I said before.....the issue moves around also.....there is never a consistent area that we can pin it down to.

bporcelli76 · ‎08-31-2019

Hello, I was reading through this post, and noticed that a customer of mine is experiencing the same problem. Or at least very similiar symptoms. I was wondering if you ever found out what the root cause was, or how you solved the issue.

Looking forward to your response.

Thank you.

Georg Pauwen · ‎08-31-2019

Hello,

which platforms and IOS versions are involved ?

bporcelli76 · ‎09-11-2019

I was wondering in your situation did you ever find the solution and if so, what was it.

Athena Northway · ‎09-01-2020

I am having the same issue on the 2960x 10 GB ports. They show activity, but no traffic is passing. There is nothing in the logs, no CRC errors, no loops, spanning tree issues. It just stops passing traffic. I can do the shut/no shut with no results. I have to reboot the stack for it to communicate again. I upgraded the 2960X stack to 15.2(7)E2 and it is still having the same issue. The other end is a 4500 running 03.07.00.E.152-3.E.

Were you able to solve the problem?

Leo Laohoo · ‎09-01-2020

Update the firmware of the 4500.

IOS-XE version 3.7.0 is extremely buggy.

darinheilman · ‎09-02-2020

The problem is still happening and i always need to reboot. It's a supreme pain in the ass.....but no one has been able to help us....so we just reboot.

I have tried different combinations of IOS images for both switches. The 2960X's are now on 15.2(4)E8 and the 4510 is on 3.11.02E

No change. Its the most frustrating network thing I have ever seen.

Leo Laohoo · ‎09-02-2020

TenGigabitEthernet2/1/2 is up, line protocol is up (connected)
Last input 00:00:02, output never, output hang never
     679360941 packets input, 876419381953 bytes, 0 no buffer
     656196403 packets output, 208423682680 bytes, 0 underruns

TenGigabitEthernet1/0/1 is up, line protocol is up (connected)
Last input 00:18:55, output 00:00:45, output hang never
     3652926151 packets input, 1158150170330 bytes, 0 no buffer
     3985857048 packets output, 5197214445380 bytes, 0 underruns

I am reviewing this thread and I've noticed something.

1. Look at Ten 1/0/1. The port stopped receiving traffic for almost 19 minutes.

2. Look at the packet input/output of Ten 1/0/1 and compare the numbers with Ten 2/1/2. They don't match.

Can you describe the network between the two switches? Are they point-to-point? Are there any cross-connect or media converter between the two?
Next, what is the distance between the two? I am noticing both are using LRM optics.

darinheilman · ‎09-03-2020

its not a point to point from our core 4510 to this 2960X.

I will get a network diagram worked up for the connection. However, we have had this SAME issue for a different 2960X that IS point to point directly connected to the 4510 with a relatively short MM fiber run (200 ft approx.).....

Stay tuned.