Trunk between two switches randomly disconnects

[nkd2400]

Hello,

I have taken over a pre-existing network and I have two switches (Cisco Catalyst 3750G-12S) which are connected to each other using SFP's with LC duplex fiber. The two switch's ports are configured as trunks with the following config:

interface GigabitEthernet1/0/3

 switchport trunk encapsulation dot1q

 switchport mode trunk

 load-interval 30

end

I am seeing random 3 second long disconnections between the two switches. The weird thing is, even though both switches are logging the same (debugging) I only see the LINEPROTO-5-UPDOWN messages on the first switch when the disconnection happens. Each time it last roughly 3 seconds, and then it's good again. This is happening 5 to 10 times per day and is causing issues within our network for network latency sensitive applications.

Before I start replacing cabling, SFP modules, and/or switches does anyone have any tips or suggestions on what could be the issue? I'm specifically curious as to why only one switch is giving me the Interface changed state messages in the logs and the other doesn't show anything?

Thanks!

[Leo Laohoo]

Post the complete output to the command "sh interface Gi1/0/3".

[nkd2400]

Thanks for the response:

Switch 01

01#sh interfaces Gi1/0/3
GigabitEthernet1/0/3 is up, line protocol is up (connected)
  Hardware is Gigabit Ethernet, address is 
  Description: 
  MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive not set
  Full-duplex, 1000Mb/s, link type is auto, media type is 1000BaseSX SFP
  input flow-control is off, output flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:18, output 00:00:01, output hang never
  Last clearing of "show interface" counters 5y4w
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 121972
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  30 second input rate 110000 bits/sec, 32 packets/sec
  30 second output rate 211000 bits/sec, 55 packets/sec
     432507682 packets input, 155407098901 bytes, 0 no buffer
     Received 22759026 broadcasts (16099943 multicasts)
     0 runts, 0 giants, 0 throttles
     7808 input errors, 340 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 16099943 multicast, 0 pause input
     0 input packets with dribble condition detected
     5418179762 packets output, 1136546098097 bytes, 0 underruns
     0 output errors, 0 collisions, 4 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out

Switch 11

11#sh interfaces Gi1/0/3

GigabitEthernet1/0/3 is up, line protocol is up (connected)
  Hardware is Gigabit Ethernet, address is 
  Description: 
  MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive not set
  Full-duplex, 1000Mb/s, link type is auto, media type is 1000BaseSX SFP
  input flow-control is off, output flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:01, output 00:00:04, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 725
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 2905000 bits/sec, 278 packets/sec
  5 minute output rate 83000 bits/sec, 41 packets/sec
     3528531968 packets input, 919016004723 bytes, 0 no buffer
     Received 2880698341 broadcasts (1512223191 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 1512223191 multicast, 0 pause input
     0 input packets with dribble condition detected
     430408680 packets output, 155085979828 bytes, 0 underruns
     0 output errors, 0 collisions, 1 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out

I'm going to look into what the interface resets represents.

[mohammed01701]

Hi!

You do have CRC errors on switch 01, what I can see there is "7808 input errors, 340 CRC" and that is telling you, at you have some kind of media issue some Deepak kumar told you before:

Are the both switches on the same building or are you using the building fiber infrastructure, if is the second you have to measure the fiber on building? 

Have you tried to use another interface on the switches and see if that helps?

Why have you allowed all vlans on that interface?

Are you using some kind off mls rate-limit on the switches?

/Mohammed 

[nkd2400]

Hi there,

The input errors, would indicate that it is packets received on that interface of Switch 01, right? Just trying to understand that.

Both switches are in different buildings about 100 meters apart. The inter-building connections are not handled by my section, so I would have to get back to you on any measurements.

Unfortunately, all the interfaces are in use, but this weekend I will come in and swap two interfaces to see what happens. I'm trying to minimize any outages. Also, this is a very old switch, and we don't have any replacement on hand (more have been ordered) so I'm hesitant to mess around with it too much yet.

I inherited this configuration, but will look into only allowing the specified VLANs that are needed on that trunk. Is there a benefit to doing it like this, other than saving bandwith? Security considerations?

I'm not aware of any rate-limits on the switches. I looked through the config, and it's a pretty basic setup.

[Leo Laohoo]

I don't see any issues with the output. The line errors are very tiny.
I would recommending replacing the modules. If this was a cable issue, then the line errors would be large.

[nkd2400]

Thanks for the feedback. Since the ports are all full, I will be swapping a known good connection (Gi1/0/7 on S01) with the suspect connection (Gi1/0/3 on S01) and see which port fails. I do have extra SFP's so I can also swap that in place afterwards on whichever port is failing (on S11 Gi1/0/3 as well).

I will update here whatever I find out.

[nkd2400 · Accepted solution]

So, just to update anyone who comes upon this, here is what was done and now the issue is resolved.

I swapped out SFP and port on Switch 01, but that didn't make a difference. 

So I swapped out the SFP on Switch 11 and no difference. When swapping ports (because I didn't have any available, I had to swap two LC cables) it seems to work fine now.

Normally, this would indicate that the issue was the port. However, the old port (Gi1/0/3) is now working properly connected to the other switch (Switch 52) as a trunk. In the end, I think the LC cable may have just needed to be re-seated or something like that? At this point, I don't want to mess with anything, especially since I don't have a cable tester nor cleaner available right now. Wish I had a cable scope, that could have helped I'm sure as well.

I feel a bit silly marking this as the solution, but I do want to show that this was completed. Thanks for all the responses and help with this. It's still strange that only one switch (Switch 1) would show the interface going down, and not both. That, I don't have any answers for.

[Deepak Kumar]

Hi,

It seems like a Media issue but shares an output from the command "Show interface <Trunk Port Number>" and Both side Port configuration also.

Regards,

Deepak Kumar

[nkd2400]

Thanks for the response. I posted the results of the show interface gi1/0/3 above.

The configuration of the ports is as below:

Switch 01

interface GigabitEthernet1/0/3

 switchport trunk encapsulation dot1q

 switchport mode trunk

 load-interval 30

end

Switch 11

interface GigabitEthernet1/0/3
 description Link to 
 switchport trunk encapsulation dot1q
 switchport mode trunk
end

[Leo Laohoo]

Post the complete output to the command "sh controll e Gi1/0/3" from both switches.

[nkd2400]

Thanks for the response, here is the output of those commands from the two switches:

01#sh controllers e Gi1/0/3

     Transmit GigabitEthernet1/0/3            Receive
    868580888 Bytes                       1693528593 Bytes
    705657646 Unicast frames               414499879 Unicast frames
   2119112122 Multicast frames              16182961 Multicast frames
   2608101073 Broadcast frames               6676658 Broadcast frames
            0 Too old frames               995232170 Unicast bytes
            0 Deferred frames             2546633805 Multicast bytes
            0 MTU exceeded frames          720317149 Broadcast bytes
            0 1 collision frames                   0 Alignment errors
            0 2 collision frames                 343 FCS errors
            0 3 collision frames                   0 Oversize frames
            0 4 collision frames                   0 Undersize frames
            0 5 collision frames                  10 Collision fragments
            0 6 collision frames
            0 7 collision frames             5191933 Minimum size frames
            0 8 collision frames           239672186 65 to 127 byte frames
            0 9 collision frames            73752675 128 to 255 byte frames
            0 10 collision frames           38865327 256 to 511 byte frames
            0 11 collision frames            5403377 512 to 1023 byte frames
            0 12 collision frames           72210597 1024 to 1518 byte frames
            0 13 collision frames                  0 Overrun frames
            0 14 collision frames                  0 Pause frames
            0 15 collision frames
            0 Excessive collisions              7520 Symbol error frames
            0 Late collisions                     94 Invalid frames, too large
            0 VLAN discard frames            2270639 Valid frames, too large
            0 Excess defer frames                543 Invalid frames, too small
    365437144 64 byte frames                       0 Valid frames, too small
   3580653998 127 byte frames
    838371085 255 byte frames                      0 Too old frames
    206031433 511 byte frames                      0 Valid oversize frames
     48002944 1023 byte frames                     0 System FCS error frames
    390531452 1518 byte frames                     0 RxPortFifoFull drop frame
      3842785 Too large frames
            0 Good (1 coll) frames
            0 Good (>1 coll) frames

11#show contr e Gi1/0/3

     Transmit GigabitEthernet1/0/3            Receive
   1371231074 Bytes                       2377198174 Bytes
    412495154 Unicast frames               660580172 Unicast frames
     16182976 Multicast frames            1513612232 Multicast frames
      6581376 Broadcast frames            1369027039 Broadcast frames
            0 Too old frames               266252268 Unicast bytes
            0 Deferred frames               58514723 Multicast bytes
            0 MTU exceeded frames         1079452371 Broadcast bytes
            0 1 collision frames                   0 Alignment errors
            0 2 collision frames                   0 FCS errors
            0 3 collision frames                   0 Oversize frames
            0 4 collision frames                   0 Undersize frames
            0 5 collision frames                   0 Collision fragments
            0 6 collision frames
            0 7 collision frames            39970275 Minimum size frames
            0 8 collision frames          2435284812 65 to 127 byte frames
            0 9 collision frames           488503169 128 to 255 byte frames
            0 10 collision frames          148110400 256 to 511 byte frames
            0 11 collision frames           39217235 512 to 1023 byte frames
            0 12 collision frames          388290764 1024 to 1518 byte frames
            0 13 collision frames                  0 Overrun frames
            0 14 collision frames                  0 Pause frames
            0 15 collision frames
            0 Excessive collisions                 0 Symbol error frames
            0 Late collisions                      0 Invalid frames, too large
            0 VLAN discard frames            3842788 Valid frames, too large
            0 Excess defer frames                  0 Invalid frames, too small
      3850070 64 byte frames                       0 Valid frames, too small
    239372648 127 byte frames
     73511627 255 byte frames                      0 Too old frames
     38741677 511 byte frames                      0 Valid oversize frames
      5389644 1023 byte frames                   280 System FCS error frames
     72123311 1518 byte frames                     0 RxPortFifoFull drop frame
      2270809 Too large frames
            0 Good (1 coll) frames
            0 Good (>1 coll) frames

[nkd2400]

Sorry, duplicate post.