09-09-2013 02:02 PM - edited 03-07-2019 03:22 PM
We have some Cisco 2960 switches running c2960s-universalk9-mz.122-55.SE7.bin. We downgraded to this from c2960s-universalk9-mz.122-58.SE2.bin at the advice of our support provider when this issue first started. Sometimes, a couple of VLANs on a switch will lost all connectivity to the rest of the network. There are no errors in the log, spanning tree looks completely normal, etc. The only thing that brings connectivity back is either rebooting the switch or deleting the affected VLANs from the switch and adding them back in. Once either of those happens it is like everything is back to normal. This has happened on both stacked and stand-alone 2960 switches. Has anyone ever heard of this? I'd like to hear any suggestions as to where to go next.
09-10-2013 05:54 AM
Can you provide a topology diagram and perhaps the config from whatever device is doing your intervlan routing?
09-10-2013 06:10 AM
Hello jcrussell,
I would suggest you to try the following approach. Identifie a particular "broken" VLAN ID, source and destination MAC addresses pair and investigate starting from there. You should be aware what are incoming and outgoing interfaces for this flow supposed to be.
! Check interfaces statistics
show interface
show interfaces
show interfaces
! Verify STP
show spanning-tree vlan
show spanning-tree interface
! Verify CAM table population
show mac address-table interface
show mac address-table dynamic address
! Verify if the outgoing interface are programmed correctly
show platform forward
If still you won't be able to find out the reason for the issue, please, open a TAC service request.
Thank you
--
Best regards,
Dmitry Skotnikov
09-10-2013 06:26 AM
Thank you both very much. A TAC case is already open and is making zero progress. With that said, I am working on sanitizing the config from our core. I am attaching a cheesy diagram of the setup, which is fairly simple. The VLANs in question only exist on the switches depicted in the diagram, not on any others.
As far as the switches go, there are basically zero errors on any interfaces in the affected vlans:
GigabitEthernet1/0/1 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is c40a.cbXX.XXXX (bia c40a.cbXX.XXXX)
Description: **REDACTED**
MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 100Mb/s, media type is 10/100/1000BaseTX
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:19, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 283
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 35000 bits/sec, 21 packets/sec
5 minute output rate 47000 bits/sec, 23 packets/sec
110821 packets input, 18547529 bytes, 0 no buffer
Received 1077 broadcasts (855 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 855 multicast, 0 pause input
0 input packets with dribble condition detected
212861 packets output, 73744196 bytes, 0 underruns
0 output errors, 0 collisions, 1 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 PAUSE output
0 output buffer failures, 0 output buffers swapped out
Syslogs show no spanning tree topology changes either. That was my first thought when this happened. Topology change times line up with the switch reboot times. From the core:
VLAN0148 is executing the rstp compatible Spanning Tree protocol
Bridge Identifier has priority 24576, sysid 148, address 588d.0941.ef00
Configured hello time 2, max age 20, forward delay 15, transmit hold-count 6
We are the root of the spanning tree
Topology change flag not set, detected flag not set
Number of topology changes 2 last change occurred 14:07:47 ago
from Port-channel10
Times: hold 1, topology change 35, notification 2
hello 2, max age 20, forward delay 15
Timers: hello 0, topology change 0, notification 0, aging 300
Port 560 (Port-channel10) of VLAN0148 is designated forwarding
Port path cost 3, Port priority 128, Port Identifier 128.560.
Designated root has priority 24724, address 588d.0941.ef00
Designated bridge has priority 24724, address 588d.0941.ef00
Designated port id is 128.560, designated path cost 0
Timers: message age 0, forward delay 0, hold 0
Number of transitions to forwarding state: 1
Link type is point-to-point by default
BPDU: sent 25246, received 6
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide