Ever heard of a VLAN crashing?

[robert.bahr]

Today I was totally flustered and confused when I saw the state of my switched network. I'm administering a military network with approx 50 users; we run W2K3 server with DNS, AD, and Exchange local to our subnet. We draw services from a fiber pedestal here.

Today our distant end went down (they wouldn't tell me how or why) and our network went down subsequently. We had no local L3 connectivity from any node to any node on our network however, we soon figured out that no computer, server, router or switch could ping any other node. All of our link lights on our switches went amber, and as far as I know we did nothing to provoke this. Our users are on vlan 3, and our uplink was on vlan 5, see diagram.

I consoled into our switch (and later our other switches) and find that Vlans 3 and 5 are down; not administratively down, but down. Keep in mind our network configuration has worked for some time. After an hour of needless tinkering (sorry, troubleshooting), I entered vlan database and forced the vlans to active, which I had to do for every switch. Approximately an hour later, distant end came up and we were golden.

Finally, my question. What could have caused this? Note that remote administration is not an option on this network, and it's highly unlikely someone consoled into our switches. Network diagram attached.

[Edison Ortiz]

What VTP mode those switches are set to?

VTP Client, Server or Transparent?

On the first 2, a corruption on the vlan.dat file may be the cause. You mentioned you had to create the vlans one again, were they missing when you issue show vlan?

What version of code on these switches?

__

Edison.

[akbindal]

Hi Robert..

You told that the lights on the switches went Amber..was this seen on the 2 2950's or the distant 3750 as well..

Were there any spanning tree changes in the network..

Also, you told that VLANs were showing down..do you mean they were showing inactive in show vlan output(if yes this could be a VTP problem as well or when no siwtchports have been assigned on those VLAN..have we statically assigned the ports or use VMPS server for port assignment in these VLANs)..

What are the VTP modes in which 2 2950's and 3750 run and how is 2950 connected to 3750 switch here..

Any logs from these switches would be helpful as well for isolating the problem..

Akhil

[robert.bahr]

The problem has been worked around now, we ran a link from the fiber pedestal directly to our router, and it works fine. I'm still curious about this problem however.

The lights on the (2) 2950's did go amber on all ports plugged in. The distant 3750 I didn't see, I only know it's a 3750 from a show cdp neighbors.

Don't know if there were any spanning tree changes, there weren't on my end.

I did a show ip interface brief, and under vlan's 3 and 5 i had a down down status. I've seen them admin down before, but not down. I didn't do a show vlan output, I've never used that command before. When we were troubleshooting however, we would have distant end no shut our port and when it connected, we would get a VLAN mismatch error everytime. We found out later that they were on VLAN 1, it stopped the error, but would shut off the ports like before.

The VTP modes as far as I know are all default, we usually don't use VTP and manually configure.

[Edison Ortiz · Accepted solution]

The layer3 Vlan interfaces depend on the layer2 Vlan to be on the vlan database. If that information is missing, the layer3 Vlan will remain down.

If it occurs again, I recommend checking your vlan information with the show vlan command.

If you are running vtp modes in default mode, all switches are running vtp server. If you haven't set a vtp domain name, the name is null and the switch with the highest revision number will rule what vlan other switches must have (if they are inter-switch links).

I highly suggest you implement some kind of vtp configuration to prevent this in the future. Adding a switch with a higher revision can easily wipe out the vlan configuration on the remaining switches and perhaps that's exactly what happened.

Please take a moment and read this link:

http://www.cisco.com/en/US/docs/switches/lan/catalyst3750/software/release/12.2_40_se/configuration/guide/swvtp.html

HTH,

__

Edison.

[robert.bahr]

Edison, thanks for your advice and knowledge. We'll implement this into our scheme.

[akbindal]

Hi Robert,

Thanks for the updates..

I agree that since we do not have any VTP config. what has happened here is that every switch is working in server mode and when the revision number is changed on any switch it will replicate its VLAN database on to the other switch..so even if you install a new switch with a higher revision number it will wipe out all VLANs from the database of other switch if it doesn't have any and since, the L3 VLAN interfaces depend upon local L2 VLAN database and interfaces in that VLAN..it would have gone down as there might be no VLAN info on that switch for those VLANs..

As recommended please implement a VTP domain in your network to put the issue out of the network..

Thanks

Akhil