02-18-2015 05:12 PM - edited 03-05-2019 12:50 AM
I changed a switch's VTP mode from Server to Transparent today and it caused an outage. I'm a little baffled as to why this occurred and was hoping someone could offer me insight on what might have happened.
Here is the topology.
One switch operating as the VTP server. Model: WS-C4507R running Version 12.2(31)SG2
sw01#show vtp status
VTP Version : 2
Configuration Revision : 8
Maximum VLANs supported locally : 1005
Number of existing VLANs : 23
VTP Operating Mode : Server
VTP Domain Name : messaging
VTP Pruning Mode : Enabled
VTP V2 Mode : Disabled
VTP Traps Generation : Disabled
MD5 digest : 0xF9 0x54 0x8A 0xDD 0x80 0x6F 0x75 0xD5
Configuration last modified by x.x.x.x at 7-16-13 17:38:52
Local updater ID is x.x.x.x on interface Vl60 (preferred interface)
Preferred interface name is vlan60
Directly subtending the switch listed above was a OS-CIGESM-18TT-EBU running Version 12.1(22)EA11
bc01#sh vtp status
VTP Version : 2
Configuration Revision : 8
Maximum VLANs supported locally : 250
Number of existing VLANs : 23
VTP Operating Mode : Client
VTP Domain Name : messaging
VTP Pruning Mode : Enabled
VTP V2 Mode : Disabled
VTP Traps Generation : Disabled
MD5 digest : 0xF9 0x54 0x8A 0xDD 0x80 0x6F 0x75 0xD5
Configuration last modified by x.x.x.x at 7-16-13 17:38:52
My understanding is that in VTP server mode will send out advertisements to clients. When a Vlan is added to the server the config revision number is incremented by 1 and vtp advertisements is sent out to compare the revision number. If the revision number is higher then subset advertisements are sent out containing all the vlan information. Clients will recieve this and update their vlan database. In addition a VTP server will store this vlan database in vlan.dat
A VTP Client does not store the vlan database, It stores this information in memory.
This is what i think may have happened.
When I switched from VTP server to VTP trasparent, The VTP client lost all its vlan information and this started the outage.
The outage was resolved when i changed the switch back to vtp server.
Shouldn't the client fall back to its local vlan.dat file in the event it doesn't see the VTP server?
Also the output from above was from after the outage. If the client truly lost all vlan information shouldn't the configuration modification date be the current date?
bc01#dir flash:/vlan.dat
Directory of flash:/vlan.dat
5 -rwx 1636 Jul 16 2013 17:38:52 +00:00 vlan.dat
7612416 bytes total (2682368 bytes free)
bc01#
When i decoded the vlan.dat it has all the necessary vlans needed to sustain layer 2 traffic.
VLAN NAME
---- -------------------
1 default
60 mgmt
61 servers
63 test
90 backend
92 vip
99 MGMT
120 misc
121 media
122 notifications
123 mail
124 application
125 autoattendant
126 web
127 directory
128 reminders
129 resolver
130 faxgw
260 provisioning_server
1002 fddi-default
1003 token-ring-default
1004 fddinet-default
1005 trnet-default
When i perform the same test in our lab environment, layer 2 connectivity is never compromised. The client switch never loses its vlan configuration. I'm really at a loss as why this happened.
Solved! Go to Solution.
02-20-2015 03:35 AM
Friends,
The symptoms described here are consistent with the quirks of VTP Pruning.
With VTP Pruning active, switches send VTP Join messages every 6 seconds to each other, telling the other switch what VLANs are active on the sender. By receiving a VTP Join message from a neighboring switch, the receiving switch knows if any VLANs on the neighbors are active and worth of forwarding traffic into.
The continuous signalling using VTP Joins is crucial, and in fact, it appears to be VTP's weak spot. If the VTP Joins stop being received for whatever reason, the receiving switch will start assuming that the neighboring switch considers all VLANs being inactive, and prunes all VLANs from the trunk. This is what most probably happened in this incident: VTP Pruning was on, and later, one of two interconnected switches was reconfigured from a VTP Server/Client mode to VTP Transparent. As a result, the Transparent switch stopped sending VTP Joins and its neighboring switch pruned off all VLANs from the trunk, thinking that they are all unused.
In essence, this VTP Join signalling uses a fail-closed approach: if no VTP Joins are received, all VLANs are considered as being unused at the neighbor, resulting in their pruning off the trunk where the Joins have stopped being received. Quite an unfortunate choice of implementation in my opinion: with VTP Pruning, there is no risk in fail-open approach: you just get the normal switching behavior as if VTP Pruning was not in use at all.
The idea about a VTP Client switch losing its config has its merit but is not true with today's switches. Very old switches running CatOS appear to have stored the VLAN database in RAM only, so if they got restarted, they initially lost the entire VLAN configuration. However, all IOS-based switches to my best knowledge store the VLAN database in their FLASH, regardless of the mode (Server, Client, Transparent, or Off).
Perhaps this help.
Best regards,
Peter
02-19-2015 02:42 PM
I don't think the client switch lost its configuration.
I did a bit of digging around and a few people reported similar issues when switching to transparent mode and they were suggesting it is because you have pruning enabled.
When you changed the 4500 to transparent it no longer participates in VTP pruning so your client switch believes all those vlans can be pruned off the trunk link.
Which is exactly what it does.
I tested this in a lab with two switches connected via a trunk link and a host connected to each switch. The lab only emulates L3 switches.
The hosts were both in vlan 10 and that vlan was allowed on the trunk link.
I enabled VTP pruning when there was a server and client and then changed the server to transparent and the client switch did indeed prune the vlan ie. a "sh int trunk" showed vlan 10 had been pruned on the trunk link as far as the client was concerned.
The transparent switch was still allowing both vlans on the trunk.
However I could still initiate a ping from the host connected to the client switch even though the client switch was saying it has pruned the vlan off the trunk link.
So I can't say for sure this was the problem you had.
My understanding of pruning was that if a vlan was pruned off the trunk traffic would not pass but that clearly wasn't happening in the lab I setup.
So either the lab switches were behaving differently than real switches or I just don't understand pruning as well as I thought I did (which is quite likely :-))
I don't have any real switches to test with although perhaps you do ?
There is someone a lot more knowledgeable than me in these forums called Peter Paluch and i'll see if I can get him to contribute because I know he is very familiar with VTP.
Jon
02-20-2015 03:35 AM
Friends,
The symptoms described here are consistent with the quirks of VTP Pruning.
With VTP Pruning active, switches send VTP Join messages every 6 seconds to each other, telling the other switch what VLANs are active on the sender. By receiving a VTP Join message from a neighboring switch, the receiving switch knows if any VLANs on the neighbors are active and worth of forwarding traffic into.
The continuous signalling using VTP Joins is crucial, and in fact, it appears to be VTP's weak spot. If the VTP Joins stop being received for whatever reason, the receiving switch will start assuming that the neighboring switch considers all VLANs being inactive, and prunes all VLANs from the trunk. This is what most probably happened in this incident: VTP Pruning was on, and later, one of two interconnected switches was reconfigured from a VTP Server/Client mode to VTP Transparent. As a result, the Transparent switch stopped sending VTP Joins and its neighboring switch pruned off all VLANs from the trunk, thinking that they are all unused.
In essence, this VTP Join signalling uses a fail-closed approach: if no VTP Joins are received, all VLANs are considered as being unused at the neighbor, resulting in their pruning off the trunk where the Joins have stopped being received. Quite an unfortunate choice of implementation in my opinion: with VTP Pruning, there is no risk in fail-open approach: you just get the normal switching behavior as if VTP Pruning was not in use at all.
The idea about a VTP Client switch losing its config has its merit but is not true with today's switches. Very old switches running CatOS appear to have stored the VLAN database in RAM only, so if they got restarted, they initially lost the entire VLAN configuration. However, all IOS-based switches to my best knowledge store the VLAN database in their FLASH, regardless of the mode (Server, Client, Transparent, or Off).
Perhaps this help.
Best regards,
Peter
02-20-2015 03:48 AM
Hi Peter
Many thanks for joining in.
I agree with all you say and that is what I saw in the lab I ran except even with the vlan pruned on the trunk link the switch still forwarded traffic for that vlan on the trunk link.
I guess I will have to put that down to the switches in the lab I was using.
Jon
02-20-2015 04:37 AM
Hi Jon,
It's a pleasure.
Regarding the observation you have made in the lab: You are saying that the traffic was still forwarded despite VLANs being pruned off the trunk. Hmmm... Were both flow directions, ingress and egress, equally allowed?
Best regards,
Peter
02-20-2015 04:52 AM
Yes they were.
When I changed the server to transparent it was still allowing the vlan on the trunk.
The client switch however had pruned it off.
So I deliberately tried the ping from the host connected to the client (although I did try the other way as well). I also cleared all mac address tables on the switches and the arp entries on the hosts between every test just to be sure although as I understand it that shouldn't make any difference.
The ping always worked which confused me because the client switch believes there are no active clients in that vlan reachable via the trunk link.
I tested it a number of times to make sure and the results were always the same.
Jon
02-20-2015 07:30 AM
Hi Jon,
Hmm, that's strange. Do you believe you have time to reproduce your lab experiment and post the show commands that prove that the VLANs have indeed been pruned off the trunk?
Thank you!
Best regards,
Peter
02-20-2015 08:03 AM
Peter
I can although the only command I ran was "sh int trunk" as this shows which vlans are active and forwarding on the link ie. not pruned.
Like I say it was only pruned on the client switch not the transparent switch which still showed both vlans forwarding.
Remember these are not real switches, I am using the CTE lab where they have emulated switches.
But it is quick to setup so can you let me know before I do it whether there are other show commands or tests you want me to do while I have it up and running ?
Jon
02-20-2015 08:10 AM
Hi,
Thank you all for your replies so far. I ran this in the lab as stated by Jon and it never dropped the traffic to the nodes.
vtp server
vtp transparent
vlab-node1#sh interfaces pruning
Port Vlans pruned for lack of request by neighbor
Gi0/1 120,371,375-376,473,574-578,661,666-667,672-676,774,776
Gi0/2 120,375-376,473,576-578,661,666-667
Port Vlan traffic requested of neighbor
Gi0/1 1,371,574-575,672-676,774,776
Gi0/2 1,371,672-676,776
vlab-node1#sh interfaces trunk
Port Mode Encapsulation Status Native vlan
Gi0/1 on 802.1q trunking 1
Gi0/2 on 802.1q trunking 1
Port Vlans allowed on trunk
Gi0/1 1-4094
Gi0/2 1-4094
Port Vlans allowed and active in management domain
Gi0/1 1,120,371,375-376,473,574-578,661,666-667,672-676,774,776
Gi0/2 1,120,371,375-376,473,574-578,661,666-667,672-676,774,776
Port Vlans in spanning tree forwarding state and not pruned
Gi0/1 1
Gi0/2 1,371,574-575,672-676,774,776
vlab-node1#sh mac address-table dynamic
Mac Address Table
-------------------------------------------
Vlan Mac Address Type Ports
---- ----------- -------- -----
674 0021.2827.507d DYNAMIC Gi0/32
674 0021.2882.effe DYNAMIC Gi0/30
674 0021.288f.8ee5 DYNAMIC Gi0/28
674 0050.564a.e95e DYNAMIC Gi0/36
674 0050.56a3.106f DYNAMIC Gi0/26
674 00a0.9825.b99d DYNAMIC Gi0/34
674 18ef.63af.5746 DYNAMIC Gi0/2
674 a4ba.db3d.1d06 DYNAMIC Gi0/26
674 a4ba.db3d.65ff DYNAMIC Gi0/25
1 0025.459d.f3a3 DYNAMIC Gi0/1
1 18ef.63c2.6588 DYNAMIC Gi0/2
371 0000.0000.fe00 DYNAMIC Gi0/2
371 0000.0000.fe01 DYNAMIC Gi0/2
371 0000.0c07.ac00 DYNAMIC Gi0/2
371 0014.4f25.9b7e DYNAMIC Gi0/2
371 18ef.63c2.65c3 DYNAMIC Gi0/2
371 fcfb.fb30.37c0 DYNAMIC Gi0/2
473 0000.0000.fe00 DYNAMIC Gi0/2
473 0000.0000.fe01 DYNAMIC Gi0/2
473 0014.4f25.9b7f DYNAMIC Gi0/2
574 0000.0c07.ac00 DYNAMIC Gi0/2
574 0050.5674.7843 DYNAMIC Gi0/2
574 18ef.63c2.65c7 DYNAMIC Gi0/2
672 18ef.63af.5744 DYNAMIC Gi0/2
673 18ef.63af.5745 DYNAMIC Gi0/2
675 18ef.63af.5747 DYNAMIC Gi0/2
676 0021.2882.f009 DYNAMIC Gi0/31
676 18ef.63af.5748 DYNAMIC Gi0/2
575 0000.0c07.ac00 DYNAMIC Gi0/2
575 18ef.63c2.65c4 DYNAMIC Gi0/2
576 0021.2882.f008 DYNAMIC Gi0/2
776 0021.2882.f00a DYNAMIC Gi0/2
776 18ef.63af.574a DYNAMIC Gi0/2
666 18ef.63c2.65c5 DYNAMIC Gi0/2
666 fcfb.fb30.37c0 DYNAMIC Gi0/2
667 18ef.63c2.65c6 DYNAMIC Gi0/2
Total Mac Addresses for this criterion: 36
Despite the lab results I would tend to agree that this was caused as a result of vtp pruning.
02-20-2015 08:24 AM
Couple of points.
I assume you tested from a client in a vlan that had been pruned ?
The only other thing I would say is your lab does not strictly represent production because in your lab you are running VTP v2 whereas you aren't in production.
I'm not sure it will make any difference but it may be worth trying if you get the time.
Jon
02-20-2015 12:51 PM
Did some digging on this
Cisco Internetwork Operating System Software
IOS (tm) CIGESM Software (CIGESM-I6Q4L2-M), Version 12.1(22)EA11, RELEASE SOFTWARE (fc2)
There is a limitation with vtp in this version of IOS. Now this isnt a 2950 but it is running the same code where this bug occurs.
If the 2950 has a port on a VLAN that is not requested by other
servers via their pruning messages, it will not receive flooded traffic
for that VLAN.
https://tools.cisco.com/bugsearch/bug/CSCdt48011
02-24-2015 05:39 AM
Hi,
Well, isn't that the very purpose of VTP Pruning to prevent flooding of VLAN traffic onto a port that did not request that VLAN in the first place?
Best regards,
Peter
02-24-2015 01:10 PM
Hello
Can this be down to the transparent switch not accepting the vma's message and both switch then acting as if pruning isn't enabled, Even though the ios is saying its being pruned?
2x cisco WS-C3560-24TS (c3560-ipbase-mz.122-25.SEE2/c3560-advipservicesk9-mz.122-46.SE.bin)
Before hosts interfaces are enabled
##################################
SW1#sh int trun
Port Mode Encapsulation Status Native vlan
Fa0/19 on 802.1q trunking 1
Port Vlans allowed on trunk
Fa0/19 1-4094
Port Vlans allowed and active in management domain
Fa0/19 1,10,20,30
Port Vlans in spanning tree forwarding state and not pruned
Fa0/19 1
SW1#sh vlan brief | ex 100
VLAN Name Status Ports
---- -------------------------------- --------- -------------------------------
1 default active Fa0/2, Fa0/3, Fa0/4, Fa0/5
Fa0/6, Fa0/7, Fa0/8, Fa0/9
Fa0/10, Fa0/11, Fa0/12, Fa0/13
Fa0/14, Fa0/15, Fa0/16, Fa0/17
Fa0/18, Fa0/20, Fa0/21, Fa0/22
Fa0/23, Fa0/24, Gi0/1, Gi0/2
10 VLAN0010 active Fa0/1
20 VLAN0020 active
30 VLAN0030 active
SW1#sh vtp status
VTP Version : running VTP2
Configuration Revision : 11
Maximum VLANs supported locally : 1005
Number of existing VLANs : 8
VTP Operating Mode : Server
VTP Domain Name : Cust
VTP Pruning Mode : Enabled
VTP V2 Mode : Enabled
VTP Traps Generation : Disabled
SW1#sh int fa0/19 pruning
Port Vlans pruned for lack of request by neighbor
Fa0/19 10,20,30
Port Vlan traffic requested of neighbor
Fa0/19 1
SW2#sh int trunk
Port Mode Encapsulation Status Native vlan
Fa0/19 on 802.1q trunking 1
Port Vlans allowed on trunk
Fa0/19 1-4094
Port Vlans allowed and active in management domain
Fa0/19 1,10,20,30
Port Vlans in spanning tree forwarding state and not pruned
Fa0/19 1
SW2#sh vlan brief | ex 100
VLAN Name Status Ports
---- -------------------------------- --------- -------------------------------
1 default active Fa0/1, Fa0/3, Fa0/4, Fa0/5
Fa0/6, Fa0/7, Fa0/8, Fa0/9
Fa0/10, Fa0/11, Fa0/12, Fa0/13
Fa0/14, Fa0/15, Fa0/16, Fa0/17
Fa0/18, Fa0/20, Fa0/21, Fa0/22
Fa0/23, Fa0/24, Gi0/1, Gi0/2
10 VLAN0010 active Fa0/2
20 VLAN0020 active
30 VLAN0030 active
SW2#sh vtp status
VTP Version : running VTP2
Configuration Revision : 11
Maximum VLANs supported locally : 1005
Number of existing VLANs : 8
VTP Operating Mode : Client
VTP Domain Name : Cust
VTP Pruning Mode : Enabled
VTP V2 Mode : Enabled
VTP Traps Generation : Disabled
MD5 digest : 0x5F 0xCC 0xAB 0xB6 0x6B 0x8E 0x5A 0x2E
SW2#sh int fa0/19 pruning
Port Vlans pruned for lack of request by neighbor
Fa0/19 10,20,30
Port Vlan traffic requested of neighbor
Fa0/19 1
After hosts interfaces are enabled
##################################
SW1#sh int fa0/19 pruning
Port Vlans pruned for lack of request by neighbor
Fa0/19 20,30
Port Vlan traffic requested of neighbor
Fa0/19 1,10
SW2#sh int fa0/19 pruning
Port Vlans pruned for lack of request by neighbor
Fa0/19 20,30
Port Vlan traffic requested of neighbor
Fa0/19 1,10
Change to vtp mode transparent and Debugs
====================================
SWxdebug sw-vlan vtp pruning
SW1(config)#vtp mode transparent
SW1
VTP PRUNING DEBUG: trunk Fa0/19 rx Join, len=166 (domain Cust)
VTP PRUNING DEBUG: trunk Fa0/19 timeout
VTP PRUNING INFO: chg domain Cust to transparent
VTP PRUNING INFO: TFa0/19,V10: st Joined ,event Trans =>new st NoPrune
VTP PRUNING INFO: TFa0/19,V20: st Pruned ,event Trans =>new st NoPrune
VTP PRUNING INFO: TFa0/19,V30: st Pruned ,event Trans =>new st NoPrune
SW2
SW2#VTP PRUNING DEBUG: trunk Fa0/19 timeout
SW2#VTP PRUNING DEBUG: trunk Fa0/19 timeout
SW2#VTP PRUNING DEBUG: trunk Fa0/19 timeout
SW2#VTP PRUNING DEBUG: trunk Fa0/19 timeout
SW1#sh int fa0/19 pruning
Port Vlans pruned for lack of request by neighbor
Fa0/19 none
Port Vlan traffic requested of neighbor
Fa0/19 1,10
SW1#sh int trunk
Port Mode Encapsulation Status Native vlan
Fa0/19 on 802.1q trunking 1
Port Vlans allowed on trunk
Fa0/19 1-4094
Port Vlans allowed and active in management domain
Fa0/19 1,10,20,30
Port Vlans in spanning tree forwarding state and not pruned
Fa0/19 1,10,20,30
SW2#sh int fa0/19 pruning
Port Vlans pruned for lack of request by neighbor
Fa0/19 10,20,30
Port Vlan traffic requested of neighbor
Fa0/19 1,10
SW2#sh int trunk
Port Mode Encapsulation Status Native vlan
Fa0/19 on 802.1q trunking 1
Port Vlans allowed on trunk
Fa0/19 1-4094
Port Vlans allowed and active in management domain
Fa0/19 1,10,20,30
Port Vlans in spanning tree forwarding state and not pruned
Fa0/19 1
SW1#clear mac address-table dynamic
SW2#clear mac address-table dynamic
Host1#ping 10.1.12.2
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.1.12.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/4 ms
Host1#sh arp
Protocol Address Age (min) Hardware Addr Type Interface
Internet 10.1.12.1 - 0000.1111.1111 ARPA FastEthernet0/0
Internet 10.1.12.2 0 0000.2222.2222 ARPA FastEthernet0/0
SW1#sh mac address-table dynamic
Mac Address Table
-------------------------------------------
Vlan Mac Address Type Ports
---- ----------- -------- -----
1 001e.794c.c515 DYNAMIC Fa0/19
10 0000.1111.1111 DYNAMIC Fa0/1
10 0000.2222.2222 DYNAMIC Fa0/19
10 001e.794c.c515 DYNAMIC Fa0/19
20 001e.794c.c515 DYNAMIC Fa0/19
30 001e.794c.c515 DYNAMIC Fa0/19
Total Mac Addresses for this criterion: 6
Host2#ping 10.1.12.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.1.12.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/4 ms
Host2#sh arp
Protocol Address Age (min) Hardware Addr Type Interface
Internet 10.1.12.1 1 0000.1111.1111 ARPA FastEthernet0/1
Internet 10.1.12.2 - 0000.2222.2222 ARPA FastEthernet0/1
SW2#sh mac address-table dynamic
Mac Address Table
-------------------------------------------
Vlan Mac Address Type Ports
---- ----------- -------- -----
1 001e.794c.c715 DYNAMIC Fa0/19
10 0000.1111.1111 DYNAMIC Fa0/19
10 0000.2222.2222 DYNAMIC Fa0/2
Total Mac Addresses for this criterion: 3
res
Paul
02-24-2015 01:58 PM
Paul
I don't know is the short answer.
If the client switch is showing the vlan as pruned as far as i understand it then it shouldn't be sending any traffic on that link because it believes there is nothing on the other end of the link for that vlan.
The client switch may be making requests to the transparent switch but it's obvious from your output that the client switch has received no join requests from the transparent switch so it has pruned the vlan off.
From what you are suggesting the VTP joins also have in effect a VTP ack from the other switch and because no ack was received by the client switch it doesn't actually prune the vlan.
I'm not aware this happens but I'm no expert on VTP so i may be wrong.
I saw the same behaviour you did, as did the OP, so something is happening.
I also agree with Peter's last post about the "bug".
I'm not sure why that is a bug as it seems to be a perfect description of VTP pruning and what it is meant to do :-)
Jon
02-20-2015 05:44 AM
Hi there,
When you change server-mode to transparent mode, those local layer2 definition from sw01 exists on client switch as well?
On the client switch, those vlans (whatever vlans maybe) vlan instance will exist only when ports are in up state for a given vlan, otherwise your clients switch does not know about those vlans.
Have you checked the trunk between sw01 and bc01 permit all vlans?
thanks
Rizwan Rafeek
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide