VTP mode transparent not working randomly in multiple switches

[MParashar]

Hi,

We have a VTP server-client setup for our Network. Details are as below.

1. Layer 3 switch configured as VTP server (with VTP version 2 and VTP pruning enabled).It's having a vlan database of around 90 vlans.

2. 57 wifi switches(cisco 2960 24PDL ,POE functionality) and 18 L2 switches (cisco 2960(different models)) in client mode.

Problem Description:

1. Due to this much large vlan database , switches cpu utilization was spiking randomly up to 99%. Main contributor processes are HULC LED, Spanning-tree, IP SNMP ,SNMP Engine. 

2. We have 3 NMS servers configured for Network monitoring(cisco prime(for wifi only), Zabbix & Observium(for both wired and wireless switches monitoring & syslogging).

3. I had tried to reduce vlan database on each wifi switch by configuring the switches in vtp transparent mode and only configuring the required vlans on each switch and it worked for me. After configuration all the switches cpu utilization has been dropped by 10-15% and no cpu spikes observed after that.

4. Problem here is few switches when wents down due to power outage or something else ,they starts misbehaving. Stops providing IP in configured vlan. Switch remain up and active on Network. I can ssh /telnet it but not provding any IP. I have to then change the vtp configuration mode of the switch from transparent to client and It starts working again.

5. Regarding topology: Most of the wi-fi switches are directly connected to L3 switch through fibre connectivity  ,few switches are connected  like L3 switch-(fibre)-L2 Switch-(copper)-wifi_Switch.

Anyone Please help me in resolving this.

[marce1000]

  >... Problem here is few switches when wents down due to power outage or something else ,they starts misbehaving.

 I hardly believes this correlates with the subject of your post , in such circumstances I would strongly advise to isolate these devices from the network, configure the vtp transparant  mode and needed vlans (as wanted) ,echo-verify the intended configuration when done and then connect them to the network again.

 M.

[MParashar]

Thank you for your response. Will definitely recheck all the switches configuration again.

[Joseph W. Doherty]

BTW, I've dealt with a VLAN issue caused by VTP pruning.  I recall (???) it was due to something like sw1 with VLAN 5 ports <> sw2 with no VLAN 5 ports <> sw3 with VLAN 5 ports with VTP pruning off VLAN 5 to sw2.  I recall VTP pruning is a bit slow to prune, so when switches first come up VLAN 5 works on sw3 but after a few minutes, it stops working.

I'm sort of surprised you saw the amount of CPU reduction you did by moving some switches to VTP transparent mode.  For those, they still need to pass along the VTP database?  If not, if your switches support it, you might also try changing the switch to VTP off mode.

For #1, HULC LED process being a large CPU consumer is not unusual.  You can search these forums and Cisco's main site for more info on that issue.  That noted, I recall, it generally doesn't impact switch operation.

Also for #1, as you have around 90 VLANs, also not surprising STP is busy.  What variant of STP are you using?  As Cisco defaults to using a per-VLAN STP, perhaps you might consider moving to MST (also assuming you don't need to maintain around 90 STP topologies).

For #1's SNMP, also unexpected if your do "frequent" and/or "extensive" polling of data.  Might be reduced by careful review and possible adjustment of what you need to obtain for management purposes and/or using SNMP device agents to notify SNMP management station of issues (rather than polling for problems).

#4 sounds like something missing in the way you've configured your devices.  Transparent and off mode VTP maintain their own VLAN database.  VTP server and clients share a VLAN database.  Makes me wonder about a possible VTP pruning issue (why I started this post with an issue I worked on with it).

[MParashar]

Hi Joseph,

Thank you for the response. Below is the VTP configuration details for both client and transparent mode.

1. With VTP client mode

test_switch1#sh vtp status
VTP Version capable : 1 to 3
VTP version running : 2
VTP Domain Name : xxxxxxxxx
VTP Pruning Mode : Enabled
VTP Traps Generation : Disabled
Device ID : 08cc.a7ff.df80
Configuration last modified by 10.0.1.1 at 3-2-22 10:04:07

Feature VLAN:
--------------
VTP Operating Mode : Client
Maximum VLANs supported locally : 1005
Number of existing VLANs : 93
Configuration Revision : 257

2. With VTP transparent mode for same switch

test_switch1#sh vtp status
VTP Version capable : 1 to 3
VTP version running : 2
VTP Domain Name : xxxxxxxx
VTP Pruning Mode : Enabled
VTP Traps Generation : Disabled
Device ID : 08cc.a7ff.df80
Configuration last modified by x.x.x.x at 3-2-22 10:04:07

Feature VLAN:
--------------
VTP Operating Mode : Transparent
Maximum VLANs supported locally : 1005
Number of existing VLANs : 7
Configuration Revision : 0

STP configuration in switch

spanning-tree mode rapid-pvst
spanning-tree portfast edge default
spanning-tree extend system-id
spanning-tree uplinkfast

As you can see a clearcut reduction in vlan database in vtp transparent mode dropped to around 8% in comparison to vtp client mode and hence the below process consuming less cpu cycles due to this reduction.

1 STP: With reduced vlan database overhead of STP has been reduced

2. IP SNMP & SNMP Engine: As for every vlan a child snmp string is defined by switch for polling SNMP data as shown below.

With VTP transparent mode:

Community name: test
Community Index: test
Community SecurityName: test
storage-type: nonvolatile active

Community name: test@1
Community Index: test@1
Community SecurityName: test
storage-type: read-only active

Community name: test@1002
Community Index: test@1002
Community SecurityName: test
storage-type: read-only active

Community name: test@1003
Community Index: test@1003
Community SecurityName: test
storage-type: read-only active

Community name: test@1004
Community Index: test@1004
Community SecurityName: test
storage-type: read-only active

Community name: test@1005
Community Index: test@1005
Community SecurityName: test
storage-type: read-only active

Community name: test@301
Community Index: test@301
Community SecurityName: test
storage-type: read-only active

Community name: test@70
Community Index: test@70
Community SecurityName: test
storage-type: read-only active

With VTP configuration  mode client this number has been increased to 93, which is a very large value and hence SNMP polling was consuming high CPU. Reason for dropping of CPU utilization in transparent mode.

I am not worried about HULC LED process. It's expected. I was worried about STP, IP SNMP & SNMP Engine processes, which has been handled now. My  only problem area is the few switches misbehaviour due to configuration changes  to transparent mode.  Earlier i did the vtp configuration mode to off in all wifi switches but misbehaviour from the same switches was there at that time also. I had then checked the details on few communities and found the VTP version 2 doesn't support vtp mode off in client switches and thought that it could be the reason of switches misbehaviour and hence changed the switches vtp mode to transparent but issue still persists.

[Joseph W. Doherty]

Ah, VTP pruning mode is enabled (which you also mentioned in OP, but I overlooked that).

I suggest disabling VTP pruning, and prune manually.

"I had then checked the details on few communities and found the VTP version 2 doesn't support vtp mode off in client switches . . ."

Hmm, not something I recall bumping into.  Yes, older switches' IOS didn't support "off" mode, but don't recall using it (when an option) being dependent on switches current VTP mode.  Further, only real difference, at least in V1/2 between server and client modes is the former allows VLAN config changes on the device, internally, I understand both work alike in how they share the VLAN database.  (I.e. the only real issue with having more than one VTP V1/2 server, is multiple servers might be updating the VLAN database at the same time, and then you have a possible race condition for which VLAN database ends up on clients [i.e. it could be from either server, assuming each has same rev number].  If you later update VLANs on just one of the servers, it will replace all VLAN databases, likely erasing changes made on other VTP servers.)

The only reason I can think of why having VTP active, or not, would change impact of STP CPU usage, is with VTP active, even with VTP pruning enabled, VLAN L2 topologies might be larger vs. how they appear without VTP active.  I.e. smaller VLAN topologies should, I think, require less STP processing.

Again, assuming you don't need so many unique L2 STP topologies, MST, perhaps, even only needing one topology instance, would, I suspect, might be the best way to decrease STP CPU consumption.

Lastly, often overlooked/misunderstood on switches, high CPU consumption, even sustained 100%, might not be adverse to switch network operation at all; because data plane, generally is supported by separate dedicated hardware, and CPU processes, are usually prioritized (low priority processes, using "excess/available" CPU are often a non-problem, beyond possibly having management software/systems flagging overall high switch CPU usage).

[MHM Cisco World]

SW get VLAN 93 from VTP server, 
the SW run PVST and generate one BPDU for each VLAN or modify it if it not Root

but that can also control 
*if the SW transit then no way you need to allow VLAN and make SW generate or modify the BPDU, 
but using MST can reduce the number of BPDU 

*the SW is not transit, here you can allow only VLAN that YOU assign access port to it other VLAN no need to generate or modify or process it 
this done by allow specific VLAN in trunk.
VTP server add VLAN to DB but each VTP client use those need it. 

[MHM Cisco World]

from cisco 
""For VTP pruning to be effective, all devices in the management domain must either support VTP pruning
or, on devices that do not support VTP pruning, you must manually configure the VLANs allowed on
trunks. ""

[MHM Cisco World]

Your issue is interest, 
the VTP pruning is have little delay, 
so we talk about manual and that what you do and SW CPU is stable, 
or what you do if you want to run VTP and VTP pruning ?
I read and read how we can solve this 
MST as mention above 
or in somehow delay BPDU from Root SW (play with STP timer)
 https://www.cisco.com/c/en/us/support/docs/lan-switching/spanning-tree-protocol/19120-122.html

check both solution.