Slow switch management when configured more than 6 VLANs

[remon.braamse]

In short, when I configure more than 6 vlans on my Cisco C1000 switch, communication to the switch on the management VLAN becomes realy slow. Configuration via SSH is realy slow and ping to the switch has a delay of around 700ms.

For the network here I have an stack of 2 Cisco C9300-24UX switches with IOSXE version 16.11.1. For 2 cameras in a sepperate building I got an Cisco C1000-16PS-2G switch with IOS version 15.2.7.E5. The switches are connected with an optical fiber.

I configured the C1000 switch and manual added 2 VLAN's for the trunk and management. After that I connected the switch to the C9300 and everything worked well. After that I configured VTP so all our VLAN's came availeble on the C1000. Directly after that SSH to the switch went realy slow and ping times where going up to 700-750ms.

After this I removed the VTP config and removed all VLANs exept the 2 for trunk and management. After reloading the switch SSH management en ping times are back to normal.
Next I manualy added the VLAN's to the switch. With the first 6 VLANs all works well. When added the 7th VLAN de pingtimes go up en SSH become slow. The more VLANs after this the higer the ping times become and the slower SSH will respond.
I tested adding the VLANs in a different order, but it is always at the 7th VLAN. Also I tested with an new Cisco C1000 switch without config, only the trunk config and the managment VLAN + managment IP and I got the same results.

I could use some help with this problem.

(I'm sorry for my writing, it is not realy good in english

[Georg Pauwen]

Hello,

which switch is the root for the Vlans ? Make sure it is not the C1000.

[remon.braamse]

Thanks for you suggestion.

I checked it and the C9300 is root for all VLANs.

[balaji.bandi]

In short, when I configure more than 6 vlans on my Cisco C1000 switch, communication to the switch on the management VLAN becomes realy slow. Configuration via SSH is realy slow and ping to the switch has a delay of around 700ms.

how is these conencted, what device is trying to loging and show slow ? where is that source resides ?

Do you high level network diagram and some config bit ? (how is your network routing ?)

7th VLAN created, all vlan go slow only 7th vlan only ?

[remon.braamse]

---

how is these conencted, what device is trying to loging and show slow ? where is that source resides ?

---

The C1000 is with optical fiber connected to the C9300 switch. At this moment for testing with an short cable of 1m. But, when i connect the switches by UTP cable the problems are the same.
I tried logging in from my laptop from an other VLAN but also from the C9300 using the same vlan that the management ip of the C1000 is on. Slow in SSH for exemple I typ "show run" and it takes around 2 seconds for the letters to appear.

---

Do you high level network diagram and some config bit ? (how is your network routing ?)

---

The C9300 is routing for 10 vlan's. In total there are 18 vlans. Do you need the complete running config or some specific bits of config?

---

7th VLAN created, all vlan go slow only 7th vlan only ?

---

Only management for the switch goes slow. Devices connected to the switch in any VLAN works well, but communication to the switch become slow. So only ping and SSH to the switch itself.

[balaji.bandi]

complete running config 

can you provide removing confideniall information

The device IP address you try login using SSH  also help here.

[remon.braamse]

Attached are the running configs from both switches.

The C9300 uses 172.31.16.1, the C1000 172.31.16.18 and my laptop has 172.31.18.180.

I tested SSH from my laptop and from the C9300. Both are slow.

[balaji.bandi]

So just to clarity :

172.31.18.180. from this device you trying to SSH to  !The C9300 uses 172.31.16.1, the C1000 172.31.16.18 "

172.31.18.180 - where is this connected ? what port ? is this on cat 9300 or C1000 ?

172.31.18.180  if you do ssh to 172.31.18.1  is that good ?

[remon.braamse]

I tried SSH from 172.31.18.180. This device is in VLAN105 at an other switch in the network.

Also i tried SSH from the C9300 itself (172.31.16.1).

In both cases I SSH to 172.31.16.18 (the C1000 switch)

[balaji.bandi]

Still we are not clear-  in this case only Cat1000 having issue with SSH or any other switch.

172.31.18.180  - what if this device connect to same switch:

what is the status if the device 172.31.16.X and connect to switches ?

can you post below output :

show vlan

show spann brief

show vtp status

check some limitation is this approval :

https://www.cisco.com/c/en/us/products/collateral/switches/catalyst-1000-series-switches/nb-06-cat1k-ser-switch-faq-cte-en.html

[remon.braamse]

Still we are not clear-  in this case only Cat1000 having issue with SSH or any other switch.

Yes, only the Catalyst1000 has this issue. Other switches are stacks witch 9200L switches. SSH to these switches works fine and ping time are 1ms or less. Also there is one Cisco SG350 on the network. This switch also has nog problems.

---

172.31.18.180  - what if this device connect to same switch:

what is the status if the device 172.31.16.X and connect to switches ?

Directly connected to the C1000 with vlan 105 (same as previous tests) using ip address 172.31.18.180, SSH to the switch is still slow and ping times are still 700ms average.

Connected to the C1000 with VLAN 101 (same as management for C1000) using ip address 172.31.16.56 SSH to the switch is still slow and ping times are still 700ms average.

---

can you post below output :

show vlan

show spann brief

show vtp status

---

Attached the outputs on both switches. Show spann brief is not accepted by both switches so I added the show spanning-tree bridge output.

---

check some limitation is this approval :

https://www.cisco.com/c/en/us/products/collateral/switches/catalyst-1000-series-switches/nb-06-cat1k-ser-switch-faq-cte-en.html

---

We do not have 64VLANs or more, so I do not think that is the problem.



 

[Leo Laohoo]

What IOS version is the C1000 on?

Post the complete output to the command "sh proc cpu sort | ex 0.00".

[remon.braamse]

The C1000 uese IOS 15.2(7)E5.

Below is the output of the asked command on the C1000 switch.

C1000#show processes cpu sorted | ex 0.00
CPU utilization for five seconds: 13%/4%; one minute: 14%; five minutes: 14%
 PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process 
 108     2326889      775211       3001  2.85%  2.78%  2.78%   0 HaySel LinkState 
 191     1766398    26428591         66  2.09%  2.04%  2.06%   0 HAYSEL Process M 
 107      212652       53241       3994  0.31%  0.25%  0.24%   0 hpm main process 
  80      192519       77460       2485  0.29%  0.28%  0.28%   0 AgingTask        
  71      592104       73885       8013  0.27%  0.66%  0.67%   0 RedEarth I2C dri 
 112      155437       77534       2004  0.19%  0.18%  0.18%   0 hpm counter proc 
 181       45507      543670         83  0.15%  0.05%  0.04%   0 Spanning Tree    
 134        1635        1029       1588  0.11%  0.04%  0.09%   1 SSH Process      
 193        5398       77452         69  0.03%  0.03%  0.02%   0 PI MATM Aging Pr 
  72       77077       64282       1199  0.01%  0.08%  0.08%   0 RedEarth Tx Mana 

C1000#

[pieterh]

you only mention the use of vlan's, but what subnets do you use on those vlan's ?
if you are using the same subnet spread across multiple vlan's then that will be the reason for the delay.

it is common practice to use a separate subnet for each vlan, traffic is more under control this way.
of course you need to add some routing configuration between those subnets to complete the network design.

[remon.braamse]

All VLANs have their own subnet. In the entire network we have no duplicate subnets.