VLAN issues

[dbronco]

I recently replaced the core switch at one of my sites that consisted of two separate C9300-48Ts and three other off brand switches and installed a new stack with a C9300X-48HX, C9300-24S and a C9300-48T (running 17.12.5). This is a layer 2 network consisting of about 800 cameras, 75 workstations for viewing and about 20 phones. 

After the replacement, all the workstations and phones came back online and all but 8 cameras came back online. The cameras are on the network - they are pingable but the video management system cannot display them due to a poor connection. These cameras are all on the same floor, plugged into the same switch (another C9300X-48HX that was replaced 3 weeks before the core replacement) and all on VLAN 4. I understand ping is not a troubleshooting tool but I think it's worth mentioning that when pinging these particular cameras, from anywhere on the compound, I receive between 15-25 ms response times. There are approximately 225 other cameras across the compound on VLAN 4, however, and none of them have the same problem. I thought maybe it was the model of camera but brought up a different model, put it on VLAN 4 and still have the slow ping times. VLAN 2 (workstations) and VLAN 27 (phones) are also on this switch and aren't experiencing any issues. I tested this new model of camera on VLAN 6 and have the same slow ping times. Changing the IP again to VLAN 8 though - no problem at all. So, the Band-Aid is in place to get these cameras back online but I need to figure out what is going on with VLAN 6 and 8 between this switch and my core switch. 

The core switch is my gateway for all the interface VLANs, it's my root bridge for Spanning Tree and the entire site is running VTP transparent. I'm not getting any errors in the logs and I don't see anything weird in Wireshark but now I'm not sure where to start troubleshooting. So, I'm reaching out to the community to see if you all can help point me in the right direction. I can have the configs posted for each switch tomorrow along with any other output that might be helpful. 

[David Ruess]

Hello,

Until we receive the configs you could try a few things:

1. Make sure Spanning tree is flowing how you want. Check each switch for Forwarding/Root/Blocked ports. Make sure traffic flo through the switches is efficient.

-I would also compare this to other VLANs. You say VLAN 4 is "slow" but VLAN 2 and 27 are working fine. See if VLAN 4 has the same ports forwarding as these "fast" VLANs.

2. I would enable Portfast on camera ports (if not already)

3. Verify your default GW for the slower VLANs is on the correct switch.

Hope that helps

-David

[pieterh]

>>> the entire site is running VTP transparent <<<
-> verify you have created all the necessary vlan's manualy , including on traversing switches and allowed on the uplinks

[Joseph W. Doherty]

You describe your network as L2 but then mention multiple VLANs, with different behaviors.(?)

You further mention 800 cameras and 75 workstations to view camera video but without any description of topology, logical or physical, nor any descriptions of the bandwidth each camera generates (which can vary considerably based on various video options).

Although we could probably construct your topology with full configurations, it would be helpful if you could also post any additional information, like network diagrams.

Also, have you examined switch interface stats?

[MHM Cisco World]

Hi friend 

Let first start check l2 storm' 

Show interface x/x | in broadcast | in multicast 

Check the boradcast and multicast count is it increase rapidly 

Then we will start looking at IOSd of IOS XE to see where drop is happened 

MHM

[MHM Cisco World]

9300_20#show platform hardware fed switch active fwd-asic resource tcam utilization

check the L2 and L3 multicast TCAM resource 

MHM

[dbronco]

Thank you for all the responses, I've attached a few files and have some notes from what I noticed today. 

The site topology (while very basic) shows how this site, we'll call Site B, connects as part of a spoke. Site A is my Hub. 

I tested removing Site B from Site A today to ensure that the problem was localized and confirmed there was no change. When disconnecting 9.6 from 9.1 and putting my laptop on VLAN4, I can ping the camera without issue. Once the switch reconnects to 9.1, my pings go from 1ms to ~23ms. I also noticed a dramatic number of output drops on 9.6 int 1/0/36 where the camera on VLAN4 is. This was the same with different Ethernet cables so I imagine this isn't a Layer 1, and probably relates to what's going on, but I'm at a loss. 

I checked spanning tree and 9.1 is the root bridge for all VLANs and all ports are forwarding. 

I ran the commands from MHM and while the L2 resources were low, I did notice my control plane percentage was higher than I would have thought, is this normal? I ran this command on my 26.1 switch and received the exact same numbers and percentage - is this expected? I've never ran these commands before. I also don't fully understand why I have v6 packets when everything should only be IPv4, this is an air-gapped network not connected to the internet. 

We have each camera to stream at approximately 3Mbps. I attached two screenshots to show bandwidth for each of the VLANs as well as out to my distro switches. What I found odd about this is that gi2/0/11 is a small 8 port switch with two cameras and a phone on it, why do I have a TX of 500 Mbps? There isn't a computer or viewing station here so in reality my RX from this switch looks correct and I should have next to nothing for TX. 

This is beyond what I've seen previously so any help is appreciated. Please let me know if additional output is necessary. Thank you all! 

[MHM Cisco World]

Control plane take 56% of the tcam 

You have large CoPP' are this CoPP same from old SW?

MHM