VLAN issues

[dbronco]

I recently replaced the core switch at one of my sites that consisted of two separate C9300-48Ts and three other off brand switches and installed a new stack with a C9300X-48HX, C9300-24S and a C9300-48T (running 17.12.5). This is a layer 2 network consisting of about 800 cameras, 75 workstations for viewing and about 20 phones. 

After the replacement, all the workstations and phones came back online and all but 8 cameras came back online. The cameras are on the network - they are pingable but the video management system cannot display them due to a poor connection. These cameras are all on the same floor, plugged into the same switch (another C9300X-48HX that was replaced 3 weeks before the core replacement) and all on VLAN 4. I understand ping is not a troubleshooting tool but I think it's worth mentioning that when pinging these particular cameras, from anywhere on the compound, I receive between 15-25 ms response times. There are approximately 225 other cameras across the compound on VLAN 4, however, and none of them have the same problem. I thought maybe it was the model of camera but brought up a different model, put it on VLAN 4 and still have the slow ping times. VLAN 2 (workstations) and VLAN 27 (phones) are also on this switch and aren't experiencing any issues. I tested this new model of camera on VLAN 6 and have the same slow ping times. Changing the IP again to VLAN 8 though - no problem at all. So, the Band-Aid is in place to get these cameras back online but I need to figure out what is going on with VLAN 6 and 8 between this switch and my core switch. 

The core switch is my gateway for all the interface VLANs, it's my root bridge for Spanning Tree and the entire site is running VTP transparent. I'm not getting any errors in the logs and I don't see anything weird in Wireshark but now I'm not sure where to start troubleshooting. So, I'm reaching out to the community to see if you all can help point me in the right direction. I can have the configs posted for each switch tomorrow along with any other output that might be helpful. 

[dbronco]

Thank you for the continued support. I got pulled into something else today but will try and get this as soon as I can

[dbronco]

Attached

[Joseph W. Doherty]

What kind of logging are you doing?

Could you post a sanitized copy of switch config?

[dbronco]

I send all my logs to a PRTG server but am not receiving anything outside of normal. 

I posted a copy of the switch config from 9.6 (where I first noticed the issue) and 9.1 (my core switch for the site) on 7/15 with a few other attachments. Please let me know what you're interested in and I'll work to get anything more detailed posted as well. 

[dbronco]

I didn't see this reply earlier, apologies. 

I can get the output posted here but when I looked earlier in the week the CPU usage was less than 2%

That said, I was investigating a few things the other day and was running a continuous ping on 192.168.4.103 (a test camera I installed on switch 9.6) and did see dropped pings. Immediately after the drop, the ping times would be back to <1ms, 3ms, 3ms, 7ms, 15ms, and then steady again at 23ms. I got pulled into something else the past 2 days and haven't had a chance to look any more into that yet. 

While I'm not doubting CoPP could be a problem, especially on a large CCTV network, I'm still struggling to understand why the CoPP limit would now be a factor after replacing this hardware. Even if I unknowingly had this issue prior to the hardware replacement, why would it become more apparent with newer, more powerful hardware?  

[wajidhassan]

Hi @dbronco ,

It sounds like the issue may be isolated to VLAN 4 on that specific switch. Since pings are high and video streams are unstable, I'd suggest checking the following:

• Port settings: Verify interface duplex/speed settings for the affected ports—mismatches can cause delay and jitter.

• Uplink path: Confirm the uplink from this switch to the core is clean (no drops/errors) and trunked correctly with all VLANs allowed.

• STP topology: Check if this switch is blocking or forwarding for VLAN 4—unexpected STP behavior could affect traffic paths.

• Camera switch buffer/CPU: Monitor CPU and buffer utilization on the access switch—camera bursts may cause buffer overflows if not handled well.

• Storm-control or QoS: See if there are any policies limiting traffic rates on those ports or VLANs.

You’ve ruled out the camera models and VLAN IPs well—next step would be verifying the physical and Layer 2 conditions.

Hope this helps guide the troubleshooting.

[MHM Cisco World]

There are one policer control three traffic toward three queues 

We see exceeded in policer which make cpu drop packet 

We check queue logging and sw-forward 

There are many packets in these queue 

1- logging 

I know you config external server but there is huge number of logging' 

I see you enable ntp and success login logging' so either you under attack of hacket try to access your SW or your ntp have issue generate these huge numbers of log message 

2- sw-forward 

Normally mcast forward by tcam but if mcast have ttl 1 then it need to forward by cpu that why we see high numbers of packet in sw-forward queue

@dbronco @Joseph W. Doherty 

MHM