VLAN issues

[dbronco]

I recently replaced the core switch at one of my sites that consisted of two separate C9300-48Ts and three other off brand switches and installed a new stack with a C9300X-48HX, C9300-24S and a C9300-48T (running 17.12.5). This is a layer 2 network consisting of about 800 cameras, 75 workstations for viewing and about 20 phones. 

After the replacement, all the workstations and phones came back online and all but 8 cameras came back online. The cameras are on the network - they are pingable but the video management system cannot display them due to a poor connection. These cameras are all on the same floor, plugged into the same switch (another C9300X-48HX that was replaced 3 weeks before the core replacement) and all on VLAN 4. I understand ping is not a troubleshooting tool but I think it's worth mentioning that when pinging these particular cameras, from anywhere on the compound, I receive between 15-25 ms response times. There are approximately 225 other cameras across the compound on VLAN 4, however, and none of them have the same problem. I thought maybe it was the model of camera but brought up a different model, put it on VLAN 4 and still have the slow ping times. VLAN 2 (workstations) and VLAN 27 (phones) are also on this switch and aren't experiencing any issues. I tested this new model of camera on VLAN 6 and have the same slow ping times. Changing the IP again to VLAN 8 though - no problem at all. So, the Band-Aid is in place to get these cameras back online but I need to figure out what is going on with VLAN 6 and 8 between this switch and my core switch. 

The core switch is my gateway for all the interface VLANs, it's my root bridge for Spanning Tree and the entire site is running VTP transparent. I'm not getting any errors in the logs and I don't see anything weird in Wireshark but now I'm not sure where to start troubleshooting. So, I'm reaching out to the community to see if you all can help point me in the right direction. I can have the configs posted for each switch tomorrow along with any other output that might be helpful. 

[dbronco]

Apologies, I was looking at the wrong switch output. Those are out to distribution switches with a lot of devices connected to them

[MHM Cisco World]

As I suspect' it can l2 storm.

Do same in distribute SW and see which port have high input 

Make sure that all SW have one single root SW per vlan 

MHM

[Joseph W. Doherty]

As I suspect' it can l2 storm.

Perhaps it is, but analysis of the high volume of continuous traffic to a video camera port may identify the cause of such a storm or perhaps another issue entirely.

Again, there's no usual reason for a video camera's egress to be continuously saturated.

Rather then continuing to look at port bandwidth consumption, let's examine what's the nature of the already noted high volume traffic.

[dbronco]

I'm finally back to troubleshooting this issue and found a few things that I think might be helpful in troubleshooting this issue but the further I dig into this, the more confused I get. If anyone has any guidance or direction, I would really appreciate it. 

Yesterday I did some remote troubleshooting from another site and ran two separate packet captures using Wireshark via the WebGUI on 9.6 (the switch with the issue with cameras on VLAN4). I ran it with the source port on 1/0/36 (access vlan 4 - camera IP 4.103) and have a ton of 'TCP ACKed unseen' packets [Screenshot(1)]. However, on the same switch, with a source port of 1/0/20 (access vlan I see no issues [Screenshot(2)]. 

The next test I ran, being remote, I ran a continuous ping to two separate cameras at the same time - 4.103 (plugged into 9.6 and the problematic camera) as well as 4.150 (a random other camera on a different switch on site) and ran a packet capture on 9.1, my core switch for the site. In screenshot(4) you can see the traffic when filtering for that IP but when filtering for 4.103 in screenshot(5), there are no results. It's as if there is no gateway for this camera.

I've confirmed that the camera does indeed have a gateway and 9.6 has the gateway set as 9.1. If there was an issue with the SVI of 4.1, I would make the assumption the other 240 cameras on VLAN 4 would also be having issues. Would that not be the case? I will be back on site tomorrow and would like to plug this camera into a different switch to see if Wireshark captures the same issues but aside from that, I'm not sure what else to test.