VLAN issues

[dbronco]

I recently replaced the core switch at one of my sites that consisted of two separate C9300-48Ts and three other off brand switches and installed a new stack with a C9300X-48HX, C9300-24S and a C9300-48T (running 17.12.5). This is a layer 2 network consisting of about 800 cameras, 75 workstations for viewing and about 20 phones. 

After the replacement, all the workstations and phones came back online and all but 8 cameras came back online. The cameras are on the network - they are pingable but the video management system cannot display them due to a poor connection. These cameras are all on the same floor, plugged into the same switch (another C9300X-48HX that was replaced 3 weeks before the core replacement) and all on VLAN 4. I understand ping is not a troubleshooting tool but I think it's worth mentioning that when pinging these particular cameras, from anywhere on the compound, I receive between 15-25 ms response times. There are approximately 225 other cameras across the compound on VLAN 4, however, and none of them have the same problem. I thought maybe it was the model of camera but brought up a different model, put it on VLAN 4 and still have the slow ping times. VLAN 2 (workstations) and VLAN 27 (phones) are also on this switch and aren't experiencing any issues. I tested this new model of camera on VLAN 6 and have the same slow ping times. Changing the IP again to VLAN 8 though - no problem at all. So, the Band-Aid is in place to get these cameras back online but I need to figure out what is going on with VLAN 6 and 8 between this switch and my core switch. 

The core switch is my gateway for all the interface VLANs, it's my root bridge for Spanning Tree and the entire site is running VTP transparent. I'm not getting any errors in the logs and I don't see anything weird in Wireshark but now I'm not sure where to start troubleshooting. So, I'm reaching out to the community to see if you all can help point me in the right direction. I can have the configs posted for each switch tomorrow along with any other output that might be helpful. 

[Joseph W. Doherty]

Not just queueing, drops too, I believe.

[wajidhassan]

Hi @dbronco ,

It sounds like the issue may be isolated to VLAN 4 on that specific switch. Since pings are high and video streams are unstable, I'd suggest checking the following:

• Port settings: Verify interface duplex/speed settings for the affected ports—mismatches can cause delay and jitter.

• Uplink path: Confirm the uplink from this switch to the core is clean (no drops/errors) and trunked correctly with all VLANs allowed.

• STP topology: Check if this switch is blocking or forwarding for VLAN 4—unexpected STP behavior could affect traffic paths.

• Camera switch buffer/CPU: Monitor CPU and buffer utilization on the access switch—camera bursts may cause buffer overflows if not handled well.

• Storm-control or QoS: See if there are any policies limiting traffic rates on those ports or VLANs.

You’ve ruled out the camera models and VLAN IPs well—next step would be verifying the physical and Layer 2 conditions.

Hope this helps guide the troubleshooting.

[MHM Cisco World]

There are one policer control three traffic toward three queues 

We see exceeded in policer which make cpu drop packet 

We check queue logging and sw-forward 

There are many packets in these queue 

1- logging 

I know you config external server but there is huge number of logging' 

I see you enable ntp and success login logging' so either you under attack of hacket try to access your SW or your ntp have issue generate these huge numbers of log message 

2- sw-forward 

Normally mcast forward by tcam but if mcast have ttl 1 then it need to forward by cpu that why we see high numbers of packet in sw-forward queue

@dbronco @Joseph W. Doherty 

MHM

[pieterh]

>>> There are approximately 225 other cameras across the compound on VLAN 4, however, and none of them have the same problem <<<

-> did you double-check the network configuration of the camera's?
looks to me like the camera's on this floor/this switch cannot find their default-gateway after the replacement of network components and the echo-reply is not reaching the correct return address
either because of ARP caching, subnet mismatch or gateway mismatch
maybe these camera's previously depended on proxy-arp configured in the old network?

[dbronco]

Hi @pieterh 

My first thought was the network config on the cameras themselves as well. Bringing a new camera up to that floor with correct configs replicated the issue though. 

I will be reviewing old configs today, what would it look like if proxy-arp was configured on the old network? It's not something we've done across the rest of our network but as I mentioned in a previous post, this site/network was used to test some theories in the past. 

[Joseph W. Doherty]

I will be reviewing old configs today, what would it look like if proxy-arp was configured on the old network?

Two things, gateway interface configured as proxy (newer switches, I believe now default it being disabled).  Host doesn't have a gateway IP defined.

[dbronco]

The old 9.1 doesn't have itself listed as ip-default gateway. But the old 9.2 was pointed to 9.1 as being the gateway. I didn't notice anything else related to this

[pieterh]

>>>
[THIS IS A CAMERA ON VLAN4]
9.6#show interface te1/0/36
TenGigabitEthernet1/0/36 is up, line protocol is up (connected)
Hardware is Ten Gigabit Ethernet, address is 68e5.9eca.ca24 (bia 68e5.9eca.ca24)
Description: VLAN4 Test
MTU 1500 bytes, BW 100000 Kbit/sec, DLY 100 usec,
reliability 255/255, txload 249/255, rxload 1/255
<<<

BW 100.000 Kbit/sec -> a 100Mbps camera connected to a TenGigabitEthernet capable interface ?
maybe that is contributing  to the problem -> set the interface speed to autonegotiate on 100Mbps max.

how many devices are attached to this vlan4 in total?
maybe a /24 subnet is not large enough and has been /23 before the replacement?
so check the camera's network configuration , check the DHCP scope and check the routers netmask for vlan4

[MHM Cisco World]

If there is high output then sure one link have high input

Use 

Show interface  | in is up | input rate

Then check which interface have high input 

Use 

Show cdp neighbor 

See if 

A- interface is connect to other SW then there is issue in other SW (l2 loop maybe)

B- interface connect to host/server disconnect it and check your network 

MHM

[MHM Cisco World]

Show interface  | in is up | input rate <<- this please alos

MHM

[dbronco]

I'll have this tomorrow morning. Thank you

[dbronco]

Sorry for the delay. Attached

[MHM Cisco World]

TenGigabitEthernet2/1/1 is up, line protocol is up (connected) 
  5 minute input rate 836511000 bits/sec, 75996 packets/sec
TenGigabitEthernet2/1/2 is up, line protocol is up (connected) 
  5 minute input rate 326231000 bits/sec, 74454 packets/sec

high input rates in many port 
let start with these two what device connect to it ? use cdp to know 

MHM 

[dbronco]

These are workstations streaming video feeds. Ports 1-10 are workstations, port 11 is a phone, ports 12-19 are cameras on VLAN 8 and port 36 is a camera on VLAN 4 that I'm testing as a baseline

[MHM Cisco World]

These ports start with 2/x/x so it meaning port of other SW' are yoh use these connect to workstations?

MHM