High Latency / Packet Loss - IP Camera System - New VLAN... Help

[Derek Parish]

I have been troubleshooting an issue today that I have yet to get to the bottom of.

We have a vendor installing a new camera system and we setup a new switch stack for them to connect the server, storage, and cameras to.

The server, storage, and cameras are all on the same subnet/vlan.

The odd issue is that everything works fine until the application on the camera server actually makes connection with an IP camera.  From there it is causing high latency and packet loss even for a different VLAN.  If I have the server and storage up with the cameras disconnected everything works fine.  If I have the cameras up with the server offline, everything works fine.  If i even have the server and cameras up at the same time but do not have the server configured to point to the cameras it still works fine, it is only when the connection between the server and the camera happens that immediately the problems start.

I have attached a really high-level diagram of the setup.

In the diagram if we have the server and cameras all turned up, the "random device vlan 1" will ultimately have latency spikes and start dropping packets.  There are no physical loops or ether-channels in this setup.

Any idea off the top of your head's?  I can get more information/configs tomorrow.

Also i know some of these configs are definitely not best practices but some of the old managent and native vlan configs were left over and we have been slowly changing everything.

Thanks.

[johnd2310]

Hi,

You will need to determine what traffic is being generated when server tries to access camera. Span a port on the 3650 stack and capture traffic from the camera. I am guessing this traffic is causing high cpu on switch. Check the processor utilization on your switches  and check if any process is maxing the cpu.

Thanks

John

[Mark Malone]

Are you using multicast or qos for the cameras to the application

are the cameras using unicast or broadcast for the feeds

what exactly are the cameras using in bw I have ip cameras they range from being able to use 3mb HD to 6mb and they can be tuned and the latter causes issues on our smaller switches like 29s and 37s that don't have large buffers

find out what dscp marking they are using

when the cameras and the application connect do the interfaces max out or anything

As John said collect the show proc cpu sorted while the issue is occurring on the switch it will give an indication to what process is causing it

I would start building up the whole picture how many cameras , total bbw with what the stack can handle in bw , do these units require multicast , if there broadcast can the vendor switch them to unicast to relieve the network etc the more info you know about the setup easier it will be to resolve it , whats the port on the server looking like when its on maxed , running low

what vlan is everything currently on

[Derek Parish]

The system is Genetec Security Desk.

I will work on getting some more details today while the issue occurs. 

Also the camera system is on VLAN27 while definite issues start occurring on  VLAN1 and thereafter other user/data vlans.

Will take a capture and get more info soon.

Thanks.0

[ole weel]

Hi Derek,

Did you solve this problem ? Reason for asking is that we have latency problems also

/Regards

Ole