02-12-2025 11:09 AM
As the title states, I'm reaching out to the community to hopefully learn more regarding CCTV network best practices and/or lessons learned.
First question is in regards to VLANs. The previous network engineer segmented and organized the network by device type, i.e.
- VLAN 24 = recording servers and workstations (viewing clients) (5 recording servers, 1 per building. 75 viewing clients)
- VLANs 64, 65, 66, 67, 68 = cameras across the campus (approx 800 total cameras)
- Another VLAN for switch management and there are a few more for ancillary devices.
Is this the most optimized configuration? Should the cameras be on the same VLAN as their recorder so that the only routing that is happening is when a camera is viewed compared to having all cameras routing through the primary switch before going to its designated recorder?
The next question, keeping the same site in mind (800 cameras, 5 recording servers, 75 windows PCs and 150 switches) is for NTP. Is there a maximum number of devices that a Catalyst 9300 will provide NTP updates to? I found this from Cisco https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus1000/sw/4_0/system_management/configuration/guide/n1000v_sys_manage/system_6ntp.pdf but that doesn't seem to be used for the same use case. I currently have the following configured on my core switch (Catalyst 9300-48T with 17.06)
ntp logging
ntp server 192.168.26.1
I have the 150 switches pointed to this switch and don't have any errors but before going any further, I want to ensure this is the right way forward.
Solved! Go to Solution.
02-12-2025 09:03 PM
Hello @dbronco
Yes, this segmentation is good but you can optimization your network again.
The current setup requires routing through the core switch, which could introduce unnecessary latency and congestion. - and it is not will good if you have another devices ( end users+printers+ etc.)
A more optimized approach would be to group cameras with their recorder in the same VLAN (per building or vlan). This way, most camera-to-recorder traffic remains at Layer 2, reducing inter-VLAN routing overhead.
P.S- it is from my real practic ))
Thanks!
02-12-2025 02:03 PM
"First question is in regards to VLANs. The previous network engineer segmented and organized the network by device type, i.e.
- VLAN 24 = recording servers and workstations (viewing clients) (5 recording servers, 1 per building. 75 viewing clients)
- VLANs 64, 65, 66, 67, 68 = cameras across the campus (approx 800 total cameras)
- Another VLAN for switch management and there are a few more for ancillary devices.
Is this the most optimized configuration?"
It is. 800 devices in 5 vlans means 160 device per vlan. If you have a /24 network in each vlan, you still have room for mode devices and the broadcast domains is not big.
"Should the cameras be on the same VLAN as their recorder so that the only routing that is happening is when a camera is viewed compared to having all cameras routing through the primary switch before going to its designated recorder? "
800 device in one vlan is possible but this would represent a big broadcast domain. It could cause performance issue.
"The next question, keeping the same site in mind (800 cameras, 5 recording servers, 75 windows PCs and 150 switches) is for NTP. Is there a maximum number of devices that a Catalyst 9300 will provide NTP updates to? I found this from Cisco https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus1000/sw/4_0/system_management/configuration/guide/n1000v_sys_manage/system_6ntp.pdf but that doesn't seem to be used for the same use case. I currently have the following configured on my core switch (Catalyst 9300-48T with 17.06) "
The link is for NX-OS. But, I dont see any reason why the switch would not be able to handle NTP to, at least, the amount of device if supports. Meaning, in a 48 port switch, it must support at least 48 NTP clients. NTP should be a very low traffic.
02-12-2025 04:50 PM - edited 02-12-2025 05:48 PM
@dbronco wrote:
The previous network engineer segmented and organized the network by device type
It is not only important to segment them but also block them in the firewall level. Various CCTV cameras models are well known to have bad codes (copy-n-pasted from other manufacturer) so they have been exploited and have joined numerous Mirai bot net (Over 25,000 IoT CCTV Cameras Used In DDoS Attack). Lock the CCTV network down in the firewall so they will not be the cause of trouble. Lock the CCTV subnet down that nothing else, other than the CCTV & NVR only, can access this subnet. Lock the CCTV subnet that they do not have access to the internet (and they do not need to).
Next, because a lot of the CCTV design have been shamelessly copy-n-pasted off another manufacturer's design, most of them share the same design fault. You know you've potentially bought a "lemon" when the manufacturer does not have any updated drivers for the CCTV or NVR (remember, copy-n-pasted design).
For NTP, have a look at DHCP Option 150. But this goes both ways -- If the CCTV supports DHCP Option 150, then fine. If not, you've bought a lemon.
We have over hundred of thousands of CCTV in our network. Some of our current CCTV would not even support DHCP! And this model is one of the big named brands!
02-13-2025 03:23 AM
That's interesting to know, thanks for sharing. We have roughly 3,000 total devices so we're a much smaller scale compared to yours and have stuck with static addressing. Thanks for reiterating the firewall as well, definitely something to continue to keep in mind
02-12-2025 09:03 PM
Hello @dbronco
Yes, this segmentation is good but you can optimization your network again.
The current setup requires routing through the core switch, which could introduce unnecessary latency and congestion. - and it is not will good if you have another devices ( end users+printers+ etc.)
A more optimized approach would be to group cameras with their recorder in the same VLAN (per building or vlan). This way, most camera-to-recorder traffic remains at Layer 2, reducing inter-VLAN routing overhead.
P.S- it is from my real practic ))
Thanks!
02-13-2025 03:26 AM
This is exactly what I was trying to find out, thank you. Is there a command to run to figure out how much CPU load is being used towards this inter-VLAN routing? Could you share any more about how you're implementing this? Is this a CCTV industry standard or more of a best practice with data you've seen?
02-13-2025 07:54 AM
You can use this commands: from / https://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst3750/software/troubleshooting/cpu_util.html
show processes cpu sorted
show controllers cpu-interface
In my case it is IP-cameras's /
Thanks !
02-13-2025 08:18 AM
"Is this the most optimized configuration?"
Depends what you believe is optimal which depends much on the needs of the networking application to be supported and the network hardware to be used.
The camera hosts will only send a unicast video stream to a recording host? If so, very likely using even single VLAN for just those hosts, as a /22 (or even larger) would be fine too.
Conversely, on a L3 switch, even if each camera host was routed as a /32 network, wouldn't make much difference to the switch.
In other words, forwarding, in a L2/L3 switched environment, for camera hosts, isn't much, if any, impacted by number of hosts, number of VLANs, subnet sizes, or L2 vs. L3 forwarding.
Does this mean there's no concerns? It does not!
You didn't mention bandwidth consumption of each camera host, both now, and possible increases in the future. There can be a very huge bandwidth difference in low def black and white vs. high def color.
Then there's issues, like raised by @Leo Laohoo , i.e. the camera host "quality".
Although you've focused on the camera host aspect, there's also all the similar considerations for all your other network hosts.
So, is your network design optimal, cannot say; insufficient information.
As to your questions about having your network devices support host NTP queries. First, realize supporting NTP, at all, takes processing resources that might be used for more specific networking purposes, so, ideally, you don't want to use such without reviewing need and alternatives.
For example, what if you configured a time service server? That off loads some complex processing from network devices. Yes, a network switch may still need to forward time information between hosts, but switches have dedicated hardware for that. They normally wouldn't have dedicated hardware to service NTP queries.
As an alternative approach, possibly you take advantage of the NTP architecture, i.e. you distribute the NTP servicing.
Basically, you're asking, how many direct NTP hosts can I direct to my main switch. My counter question, why not consider an alternative design, which dramatically reduces discovering what straw number breaks the camel's back.
BTW, I doubt what you have have is "bad" or courting immediate failure, but to fully optimize a network often requires deeper considerations beyond just using /24 subnets.
I'm also not suggesting an "optimal" network is a requirement. For example, again, if you want to use /24s, and if there's no real downside, go ahead and use them.
Anyway, IMO, the real "best practice" is considering your requirements and trade-offs you may need to make.
Remember an industry "best practice" is like buying a suit off the rack, which may be a perfect fit or far from it.
02-13-2025 05:44 PM
@Joseph W. Dohert wrote:
Then there's issues, like raised by @Leo Laohoo , i.e. the camera host "quality".
Even though we have more than a hundred thousand CCTV (and over a hundred NVR) and we are not responsible for them, the vendors call us to help them troubleshoot these infernal trash heaps.
Straight out of the box, these cameras are not guaranteed to work. Not even 50% of the time! They may show up as "Ieee" (drawing power) and that's it. If the cameras do not show a MAC address nor any "outbound" traffic, consider the camera as "boot into ROMMON".
Helpful Hint: Take a known, working VoIP handset. It is a lot faster to replace the suspected faulty camera with a VoIP and watch the phone boot up and join CUCM. If the phone boots up and joins CUCM, then it is definitely NOT a "network issue"!
CCTV drivers are not "for show". Update them or get pwn-ed.
One final thing: Just about anyone can get a result if I search for the model number of the CCTV plus the word "password".
02-14-2025 09:53 AM
@Joseph W. Doherty Thank you for the continued discussion and bringing these points up. In an effort to continue learning, I have a few things to add and some questions.
"You didn't mention bandwidth consumption of each camera host, both now, and possible increases in the future. There can be a very huge bandwidth difference in low def black and white vs. high def color."
We average about 3MB per camera. And they are unicast to the recorder to your earlier point. Does that change your comment about that number of cameras not impacting the switch? I'm still trying to understand packets per second compared to bandwidth on a switch so I apologize if this is a novice question.
"My counter question, why not consider an alternative design, which dramatically reduces discovering what straw number breaks the camel's back."
Using the Cisco switch is actually the alternative design. We've used dedicated time servers and services such as NetTime and haven't had the best luck with it. One of the Pros, or so we thought, was by eliminating additional hardware i.e. additional time servers, and pointing all the devices to the network devices where NTP traffic was already passing, that it would actually be more efficient. But you bring up some good points now and I'll need to continue to monitor this and potentially redesign what we're working on. Is there a way to see how NTP is impacting the switch?
"For example, again, if you want to use /24s, and if there's no real downside, go ahead and use them."
Would the alternative be to segment further and utilize smaller subnets? What are the downsides of using /24s in a network like this?
"Remember an industry "best practice" is like buying a suit off the rack, which may be a perfect fit or far from it."
What a perfect analogy! Thank you
02-14-2025 01:46 PM
@dbronco wrote:
We average about 3MB per camera. And they are unicast to the recorder to your earlier point. Does that change your comment about that number of cameras not impacting the switch? I'm still trying to understand packets per second compared to bandwidth on a switch so I apologize if this is a novice question.
Firstly, no need to apologize for asking a question, novice or not, and we all were novices at some point. (The good thing about identifying yourself as a novice, it helps to guide how technical, and/or expansive, a reply should be.)
No, my comment doesn't change about switch performance, because every (?) current gen Cisco Enterprise class switch tends to support wire-rate on all their ports concurrently.
As to trying to understand PPS's interrelationship with bandwidth, frames are always transmitted at wire-rate, how many you need to transmit, per second, to completely use all the line's bandwidth, depends on the frame size.
For gig Ethernet, the 100% utilization, minimum size packets would be 1,488,096, PPS, (and for [standard] maximum packets would be 81,274 PPS). (BTW, you can scale the PPS values based on bandwidth.)
A Cisco wire-rate capable 48 gig port switch should have a total PPS rate, minimally, of 71,428,608 PPS, and a fabric bandwidth, minimally, of 96 Gbps. (BTW, there other important attributes of switches, for their performance, besides PPS and fabric bandwidth.)
@dbronco wrote:
"For example, again, if you want to use /24s, and if there's no real downside, go ahead and use them."
Would the alternative be to segment further and utilize smaller subnets? What are the downsides of using /24s in a network like this?
Smaller subnets, for the cameras? Insufficient information to say. Personally, I don't like to span VLAN between switches that also crosses a L3 capable device.
Downsides of using /24s? If compared to going larger? Just more subnets to manage. Easier to run out of address space in one subnet while space still available in another subnet.
Since you liked my off the rack suit analogy, think of a /24 as a 38R suit. Noting inherently bad with them, and often close enough to be a "good fit" that there's no need to go further.
Back to NTP. . .
Again, big difference to a switch forwarding a transit NTP packet vs. processing a NTP packet.
As to what to watch for, check CPU processing stats. Ideally, on a switch, you don't want to see "high" software process consumption, in aggregate or for individual processes.
02-19-2025 03:24 AM
Thanks for all the discussion. This has been helpful
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide