IP Cameras and 2951 Router Issue

[tensi0n519]

Hello all,

I am having the following issue with my home lab:

Equipment: Cisco Catalyst Switch 2960-S POE 750W, Cisco 2951 Router running IOS 15.1

I am having an issue with a few IP (POE) cameras causing my switch CPU to run high which causes connectivity issues and my IP cameras to drop frames. I have the camera set up on their own VLAN. It appears that no matter what IP camera I plug into the switch, it starts sending several packets to the router which bogs things down. Prior to using the 2951 router, I was using a pfsense box with the 2960 switch. I had the same vlan setup and the cameras were all working without issues and I was getting no high cpu usage on the switch. I can't see to figure out what the problem is.

The cameras are on vlan 70 and plugged into switchports 1/0/1,3,5, etc. The router is plugged into the switchport 1/0/42. I have a number of devices reserved by client-id or hardware-address, including the cameras. Here are the switch and router configs. Some items like my IP and password stuff I removed from the configs. Hopefully this all makes sense. Thank you all for the help!

Router config:

MainRouter#sh run
Building configuration...

Current configuration : 11735 bytes
!
! No configuration change since last restart
version 15.1
service timestamps debug datetime msec
service timestamps log datetime msec
no service password-encryption
!
hostname MainRouter
!
boot-start-marker
boot-end-marker
!
!
! card type command needed for slot/vwic-slot 0/2
enable secret
!
no aaa new-model
!
clock timezone PST -8 0
clock summer-time PDT recurring
!
crypto pki token default removal timeout 0
!
!
no ipv6 cef
ip source-route
ip cef
!
!
!
ip dhcp excluded-address 10.10.1.1 10.10.1.100
ip dhcp excluded-address 10.10.10.1 10.10.10.100
ip dhcp excluded-address 10.10.10.200 10.10.10.254
ip dhcp excluded-address 10.10.20.1 10.10.20.100
ip dhcp excluded-address 10.10.20.200 10.10.20.254
ip dhcp excluded-address 10.10.30.1 10.10.30.100
ip dhcp excluded-address 10.10.30.200 10.10.30.254
ip dhcp excluded-address 10.10.40.1 10.10.40.100
ip dhcp excluded-address 10.10.40.200 10.10.40.254
ip dhcp excluded-address 10.10.50.1 10.10.50.100
ip dhcp excluded-address 10.10.50.200 10.10.50.254
ip dhcp excluded-address 10.10.60.1 10.10.60.100
ip dhcp excluded-address 10.10.60.200 10.10.60.254
ip dhcp excluded-address 10.10.70.1 10.10.70.100
ip dhcp excluded-address 10.10.70.200 10.10.70.254
!
ip dhcp pool MainLAN
import all
network 10.10.1.0 255.255.255.0
default-router 10.10.1.1
dns-server 209.18.47.61 209.18.47.62
domain-name MainLAN
!
ip dhcp pool Management
network 10.10.10.0 255.255.255.0
default-router 10.10.10.1
dns-server 209.18.47.61 209.18.47.62
domain-name Management
!
ip dhcp pool Servers
network 10.10.20.0 255.255.255.0
default-router 10.10.20.1
dns-server 209.18.47.61 209.18.47.62
domain-name Servers
!
ip dhcp pool SecureData
network 10.10.30.0 255.255.255.0
default-router 10.10.30.1
dns-server 209.18.47.61 209.18.47.62
domain-name SecureData
!
ip dhcp pool GuestData
network 10.10.40.0 255.255.255.0
default-router 10.10.40.1
dns-server 209.18.47.61 209.18.47.62
domain-name GuestData
!
ip dhcp pool GuestWifi
network 10.10.60.0 255.255.255.0
default-router 10.10.60.1
dns-server 209.18.47.61 209.18.47.62
domain-name GuestWiFi
!
ip dhcp pool CCTV
network 10.10.70.0 255.255.255.0
default-router 10.10.70.1
dns-server 209.18.47.61 209.18.47.62
domain-name CCTV
!
ip dhcp pool MainSwitch
host 10.10.10.2 255.255.255.0
client-identifier 0164.d814.071a.80
!
ip dhcp pool ServerSwitch
host 10.10.10.3 255.255.255.0
client-identifier 0134.bdc8.098d.80
!
ip dhcp pool FamilyRoomSwitch
host 10.10.10.4 255.255.255.0
hardware-address 6cb0.ce00.a246
!
ip dhcp pool MasterBedSwitch
host 10.10.10.5 255.255.255.0
client-identifier 0178.d294.aace.d1
!
ip dhcp pool GarageSwitch
host 10.10.10.6 255.255.255.0
client-identifier 01a0.0460.2129.36
!
ip dhcp pool ShedSwitch
host 10.10.10.7 255.255.255.0
hardware-address 841b.5e69.551d
!
ip dhcp pool HallwayAP
host 10.10.10.10 255.255.255.0
client-identifier 0180.2aa8.8688.49
!
ip dhcp pool GarageAP
host 10.10.10.11 255.255.255.0
client-identifier 01fc.ecda.f353.31
!
ip dhcp pool BackyardAP
host 10.10.10.12 255.255.255.0
client-identifier 0180.2aa8.8688.e5
!
ip dhcp pool NVR
host 10.10.70.2 255.255.255.0
client-identifier 0128.57be.94af.2f
!
ip dhcp pool FrontGateCamera
host 10.10.70.3 255.255.255.0
client-identifier 0100.18ae.6374.e4
!
ip dhcp pool CourtyardCamera
host 10.10.70.4 255.255.255.0
client-identifier 0100.18ae.6374.f2
!
ip dhcp pool DrivewayCamera
host 10.10.70.5 255.255.255.0
client-identifier 0100.18ae.6374.d0
!
ip dhcp pool PatioCamera
host 10.10.70.6 255.255.255.0
client-identifier 0100.18ae.529b.7a
!
ip dhcp pool WestSideCamera
host 10.10.70.7 255.255.255.0
client-identifier 0100.18ae.6375.13
!
ip dhcp pool BackyardCamera
host 10.10.70.8 255.255.255.0
client-identifier 0100.18ae.6374.f7
!
ip dhcp pool EastSideCamera
host 10.10.70.9 255.255.255.0
client-identifier 0100.18ae.6375.02
!
ip dhcp pool SouthPTZCamera
host 10.10.70.10 255.255.255.0
client-identifier 01f0.0000.2081.f4
!
ip dhcp pool ChickenCamera
host 10.10.70.11 255.255.255.0
hardware-address e061.b251.3f9e
!
ip dhcp pool GarageCamera
host 10.10.70.12 255.255.255.0
client-identifier 0100.13e2.fa63.0f
!
ip dhcp pool ShedCamera
host 10.10.70.13 255.255.255.0
client-identifier 0100.18ae.6374.da
!
ip dhcp pool WeatherCamera
host 10.10.70.14 255.255.255.0
client-identifier 013c.8cf8.a104.ba
!
!
!
multilink bundle-name authenticated
!
!
!
!
!
voice-card 0
!
!
!
!
!
!
!
license udi pid CISCO2951/K9 sn FTX1819AJKF
hw-module pvdm 0/0
!
hw-module pvdm 0/1
!
hw-module sm 2
!
!
!
!
redundancy
!
!
!
!
!
!
!
!
!
!
!
interface Embedded-Service-Engine0/0
no ip address
shutdown
!
interface GigabitEthernet0/0
ip address dhcp
ip nat outside
ip virtual-reassembly in
duplex auto
speed auto
!
interface GigabitEthernet0/1
ip address 10.10.1.1 255.255.255.0
ip nat inside
ip virtual-reassembly in
duplex auto
speed auto
!
interface GigabitEthernet0/1.10
description Management VLAN
encapsulation dot1Q 10
ip address 10.10.10.1 255.255.255.0
ip nat inside
ip virtual-reassembly in
!
interface GigabitEthernet0/1.20
description Servers VLAN
encapsulation dot1Q 20
ip address 10.10.20.1 255.255.255.0
ip nat inside
ip virtual-reassembly in
!
interface GigabitEthernet0/1.30
description SecureData VLAN
encapsulation dot1Q 30
ip address 10.10.30.1 255.255.255.0
ip nat inside
ip virtual-reassembly in
!
interface GigabitEthernet0/1.40
description GuestData VLAN
encapsulation dot1Q 40
ip address 10.10.40.1 255.255.255.0
ip nat inside
ip virtual-reassembly in
!
interface GigabitEthernet0/1.60
description GuestWifi VLAN
encapsulation dot1Q 60
ip address 10.10.60.1 255.255.255.0
ip nat inside
ip virtual-reassembly in
!
interface GigabitEthernet0/1.70
description CCTV VLAN
encapsulation dot1Q 70
ip address 10.10.70.1 255.255.255.0
ip nat inside
ip virtual-reassembly in
!
interface GigabitEthernet0/2
no ip address
shutdown
duplex auto
speed auto
!
ip forward-protocol nd
!
no ip http server
no ip http secure-server
!
ip nat inside source list 1 interface GigabitEthernet0/0 overload
ip nat inside source list 20 interface GigabitEthernet0/0 overload
ip nat inside source list 30 interface GigabitEthernet0/0 overload
ip nat inside source list 40 interface GigabitEthernet0/0 overload
ip nat inside source list 60 interface GigabitEthernet0/0 overload
ip nat inside source list 70 interface GigabitEthernet0/0 overload
ip route 0.0.0.0 0.0.0.0 ***IP ADDRESS REMOVED** 254
!
access-list 1 permit 10.10.1.0 0.0.0.255
access-list 20 permit 10.10.20.0 0.0.0.255
access-list 30 permit 10.10.30.0 0.0.0.255
access-list 40 permit 10.10.40.0 0.0.0.255
access-list 60 permit 10.10.60.0 0.0.0.255
access-list 70 permit 10.10.70.0 0.0.0.255
!
!
!
!
nls resp-timeout 1
cpd cr-id 1
!
!
control-plane
!
!
!
!
mgcp profile default
!
!
!
!
!
gatekeeper
shutdown
!
!
!
line con 0
exec-timeout 0 0
line aux 0
line 2
no activation-character
no exec
transport preferred none
transport input all
transport output pad telnet rlogin lapb-ta mop udptn v120 ssh
stopbits 1
line vty 0 4
password
login
transport input all
!
scheduler allocate 20000 1000
ntp master 1
end

MainRouter#

Switch Config:

MainSwitch#sh run
Building configuration...

Current configuration : 8655 bytes
!
! Last configuration change at 07:27:21 PST Sat Jan 4 2020 by ryan
! NVRAM config last updated at 07:28:24 PST Sat Jan 4 2020 by ryan
!
version 15.0
no service pad
service timestamps debug datetime msec
service timestamps log datetime msec
service password-encryption
!
hostname MainSwitch
!
boot-start-marker
boot-end-marker
!
enable password
!
username
no aaa new-model
clock timezone PST -8 0
clock summer-time PDT recurring
switch 1 provision ws-c2960s-48fps-l
!
!
no ip domain-lookup
ip domain-name mainswitch.skynet
vtp mode transparent
!
!
!
!
!
spanning-tree mode rapid-pvst
spanning-tree loopguard default
spanning-tree portfast bpduguard default
spanning-tree extend system-id
!
!
!
!
!
!
!
!
!
vlan internal allocation policy ascending
!
vlan 10
name Management
!
vlan 20
name Servers
!
vlan 30
name SecureData
!
vlan 40
name GuestData
!
vlan 50
name SecureWiFi
!
vlan 60
name GuestWifi
!
vlan 70
name CCTV
!
vlan 71
!
vlan 150
name VOICE
!
vlan 900
name TestVLAN
!
ip ssh version 2
lldp run
!
!
!
!
!
!
!
!
!
!
interface Port-channel1
switchport trunk native vlan 10
switchport trunk allowed vlan 10,20,30,50,60,150
switchport mode trunk
!
interface FastEthernet0
no ip address
!
interface GigabitEthernet1/0/1
switchport access vlan 70
switchport mode access
spanning-tree portfast
spanning-tree bpduguard enable
!
interface GigabitEthernet1/0/2
description HallwayAP
switchport trunk native vlan 10
switchport trunk allowed vlan 10,20,30,50,60,150
switchport mode trunk
!
interface GigabitEthernet1/0/3
switchport access vlan 70
switchport mode access
!
interface GigabitEthernet1/0/4
switchport access vlan 30
switchport mode access
!
interface GigabitEthernet1/0/5
switchport access vlan 70
switchport mode access
!
interface GigabitEthernet1/0/6
switchport access vlan 30
switchport mode access
switchport voice vlan 150
!
interface GigabitEthernet1/0/7
switchport access vlan 70
switchport mode access
!
interface GigabitEthernet1/0/8
switchport access vlan 30
switchport mode access
!
interface GigabitEthernet1/0/9
switchport access vlan 70
switchport mode access
!
interface GigabitEthernet1/0/10
switchport access vlan 30
switchport mode access
switchport voice vlan 150
!
interface GigabitEthernet1/0/11
switchport access vlan 70
switchport mode access
!
interface GigabitEthernet1/0/12
switchport access vlan 30
switchport mode access
!
interface GigabitEthernet1/0/13
switchport access vlan 70
switchport mode access
!
interface GigabitEthernet1/0/14
switchport trunk native vlan 10
switchport trunk allowed vlan 10,20,30,50,60,150
switchport mode trunk
!
interface GigabitEthernet1/0/15
switchport access vlan 70
switchport mode access
!
interface GigabitEthernet1/0/16
description Family Room Switch
switchport trunk native vlan 10
switchport trunk allowed vlan 10,30,40
switchport mode trunk
!
interface GigabitEthernet1/0/17
switchport access vlan 70
switchport mode access
!
interface GigabitEthernet1/0/18
switchport access vlan 40
switchport mode access
!
interface GigabitEthernet1/0/19
switchport access vlan 70
switchport mode access
!
interface GigabitEthernet1/0/20
description Master Bedroom Switch
switchport trunk native vlan 10
switchport trunk allowed vlan 10,30
switchport mode trunk
!
interface GigabitEthernet1/0/21
switchport access vlan 70
switchport mode access
!
interface GigabitEthernet1/0/22
switchport trunk native vlan 10
switchport trunk allowed vlan 10,20,30,50,60,150
switchport mode trunk
!
interface GigabitEthernet1/0/23
switchport access vlan 70
switchport mode access
!
interface GigabitEthernet1/0/24
switchport access vlan 30
switchport mode access
!
interface GigabitEthernet1/0/25
switchport access vlan 70
switchport mode access
!
interface GigabitEthernet1/0/26
switchport access vlan 30
switchport mode access
!
interface GigabitEthernet1/0/27
!
interface GigabitEthernet1/0/28
switchport access vlan 40
switchport mode access
!
interface GigabitEthernet1/0/29
switchport access vlan 30
switchport mode access
switchport voice vlan 150
!
interface GigabitEthernet1/0/30
switchport access vlan 40
switchport mode access
!
interface GigabitEthernet1/0/31
switchport mode access
switchport voice vlan 150
!
interface GigabitEthernet1/0/32
switchport access vlan 30
switchport mode access
!
interface GigabitEthernet1/0/33
switchport trunk native vlan 10
switchport trunk allowed vlan 1-99,150
switchport mode trunk
!
interface GigabitEthernet1/0/34
!
interface GigabitEthernet1/0/35
switchport access vlan 30
switchport mode access
!
interface GigabitEthernet1/0/36
switchport access vlan 900
switchport mode access
spanning-tree portfast
!
interface GigabitEthernet1/0/37
switchport access vlan 71
switchport mode access
!
interface GigabitEthernet1/0/38
switchport access vlan 30
switchport mode access
power inline never
!
interface GigabitEthernet1/0/39
switchport access vlan 71
switchport mode access
!
interface GigabitEthernet1/0/40
switchport access vlan 30
switchport mode access
power inline never
!
interface GigabitEthernet1/0/41
switchport access vlan 71
switchport mode access
!
interface GigabitEthernet1/0/42
description MainRouter
switchport trunk allowed vlan 10,20,30,40,50,60,70
switchport mode trunk
power inline never
!
interface GigabitEthernet1/0/43
switchport access vlan 71
switchport mode access
!
interface GigabitEthernet1/0/44
description BackupRouter
switchport trunk allowed vlan 10,20,30,40,50,60,70,900
switchport mode trunk
!
interface GigabitEthernet1/0/45
switchport trunk native vlan 10
switchport trunk allowed vlan 10,20,30,50,60,150
switchport mode trunk
channel-protocol lacp
channel-group 1 mode active
!
interface GigabitEthernet1/0/46
switchport trunk native vlan 10
switchport trunk allowed vlan 10,20,30,50,60,150
switchport mode trunk
channel-protocol lacp
channel-group 1 mode active
!
interface GigabitEthernet1/0/47
switchport trunk native vlan 10
switchport trunk allowed vlan 10,20,30,50,60,150
switchport mode trunk
channel-protocol lacp
channel-group 1 mode active
!
interface GigabitEthernet1/0/48
switchport trunk native vlan 10
switchport trunk allowed vlan 10,20,30,50,60,150
switchport mode trunk
channel-protocol lacp
channel-group 1 mode active
!
interface GigabitEthernet1/0/49
shutdown
!
interface GigabitEthernet1/0/50
shutdown
!
interface GigabitEthernet1/0/51
shutdown
!
interface GigabitEthernet1/0/52
shutdown
!
interface Vlan1
no ip address
!
interface Vlan10
ip address 10.10.10.2 255.255.255.0
!
ip default-gateway 10.10.10.1
ip http server
ip http secure-server
!
!
!
vstack
!
line con 0
exec-timeout 0 0
logging synchronous
line vty 0 4
exec-timeout 0 0
password
login local
transport input ssh
line vty 5 15
exec-timeout 0 0
password 7
login local
transport input ssh
!
ntp server 10.10.1.1
end

MainSwitch#

[Georg Pauwen]

Hello,

what brand/type/model are the IP cameras ?

[tensi0n519]

I'm not exactly sure. A family member of mine installs commercial security cameras and these are the ones I am using.

https://www.unixcctv.com/product/ip-power-mc532v-5mp-network-ip66-outdoor-ir-eyeball-camera-h-265-3-3-12mm-motorized-vf-lens-up-to-100ft-dc-12v-poe/

I'm afraid I don't have any further info on them. I've had them up and running for about 3 years now. Of course they weren't hooked up to Cisco gear until recently.

Thank you.

[Leo Laohoo]

CPU of the switch checks out fine.
Router CPU looks good too.
What exactly is the problem?

Could be a multi-cast storm?

[tensi0n519]

The more I've been researching this problem, the more I've been thinking the same thing as you. How would I go about fixing that?

[Leo Laohoo]

Fixing what? I don't know what the problem is. Tell us what is going wrong (minus the "high CPU" bit).

[gaston.benitez]

hi.

Can you try to configure portfast on all cammera ports?

also. please. can you share the following output from the switch and the router ?

//from the switch

show spanning-tree detail | inc ieee|occurr|from|is exec (take this comand 2 or 3 times every 3 minutes)

show process cpu | e 0.00

//from the router

show process cpu | e 0.00

BR

[tensi0n519]

Thank you very much for your response. Here is the information you requested.

"Can you try to configure portfast on all cammera ports?"

I initially had all of the camera ports with portfast configured, exactly like switchport 1. The results are the same and the switch and router have even higher cpu. I have a couple different brand cameras and all have the same results of different ports.

"also. please. can you share the following output from the switch and the router ?"

SWITCH:

MainSwitch#show spanning-tree detail | inc ieee|occurr|from|is exec
VLAN0010 is executing the rstp compatible Spanning Tree protocol
Number of topology changes 11 last change occurred 00:04:31 ago
from GigabitEthernet1/0/42
VLAN0020 is executing the rstp compatible Spanning Tree protocol
Number of topology changes 8 last change occurred 00:04:32 ago
from GigabitEthernet1/0/42
VLAN0030 is executing the rstp compatible Spanning Tree protocol
Number of topology changes 26 last change occurred 00:04:54 ago
from GigabitEthernet1/0/42
VLAN0040 is executing the rstp compatible Spanning Tree protocol
Number of topology changes 4 last change occurred 00:04:32 ago
from GigabitEthernet1/0/42
VLAN0050 is executing the rstp compatible Spanning Tree protocol
Number of topology changes 8 last change occurred 00:04:32 ago
from GigabitEthernet1/0/42
VLAN0060 is executing the rstp compatible Spanning Tree protocol
Number of topology changes 8 last change occurred 00:04:32 ago
from GigabitEthernet1/0/42
VLAN0070 is executing the rstp compatible Spanning Tree protocol
Number of topology changes 2 last change occurred 00:04:32 ago
from GigabitEthernet1/0/42
VLAN0071 is executing the rstp compatible Spanning Tree protocol
Number of topology changes 4 last change occurred 05:23:55 ago
from GigabitEthernet1/0/43
VLAN0150 is executing the rstp compatible Spanning Tree protocol
Number of topology changes 7 last change occurred 05:18:43 ago
from GigabitEthernet1/0/14
MainSwitch#show spanning-tree detail | inc ieee|occurr|from|is exec
VLAN0010 is executing the rstp compatible Spanning Tree protocol
Number of topology changes 11 last change occurred 00:05:37 ago
from GigabitEthernet1/0/42
VLAN0020 is executing the rstp compatible Spanning Tree protocol
Number of topology changes 8 last change occurred 00:05:37 ago
from GigabitEthernet1/0/42
VLAN0030 is executing the rstp compatible Spanning Tree protocol
Number of topology changes 26 last change occurred 00:05:59 ago
from GigabitEthernet1/0/42
VLAN0040 is executing the rstp compatible Spanning Tree protocol
Number of topology changes 4 last change occurred 00:05:37 ago
from GigabitEthernet1/0/42
VLAN0050 is executing the rstp compatible Spanning Tree protocol
Number of topology changes 8 last change occurred 00:05:37 ago
from GigabitEthernet1/0/42
VLAN0060 is executing the rstp compatible Spanning Tree protocol
Number of topology changes 8 last change occurred 00:05:37 ago
from GigabitEthernet1/0/42
VLAN0070 is executing the rstp compatible Spanning Tree protocol
Number of topology changes 2 last change occurred 00:05:37 ago
from GigabitEthernet1/0/42
VLAN0071 is executing the rstp compatible Spanning Tree protocol
Number of topology changes 4 last change occurred 05:25:00 ago
from GigabitEthernet1/0/43
VLAN0150 is executing the rstp compatible Spanning Tree protocol
Number of topology changes 7 last change occurred 05:19:37 ago
from GigabitEthernet1/0/14
MainSwitch#show spanning-tree detail | inc ieee|occurr|from|is exec
VLAN0010 is executing the rstp compatible Spanning Tree protocol
Number of topology changes 11 last change occurred 00:07:52 ago
from GigabitEthernet1/0/42
VLAN0020 is executing the rstp compatible Spanning Tree protocol
Number of topology changes 8 last change occurred 00:07:52 ago
from GigabitEthernet1/0/42
VLAN0030 is executing the rstp compatible Spanning Tree protocol
Number of topology changes 26 last change occurred 00:08:14 ago
from GigabitEthernet1/0/42
VLAN0040 is executing the rstp compatible Spanning Tree protocol
Number of topology changes 4 last change occurred 00:07:53 ago
from GigabitEthernet1/0/42
VLAN0050 is executing the rstp compatible Spanning Tree protocol
Number of topology changes 8 last change occurred 00:07:53 ago
from GigabitEthernet1/0/42
VLAN0060 is executing the rstp compatible Spanning Tree protocol
Number of topology changes 8 last change occurred 00:07:53 ago
from GigabitEthernet1/0/42
VLAN0070 is executing the rstp compatible Spanning Tree protocol
Number of topology changes 2 last change occurred 00:07:53 ago
from GigabitEthernet1/0/42
VLAN0071 is executing the rstp compatible Spanning Tree protocol
Number of topology changes 4 last change occurred 05:27:16 ago
from GigabitEthernet1/0/43
VLAN0150 is executing the rstp compatible Spanning Tree protocol
Number of topology changes 7 last change occurred 05:21:57 ago
from GigabitEthernet1/0/14

MainSwitch#show process cpu | e 0.00
CPU utilization for five seconds: 55%/14%; one minute: 43%; five minutes: 41%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
4 567266 32328 17547 8.20% 1.26% 0.95% 0 Check heaps
10 110668 957 115640 2.10% 0.26% 0.18% 0 Licensing Auto U
86 28843 2541360 11 0.19% 0.08% 0.06% 0 RedEarth Rx Mana
104 34331 1429174 24 0.09% 0.12% 0.10% 0 HLFM address lea
122 29106 524312 55 0.09% 0.07% 0.08% 0 hpm main process
126 213345 57418 3715 0.39% 0.41% 0.39% 0 hpm counter proc
157 15347231 1431417 10721 23.51% 24.04% 24.32% 0 Hulc LED Process
168 124026 11466 10816 0.19% 0.20% 0.19% 0 HQM Stack Proces
185 273432 229055 1193 0.39% 0.32% 0.38% 0 Auth Manager
214 114551 407635 281 0.19% 0.11% 0.12% 0 Spanning Tree
285 14815 126437 117 0.19% 0.06% 0.03% 0 Marvell wk-a Pow

MainSwitch#show process cpu | e 0.00
CPU utilization for five seconds: 46%/13%; one minute: 44%; five minutes: 41%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
126 213397 57428 3715 0.49% 0.41% 0.39% 0 hpm counter proc
157 15350277 1431669 10721 26.28% 24.19% 24.34% 0 Hulc LED Process
168 124052 11468 10817 0.19% 0.20% 0.20% 0 HQM Stack Proces
169 34803 22916 1518 0.09% 0.06% 0.06% 0 HRPC qos request
185 273464 229095 1193 0.29% 0.31% 0.37% 0 Auth Manager
214 114574 407673 281 0.19% 0.13% 0.13% 0 Spanning Tree
285 14821 126459 117 0.09% 0.06% 0.03% 0 Marvell wk-a Pow
338 3827 622 6152 0.29% 0.49% 0.38% 5 SSH Process
MainSwitch#show process cpu | e 0.00
CPU utilization for five seconds: 49%/13%; one minute: 42%; five minutes: 41%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
4 574579 32737 17551 3.60% 1.40% 1.01% 0 Check heaps
55 23290 982 23716 0.59% 0.06% 0.01% 0 Per-minute Jobs
85 22188 537195 41 0.09% 0.03% 0.01% 0 RedEarth Tx Mana
86 29468 2570169 11 0.19% 0.11% 0.07% 0 RedEarth Rx Mana
104 35371 1445741 24 0.29% 0.12% 0.11% 0 HLFM address lea
106 17459 1446823 12 0.29% 0.08% 0.05% 0 HLFM address ret
122 29831 531283 56 0.09% 0.12% 0.09% 0 hpm main process
126 216304 58085 3723 0.50% 0.31% 0.37% 0 hpm counter proc
157 15534913 1448009 10728 24.11% 24.35% 24.34% 0 Hulc LED Process
168 125564 11599 10825 0.19% 0.19% 0.19% 0 HQM Stack Proces
169 35293 23178 1522 0.09% 0.07% 0.06% 0 HRPC qos request
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
185 276356 231798 1192 0.49% 0.36% 0.37% 0 Auth Manager
200 17309 1446125 11 0.09% 0.06% 0.02% 0 IP ARP Retry Age
214 115195 411146 280 0.19% 0.07% 0.07% 0 Spanning Tree

ROUTER:

MainRouter#show process cpu | e 0.00
CPU utilization for five seconds: 12%/5%; one minute: 6%; five minutes: 7%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
31 480484 9790633 49 5.67% 2.71% 3.18% 0 ARP Input
68 140 18134 7 0.15% 0.14% 0.15% 0 Per-Second Jobs
86 248 72494 3 0.15% 0.19% 0.18% 0 Netclock Backgro
107 1988 36240 54 0.07% 0.04% 0.05% 0 BPSM stat Proces
111 12 2294562 0 0.07% 0.11% 0.10% 0 Ethernet Msec Ti
148 84 159 528 0.39% 0.12% 0.03% 644 Virtual Exec

MainRouter#show process cpu | e 0.00
CPU utilization for five seconds: 11%/5%; one minute: 7%; five minutes: 7%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
31 480932 9800288 49 5.75% 3.16% 3.26% 0 ARP Input
68 140 18142 7 0.15% 0.13% 0.15% 0 Per-Second Jobs
86 248 72526 3 0.15% 0.19% 0.18% 0 Netclock Backgro
107 1992 36256 54 0.07% 0.04% 0.05% 0 BPSM stat Proces
111 12 2295555 0 0.07% 0.11% 0.10% 0 Ethernet Msec Ti

MainRouter#show process cpu | e 0.00
CPU utilization for five seconds: 12%/5%; one minute: 7%; five minutes: 7%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
31 481160 9807673 49 5.75% 3.37% 3.30% 0 ARP Input
68 140 18148 7 0.15% 0.13% 0.15% 0 Per-Second Jobs
86 248 72550 3 0.15% 0.18% 0.17% 0 Netclock Backgro
107 1992 36268 54 0.07% 0.04% 0.05% 0 BPSM stat Proces
111 12 2296313 0 0.15% 0.11% 0.10% 0 Ethernet Msec Ti
148 104 169 615 0.15% 0.12% 0.03% 644 Virtual Exec

MainRouter#show process cpu | e 0.00
CPU utilization for five seconds: 10%/4%; one minute: 8%; five minutes: 7%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
31 481356 9812540 49 5.67% 3.56% 3.34% 0 ARP Input
68 140 18152 7 0.07% 0.13% 0.15% 0 Per-Second Jobs
86 248 72566 3 0.23% 0.19% 0.18% 0 Netclock Backgro
111 12 2296829 0 0.07% 0.11% 0.10% 0 Ethernet Msec Ti
148 112 174 643 0.23% 0.13% 0.04% 644 Virtual Exec

MainRouter#show process cpu | e 0.00
CPU utilization for five seconds: 11%/4%; one minute: 8%; five minutes: 7%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
6 11512 2820 4082 0.39% 0.08% 0.06% 0 Check heaps
31 494912 10131602 48 5.83% 4.27% 3.61% 0 ARP Input
68 144 18568 7 0.15% 0.13% 0.15% 0 Per-Second Jobs
86 256 74225 3 0.23% 0.19% 0.18% 0 Netclock Backgro
107 2088 37105 56 0.07% 0.04% 0.05% 0 BPSM stat Proces
111 12 2349204 0 0.15% 0.12% 0.10% 0 Ethernet Msec Ti
141 7208 85409 84 0.07% 0.02% 0.02% 0 IP Input

[paul driver]

Hello

On ALL access ports enable stp portfast to negate stp transitions of the cameras and end hosts.

interface GigabitEthernet1/0/x
spanning-tree portfast
exit

Also can you confirm what is connected to the port-channel 1
Lastly you could clean up your nat configuration and change the default route on the rtr.

no ip source-route
no ip route 0.0.0.0 0.0.0.0 ***IP ADDRESS REMOVED** 254
ip route 0.0.0.0 0.0.0.0 GigabitEthernet0/0 dhcp

access-list 1 permit 10.10.20.0 0.0.0.255
access-list 1 permit 10.10.30.0 0.0.0.255
access-list 1 permit 10.10.40.0 0.0.0.255
access-list 1 permit 10.10.60.0 0.0.0.255
access-list 1 permit 10.10.70.0 0.0.0.255

no ip nat inside source list 20 interface GigabitEthernet0/0 overload
no ip nat inside source list 30 interface GigabitEthernet0/0 overload
no ip nat inside source list 40 interface GigabitEthernet0/0 overload
no ip nat inside source list 60 interface GigabitEthernet0/0 overload
no ip nat inside source list 70 interface GigabitEthernet0/0 overload
no access-list 20 permit 10.10.20.0 0.0.0.255
no access-list 30 permit 10.10.30.0 0.0.0.255
no access-list 40 permit 10.10.40.0 0.0.0.255
no access-list 60 permit 10.10.60.0 0.0.0.255
no access-list 70 permit 10.10.70.0 0.0.0.255

[tensi0n519]

port-channel 1 is connected to a second 2960-s switch, non poe 24 port. I checked the CPU processes on that and it doesn't appear to be affected by my issue. The switch is for a few of my servers which have 4 nics each.

I will go a head and make the changes to the access ports per your suggestion.

[tensi0n519]

Here's an update on my CPU usage after the changes I made. There appears to be no change. The following is my CPU usage with all 12 IP cameras plugged in. When I was using my pfsense box, I was getting a CPU usage of about 35%/0% on the 2960 switch.

MainSwitch#show process cpu | e 0.00
CPU utilization for five seconds: 72%/26%; one minute: 64%; five minutes: 62%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
4 734295 41669 17622 8.19% 1.28% 1.02% 0 Check heaps
44 18186 91523 198 0.09% 0.06% 0.04% 0 Net Background
80 30469 2422406 12 0.39% 0.19% 0.15% 0 Draught link sta
85 32745 675136 48 0.29% 0.13% 0.11% 0 RedEarth Tx Mana
86 49570 3191865 15 0.19% 0.28% 0.29% 0 RedEarth Rx Mana
104 57777 1815200 31 0.09% 0.25% 0.23% 0 HLFM address lea
122 46013 683077 67 0.59% 0.27% 0.20% 0 hpm main process
126 295725 72957 4053 0.79% 0.75% 0.74% 0 hpm counter proc
149 8126 354107 22 0.09% 0.07% 0.04% 0 Hulc Storm Contr
157 20016279 1818386 11007 29.29% 29.94% 29.63% 0 Hulc LED Process
168 160033 14563 10989 0.19% 0.21% 0.20% 0 HQM Stack Proces
169 45106 29106 1549 0.09% 0.06% 0.06% 0 HRPC qos request
185 346972 291106 1191 0.59% 0.41% 0.39% 0 Auth Manager
188 22391 25112 891 0.09% 0.04% 0.03% 0 CDP Protocol
214 140658 485451 289 0.19% 0.18% 0.16% 0 Spanning Tree
231 14908 728883 20 0.09% 0.13% 0.09% 0 UDLD

MainRouter#show process cpu | e 0.00
CPU utilization for five seconds: 29%/11%; one minute: 25%; five minutes: 23%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
31 1087608 21196507 51 17.35% 14.42% 13.30% 0 ARP Input
68 252 33825 7 0.07% 0.13% 0.15% 0 Per-Second Jobs
86 468 135212 3 0.15% 0.17% 0.17% 0 Netclock Backgro
107 5856 67596 86 0.07% 0.04% 0.05% 0 BPSM stat Proces
111 20 4285594 0 0.15% 0.13% 0.14% 0 Ethernet Msec Ti
141 14000 161489 86 0.07% 0.03% 0.02% 0 IP Input

[Leo Laohoo]

---

@tensi0n519 wrote:

157 20016279 1818386 11007 29.29% 29.94% 29.63% 0 Hulc LED Process

---

The output is not matching.  Post the complete output to the command "sh proc cpu sort | ex 0.00".
What happens if you take the cameras off?  Does Hulc LED Process go down?
And what firmware is the switch running on?  

[tensi0n519]

The switch is running the following: Cisco IOS Software, C2960S Software (C2960S-UNIVERSALK9-M), Version 15.0(2)SE11, RELEASE SOFTWARE (fc3)

I attached two CPU usage txt files. One with cams plugged in, the other without cams plugged in. Hulc LED goes down with no cameras.

[Leo Laohoo]

Post the complete output to the command "sh interface <CAMERA PORTS>". Both of them.
Also post the interface config for the two camera ports.
Is LLDP enabled? (sh lldp)

[tensi0n519]

I believe this is what you request:

Switch Port 1 (Camera 1):

MainSwitch#sh interface gigabitEthernet 1/0/1
GigabitEthernet1/0/1 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is 64d8.1407.1a81 (bia 64d8.1407.1a81)
MTU 1500 bytes, BW 100000 Kbit/sec, DLY 100 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 100Mb/s, media type is 10/100/1000BaseTX
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 372000 bits/sec, 720 packets/sec
5 minute output rate 477000 bits/sec, 922 packets/sec
37607321 packets input, 30284564638 bytes, 0 no buffer
Received 17090621 broadcasts (17427 multicasts)
0 runts, 0 giants, 0 throttles
4727 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 17427 multicast, 241 pause input
0 input packets with dribble condition detected
41547378 packets output, 2686663166 bytes, 0 underruns
0 output errors, 0 collisions, 1 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

MainSwitch#sh run interface gigabitEthernet 1/0/1
Building configuration...

Current configuration : 145 bytes
!
interface GigabitEthernet1/0/1
switchport access vlan 70
switchport mode access
spanning-tree portfast
spanning-tree bpduguard enable
end

Switch Port 3 (Camera 2):

MainSwitch#sh interface gigabitEthernet 1/0/3
GigabitEthernet1/0/3 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is 64d8.1407.1a83 (bia 64d8.1407.1a83)
MTU 1500 bytes, BW 100000 Kbit/sec, DLY 100 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 100Mb/s, media type is 10/100/1000BaseTX
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output 00:00:01, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 165000 bits/sec, 314 packets/sec
5 minute output rate 332000 bits/sec, 642 packets/sec
7365639 packets input, 8009645057 bytes, 0 no buffer
Received 1825949 broadcasts (7742 multicasts)
0 runts, 0 giants, 0 throttles
329 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 7742 multicast, 13 pause input
0 input packets with dribble condition detected
18658983 packets output, 1218049634 bytes, 0 underruns
0 output errors, 0 collisions, 1 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

MainSwitch#sh run interface gigabitEthernet 1/0/3
Building configuration...

Current configuration : 145 bytes
!
interface GigabitEthernet1/0/3
switchport access vlan 70
switchport mode access
spanning-tree portfast
spanning-tree bpduguard enable
end

MainSwitch#sh lldp

Global LLDP Information:
Status: ACTIVE
LLDP advertisements are sent every 30 seconds
LLDP hold time advertised is 120 seconds
LLDP interface reinitialisation delay is 2 seconds