Extremely High CPU utilization 2821 Routers

aarondcounts · ‎11-03-2010

I am experiencing extremely high CPU utilization on my Cisco 2821. I am using this router to handle several (120-125) GRE tunnels. I am also experiencing large amounts of input errors on both of my gigabit ethernet interfaces. All of my GRE tunnels are configured the same and I have added a sample of one of the tunnel configurations at the end of this post. I had been running over 200 tunnels on a Cisco 1760 router. another part of the router configuration that may be of some use if that I am using a route-map to route traffic from specific hosts to destinations beyond the other end of the tunnel. There is a single route-map for that but has as many entries as there are tunnels.

interface Tunnel###

ip address 10.10.10.77 255.255.255.252
no ip redirects
ip mtu 1440
ip tcp adjust-mss 1350
keepalive 20 9
tunnel source Loopback0
tunnel destination 10.1.1.240

Bert Gevers · ‎11-04-2010

Hello Aaron,

In order to have a look, would it be possible to provide us the following:

show process cpu sorted (to show the high cpu utilization)

show interfaces

Bert

aarondcounts · ‎11-04-2010

#sh proc cpu sorted
CPU utilization for five seconds: 80%/79%; one minute: 79%; five minutes: 79%

PID	Runtime(ms)	Invoked	uSecs	5Sec	1Min	5Min	TTY	Process
66	26946744	83245989	323	0.40%	0.39%	0.35%	0	IP Input
34	2694108	6744470	399	0.40%	0.11%	0.10%	0	Net Background
129	1801540	110300	16333	0.24%	0.02%	0.00%	0	Per-minute Jobs
2	354068	1304114	271	0.08%	0.02%	0.00%	0	Load Meter
90	347832	9573957	36	0.08%	0.04%	0.02%	0	CEF process
97	38460	6519629	5	0.08%	0.00%	0.00%	0	RUDPV1 Main Proc
49	48656	26080874	1	0.08%	0.01%	0.00%	0	Netclock Backgro
23	753208	1303952	577	0.08%	0.03%	0.02%	0	HC Counter Timer
8	0	1	0	0.00%	0.00%	0.00%	0	IPC Zone Manager
9	21952	6519630	3	0.00%	0.00%	0.00%	0	IPC Periodic Tim
10	18856	6519630	2	0.00%	0.00%	0.00%	0	IPC Deferred Por
12	0	1	0	0.00%	0.00%	0.00%	0	IPC BackPressure
11	0	1	0	0.00%	0.00%	0.00%	0	IPC Seat Manager
14	0	1	0	0.00%	0.00%	0.00%	0	Crash writer
13	0	1	0	0.00%	0.00%	0.00%	0	OIR Handler
16	99252	364532	272	0.00%	0.00%	0.00%	0	ARP Input
5	4408	5194	848	0.00%	0.00%	0.00%	0	Pool Manager
4	5861376	769770	7614	0.00%	0.04%	0.05%	0	Check heaps
15	47992	217349	220	0.00%	0.00%	0.00%	0	Environmental mo
20	0	1	0	0.00%	0.00%	0.00%	0	Policy Manager
21	0	2	0	0.00%	0.00%	0.00%	0	DDR Timers
3	92	6632	13	0.00%	0.00%	0.00%	0	TACACS+
7	1156	108675	10	0.00%	0.00%	0.00%	0	IPC Dynamic Cach
17	0	2	0	0.00%	0.00%	0.00%	0	ATM Idle Timer
25	0	1	0	0.00%	0.00%	0.00%	0	RO Notify Timers
26	0	2	0	0.00%	0.00%	0.00%	0	SMART
27	10952	6520536	1	0.00%	0.00%	0.00%	0	GraphIt
28	0	2	0	0.00%	0.00%	0.00%	0	Dialer event
29	0	1	0	0.00%	0.00%	0.00%	0	SERIAL A'detect
30	0	2	0	0.00%	0.00%	0.00%	0	XML Proxy Client
6	0	2	0	0.00%	0.00%	0.00%	0	Timers
19	0	37	0	0.00%	0.00%	0.00%	0	AAA_SERVER_DEADT
18	0	2	0	0.00%	0.00%	0.00%	0	AAA high-capacit
35	0	2	0	0.00%	0.00%	0.00%	0	IDB Work
36	60	2653	22	0.00%	0.00%	0.00%	0	Logger
22	4	2	2000	0.00%	0.00%	0.00%	0	Entity MIB API
38	67060	6520538	10	0.00%	0.01%	0.00%	0	Per-Second Jobs
39	1244	434696	2	0.00%	0.00%	0.00%	0	c2800 Periodic
40	0	1	0	0.00%	0.00%	0.00%	0	AggMgr Process
41	0	1	0	0.00%	0.00%	0.00%	0	dev_device_inser
42	0	1	0	0.00%	0.00%	0.00%	0	dev_device_remov
43	0	1	0	0.00%	0.00%	0.00%	0	sal_dpc_process
44	0	1	0	0.00%	0.00%	0.00%	0	ARL Table Manage
1	12	135	88	0.00%	0.00%	0.00%	0	Chunk Manager
46	0	2	0	0.00%	0.00%	0.00%	0	Eswilp Storm Con
47	0	2	0	0.00%	0.00%	0.00%	0	ESWILPPM
24	0	2	0	0.00%	0.00%	0.00%	0	Serial Backgroun
32	0	1	0	0.00%	0.00%	0.00%	0	Inode Table Dest
50	0	2	0	0.00%	0.00%	0.00%	0	SM Monitor
51	84	69	1217	0.00%	0.02%	0.00%	322	Virtual Exec
52	14268	6519629	2	0.00%	0.00%	0.00%	0	Ether-Switch RBC
53	0	1	0	0.00%	0.00%	0.00%	0	IGMP Snooping Pr
54	0	1	0	0.00%	0.00%	0.00%	0	IGMP Snooping Re
55	824	217349	3	0.00%	0.00%	0.00%	0	Call Management
56	0	2	0	0.00%	0.00%	0.00%	0	dot1x
57	0	2	0	0.00%	0.00%	0.00%	0	DTP Protocol
58	14616	6519630	2	0.00%	0.00%	0.00%	0	PI MATM Aging Pr
59	1784	652043	2	0.00%	0.00%	0.00%	0	EtherChnl
60	0	1	0	0.00%	0.00%	0.00%	0	L2X Data Daemon
61	0	2	0	0.00%	0.00%	0.00%	0	AAA Dictionary R
62	280	1840	152	0.00%	0.00%	0.00%	0	AAA Server
33	0	1	0	0.00%	0.00%	0.00%	0	Critical Bkgnd
64	0	1	0	0.00%	0.00%	0.00%	0	ACCT Periodic Pr
37	38040	6519626	5	0.00%	0.00%	0.00%	0	TTY Background
67	0	1	0	0.00%	0.00%	0.00%	0	ICMP event handl
68	0	134	0	0.00%	0.00%	0.00%	0	TurboACL
69	0	2	0	0.00%	0.00%	0.00%	0	TurboACL chunk
70	192	10827	17	0.00%	0.00%	0.00%	0	MOP Protocols
71	0	3	0	0.00%	0.00%	0.00%	0	PPP Hooks
73	0	1	0	0.00%	0.00%	0.00%	0	SSS Manager
45	0	2	0	0.00%	0.00%	0.00%	0	ESWPPM
75	0	1	0	0.00%	0.00%	0.00%	0	SSS Feature Mana
76	53028	25469710	2	0.00%	0.01%	0.00%	0	SSS Feature Time
77	0	1	0	0.00%	0.00%	0.00%	0	VPDN call manage
78	0	1	0	0.00%	0.00%	0.00%	0	L2X Socket proce
79	0	1	0	0.00%	0.00%	0.00%	0	L2X SSS manager
80	0	2	0	0.00%	0.00%	0.00%	0	L2TP mgmt daemon
81	0	1	0	0.00%	0.00%	0.00%	0	X.25 Encaps Mana
82	42488	172793	245	0.00%	0.00%	0.00%	0	IP Background
83	26856	67928	395	0.00%	0.00%	0.00%	0	IP RIB Update
84	0	2	0	0.00%	0.00%	0.00%	0	PPP IP Route
85	0	2	0	0.00%	0.00%	0.00%	0	PPP IPCP
86	11312	6473744	1	0.00%	0.00%	0.00%	0	Socket Timers
87	1212	23811	50	0.00%	0.00%	0.00%	0	TCP Timer
88	1064	892	1192	0.00%	0.00%	0.00%	0	TCP Protocols
89	0	1	0	0.00%	0.00%	0.00%	0	COPS
48	0	2	0	0.00%	0.00%	0.00%	0	Eswilp Storm Con
91	0	2	0	0.00%	0.00%	0.00%	0	L2MM
92	0	1	0	0.00%	0.00%	0.00%	0	MRD
93	0	1	0	0.00%	0.00%	0.00%	0	IGMPSN
94	0	2	0	0.00%	0.00%	0.00%	0	SNMP Timers
95	0	2	0	0.00%	0.00%	0.00%	0	SCTP Main Proces
96	0	1	0	0.00%	0.00%	0.00%	0	IUA Main Process
63	508	3377	150	0.00%	0.00%	0.00%	0	AAA ACCT Proc
98	0	1	0	0.00%	0.00%	0.00%	0	bsm_timers
99	13516	6519631	2	0.00%	0.00%	0.00%	0	bsm_xmt_proc
101	0	2	0	0.00%	0.00%	0.00%	0	Dialer Forwarder
102	13196	108675	121	0.00%	0.00%	0.00%	0	IP Cache Ager
103	159700	108677	1469	0.00%	0.00%	0.00%	0	Adj Manager
104	64	21738	2	0.00%	0.00%	0.00%	0	HTTP CORE
105	0	1	0	0.00%	0.00%	0.00%	0	IP Traceroute
106	0	1	0	0.00%	0.00%	0.00%	0	RARP Input
107	0	1	0	0.00%	0.00%	0.00%	0	PAD InCall
108	0	2	0	0.00%	0.00%	0.00%	0	X.25 Background
109	0	2	0	0.00%	0.00%	0.00%	0	PPP Bind
110	0	2	0	0.00%	0.00%	0.00%	0	PPP SSS
111	684	260824	2	0.00%	0.00%	0.00%	0	CRM_CALL_UPDATE_
112	0	2	0	0.00%	0.00%	0.00%	0	ENABLE AAA
113	0	1	0	0.00%	0.00%	0.00%	0	EM Background Pr
114	0	1	0	0.00%	0.00%	0.00%	0	Key chain liveke
115	0	5	0	0.00%	0.00%	0.00%	0	LINE AAA
116	0	2	0	0.00%	0.00%	0.00%	0	LOCAL AAA
117	19532	24796	787	0.00%	0.01%	0.00%	0	TPLUS
118	0	1	0	0.00%	0.00%	0.00%	0	PM Callback
120	396	928	426	0.00%	0.00%	0.00%	0	AAA SEND STOP EV
121	1820	652043	2	0.00%	0.00%	0.00%	0	RMON Recycle Pro
122	0	2	0	0.00%	0.00%	0.00%	0	RMON Deferred Se
123	0	1	0	0.00%	0.00%	0.00%	0	Syslog Traps
124	8	2	4000	0.00%	0.00%	0.00%	0	VLAN Manager
125	236	1383	170	0.00%	0.00%	0.00%	0	Syslog
126	0	1	0	0.00%	0.00%	0.00%	0	VPDN Scal
127	172	1689	101	0.00%	0.00%	0.00%	0	Net Input
128	1967492	1304137	1508	0.00%	0.06%	0.07%	0	Compute load avg
74	3928	869391	4	0.00%	0.00%	0.00%	0	SSS Test Client
130	14116	226112	62	0.00%	0.00%	0.00%	0	CEF Scanner
131	0	1	0	0.00%	0.00%	0.00%	0	tHUB
132	0	2	0	0.00%	0.00%	0.00%	0	tENM
133	842336	9646111	87	0.00%	0.02%	0.00%	0	HSRP (Standby)
134	44648	6519611	6	0.00%	0.00%	0.00%	0	Track
136	126928	328851	385	0.00%	0.00%	0.00%	0	IP SNMP
137	124836	163465	763	0.00%	0.00%	0.00%	0	PDU DISPATCHER
138	1302536	163475	7967	0.00%	0.00%	0.00%	0	SNMP ENGINE
139	0	1	0	0.00%	0.00%	0.00%	0	SNMP ConfCopyPro
140	672	1327	506	0.00%	0.00%	0.00%	0	SNMP Traps
142	488	23260	20	0.00%	0.00%	0.00%	0	IP-EIGRP Router
143	634156	4788627	132	0.00%	0.01%	0.00%	0	IP-EIGRP: PDM
144	640	24115	26	0.00%	0.00%	0.00%	0	IP-EIGRP Router
145	579060	4460590	129	0.00%	0.00%	0.00%	0	IP-EIGRP: PDM
146	503716	5850050	86	0.00%	0.00%	0.00%	0	IP-EIGRP: HELLO
147	493512	5725897	86	0.00%	0.00%	0.00%	0	IP-EIGRP: HELLO

#sh interface gi0/0
GigabitEthernet0/0 is up, line protocol is up
Hardware is MV96340 Ethernet, address is 001b.d439.65b0 (bia 001b.d439.65b0)
Description: Control DMZ (to Firewall)
Internet address is 10.10.30.2/28
MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 2/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, media type is T
output flow-control is XON, input flow-control is XON
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters 12:40:48
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 11416000 bits/sec, 9247 packets/sec
5 minute output rate 5008000 bits/sec, 10118 packets/sec
     198456731 packets input, 1474299571 bytes, 0 no buffer
     Received 261932 broadcasts, 0 runts, 0 giants, 0 throttles
     1514 input errors, 0 CRC, 0 frame, 0 overrun, 1514 ignored
     0 watchdog, 0 multicast, 0 pause input
     0 input packets with dribble condition detected
     221189679 packets output, 1584642221 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 2786 pause output
     0 output buffer failures, 0 output buffers swapped out

#sh interface gi0/1
GigabitEthernet0/1 is up, line protocol is up
Hardware is MV96340 Ethernet, address is 001b.d439.65b1 (bia 001b.d439.65b1)
Description: Core DMZ (to transport routers)
Internet address is 10.10.20.2/26
MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 3/255, rxload 2/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, media type is T
output flow-control is XON, input flow-control is XON
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters 12:40:50
Input queue: 0/75/2/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 9163000 bits/sec, 12797 packets/sec
5 minute output rate 15089000 bits/sec, 11908 packets/sec
     283011297 packets input, 1756970027 bytes, 0 no buffer
     Received 109970 broadcasts, 0 runts, 0 giants, 2 throttles
     279476 input errors, 0 CRC, 0 frame, 0 overrun, 279476 ignored
     0 watchdog, 0 multicast, 0 pause input
     0 input packets with dribble condition detected
     259671581 packets output, 381573842 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 280171 pause output
     0 output buffer failures, 0 output buffers swapped out

Bert Gevers · ‎11-04-2010

Thanks for the output Aaron.

As we can see from the show process cpu:

CPU utilization for five seconds: 80%/79% => 79% of the CPU utilization occurs under interrupt.

A Cisco 2821 is using interrupts mostly to perform CEF packet forwarding (which is the optimal forwarding method on the 2821). Besides some exceptions, this basically means that you are running close to the platform limitations here.

The ignored counter for the interfaces indicates the amount of received packets ignored by the interface because the interface hardware ran low on internal buffers. Basically, this occurs due to short bursts of traffic and again are an indication that at some points, the devices limits are being reached.

It may be possible to reduce this ( flow-control / QOS ), however, this will depend on how large the bursts are.

In summary, I believe you are reaching the platform limitations of the 2821 and you should consider either offloading traffic to a different device or upgrading to a more powerfull router.

HTH,

Bert

aarondcounts · ‎11-04-2010

I actually have a 2851 setup as my standby router but I don't think I would gain anything by using it over the 2821. I did find that some of my traffic is passing through the router twice and one a lot of the trips it is going in and out the same interface. I am actually changing this tonight to hopefully drop the traffic load enough to handle the bursts and drop the CPU.

esomarriba · ‎11-04-2010

Aaron,

Cisco 2821 is hitting the limits of the platform. You need to upgrade to a bigger device. Make sure you are not running IP ACCOUNTING.

HTH,

Elyinn.-

Bert Gevers · ‎11-05-2010

Hello Aaron,

In regards to changing the 2821 to the standby 2851, this may make a small difference.

Some basic performance figures of the devices can be found here:

http://www.cisco.com/web/partners/downloads/765/tools/quickreference/routerperformance.pdf

Off course, try first to offload traffic which shouldn't hit the device (eg. redirected traffic).

HTH,

Bert