Problems with 99% cpu load on a Cisco 2951 ISR

[Erik Hennerfors]

Hi
I'm having problems with a Cisco 2951 ISR when users connect to a VPN service through Open VPN the CPU throttles at 99% and is only lowered when the router is being manually reloaded. 

I have not enough knowledge about IOS the know where to begin, the only thing I manage to get is the log and that the process "IP Nat Ager" is the cause of the high cpu utilization.

I've tried to clear the nat translations (clear ip nat trans *) but that seems to do nothing except clear the nat translations ;)

Does anyone know where to start?

gimli#sh proc cpu | exclude 0.00
CPU utilization for five seconds: 99%/1%; one minute: 99%; five minutes: 99%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
90 8428 21810 386 0.16% 0.15% 0.15% 0 Per-Second Jobs
109 8588 76816 111 0.16% 0.19% 0.18% 0 Netclock Backgro
174 17608 35624 494 0.08% 0.04% 0.05% 0 IP Input
344 3915604 1741515 2248 98.05% 98.54% 96.65% 0 IP NAT Ager

Nov 10 13:52:37: %SYS-4-CHUNKSIBLINGSEXCEED: Number of siblings in a chunk has gone above the threshold. Threshold:10000 Sibling-Count:29696 Chunk:0x1480EA5C Name:NAT Fragment0 C -Process= "Chunk Manager", ipl= 6, pid= 1
-Traceback= 5CAC39Cz 400784Cz 4F6D0C4z 4F53FF4z
Nov 10 13:52:47: %SYS-4-CHUNKSIBLINGSEXCEED: Number of siblings in a chunk has gone above the threshold. Threshold:10000 Sibling-Count:29731 Chunk:0x1480EA5C Name:NAT Fragment0 C -Process= "Chunk Manager", ipl= 6, pid= 1
-Traceback= 5CAC39Cz 400784Cz 4F6D0C4z 4F53FF4z
Nov 10 13:52:57: %SYS-4-CHUNKSIBLINGSEXCEED: Number of siblings in a chunk has gone above the threshold. Threshold:10000 Sibling-Count:29790 Chunk:0x1480EA5C Name:NAT Fragment0 C -Process= "Chunk Manager", ipl= 6, pid= 1
-Traceback= 5CAC39Cz 400784Cz 4F6D0C4z 4F53FF4z
Nov 10 13:53:09: %SYS-4-CHUNKSIBLINGSEXCEED: Number of siblings in a chunk has gone above the threshold. Threshold:10000 Sibling-Count:29841 Chunk:0x1480EA5C Name:NAT Fragment0 C -Process= "Chunk Manager", ipl= 6, pid= 1
-Traceback= 5CAC39Cz 400784Cz 4F6D0C4z 4F53FF4z

I also enabled NetFlow which gets me this (I masked my ip-address, so that is *.*.*.*):

gimli#show ip cache flow
IP packet size distribution (36670541 total packets):
1-32 64 96 128 160 192 224 256 288 320 352 384 416 448 480
.000 .100 .015 .136 .391 .038 .002 .000 .000 .000 .000 .000 .000 .000 .027

512 544 576 1024 1536 2048 2560 3072 3584 4096 4608
.000 .000 .000 .002 .281 .000 .000 .000 .000 .000 .000

IP Flow Switching Cache, 278544 bytes
115 active, 3981 inactive, 85425 added
1722831 ager polls, 0 flow alloc failures
Active flows timeout in 30 minutes
Inactive flows timeout in 15 seconds
IP Sub Flow Cache, 34056 bytes
115 active, 909 inactive, 85419 added, 85419 added to flow
0 alloc failures, 0 force free
1 chunk, 4 chunks added
last clearing of statistics never
Protocol Total Flows Packets Bytes Packets Active(Sec) Idle(Sec)
-------- Flows /Sec /Flow /Pkt /Sec /Flow /Flow
TCP-WWW 19901 0.9 62 230 61.8 4.2 8.7
TCP-other 43034 2.1 348 1073 749.1 6.0 12.4
UDP-DNS 7721 0.3 1 65 0.3 0.0 15.4
UDP-NTP 2204 0.1 1 76 0.1 0.0 15.2
UDP-other 12435 0.6 1642 129 1020.6 6.9 15.4
ICMP 14 0.0 419 60 0.2 421.3 13.4
Total: 85309 4.2 429 518 1832.4 5.1 12.3

SrcIf SrcIPaddress DstIf DstIPaddress Pr SrcP DstP Pkts
Gi0/1.2 *.*.*.* Gi0/0* 104.20.45.84 06 F22B 01BB 5
Gi0/1.2 *.*.*.* Gi0/0* 52.59.43.154 06 C841 01BB 1
Gi0/1.2 *.*.*.* Gi0/0* 52.59.43.154 06 C840 01BB 4
Gi0/1.2 *.*.*.* Gi0/0* 10.0.2.13 06 D0DA 0CD3 4
Gi0/1.2 *.*.*.* Gi0/0* 10.0.2.15 06 D0D9 0CD3 4
Gi0/1.2 10.1.1.10 Gi0/0 216.58.209.142 06 C637 01BB 4

SrcIf SrcIPaddress DstIf DstIPaddress Pr SrcP DstP Pkts
Gi0/1.2 *.*.*.* Gi0/0* 10.0.2.15 06 D0D8 0CD3 4
Gi0/1.2 *.*.*.* Gi0/0* 10.0.2.13 06 D0D7 0CD3 4
Gi0/1.5 10.1.4.11 Gi0/0 77.93.242.50 11 1AE1 1AE9 1
Gi0/1.3 10.1.2.14 Gi0/0 193.182.7.162 06 D6C6 0050 2
Gi0/1.1 10.1.0.10 Null 8.8.8.8 11 DA90 0035 1
Gi0/1.5 *.*.*.* Gi0/0* 212.124.9.160 11 1AE1 F439 1
Gi0/1.2 10.1.1.10 Gi0/0 157.55.130.161 11 7F19 9C4E 1
Gi0/1.2 10.1.1.10 Gi0/0 157.55.130.157 11 7F19 9C44 1
Gi0/1.1 10.1.0.10 Null 8.8.8.8 11 FAC1 0035 1
Gi0/1.2 10.1.1.10 Gi0/0 64.4.23.155 11 7F19 9C52 1
Gi0/1.2 *.*.*.* Gi0/0* 31.13.93.36 06 FAD7 01BB 11
Gi0/1.2 *.*.*.* Gi0/0* 54.175.239.65 06 F03D 01BB 2
Gi0/1.2 *.*.*.* Gi0/0* 74.86.208.246 06 F596 01BB 514
Gi0/1.2 10.1.1.10 Gi0/0 64.233.163.125 06 EF95 1466 1
Gi0/1.2 10.1.1.8 Gi0/0 17.188.165.211 06 CCD3 1467 2
Gi0/1.1 10.1.0.10 Null 8.8.8.8 11 D993 0035 1
Gi0/1.2 10.1.1.10 Gi0/0 64.233.163.189 11 CD47 01BB 154
Gi0/1.1 10.1.0.14 Gi0/1.5 10.1.4.12 06 0202 D723 38
Gi0/1.5 10.1.4.12 Gi0/1.1 10.1.0.14 06 D723 0202 38
Gi0/1.2 *.*.*.* Gi0/0* 50.16.185.14 06 C842 01BB 1
Gi0/1.2 *.*.*.* Gi0/0* 104.16.3.9 06 C84C 01BB 3
Gi0/1.2 *.*.*.* Gi0/0* 216.58.211.142 06 C897 01BB 9
Gi0/1.2 10.1.1.10 Local 10.1.0.1 06 C70B 0016 21
Gi0/1.2 10.1.1.10 Gi0/0 52.51.193.54 06 EFEF 01BB 2
Gi0/1.2 *.*.*.* Gi0/0* 178.62.228.83 06 F243 0050 2
Gi0/1.2 10.1.1.10 Null 8.8.4.4 11 E25D 0035 1
Gi0/1.2 10.1.1.10 Null 8.8.8.8 11 CE40 0035 1
Gi0/1.2 *.*.*.* Gi0/0* 216.58.211.142 06 C839 01BB 1
Gi0/1.2 10.1.1.10 Gi0/0 74.86.208.246 06 F596 01BB 514
Gi0/1.2 10.1.1.10 Gi0/0 52.72.3.68 06 C83E 01BB 1
Gi0/1.1 10.1.0.10 Null 8.8.8.8 11 CE80 0035 1
Gi0/1.2 *.*.*.* Gi0/0* 64.4.23.155 11 7F19 9C52 1
Gi0/1.1 10.1.0.10 Null 8.8.8.8 11 CE9E 0035 1
Gi0/1.3 *.*.*.* Gi0/0* 193.182.7.162 06 D6C6 0050 2

SrcIf SrcIPaddress DstIf DstIPaddress Pr SrcP DstP Pkts
Gi0/1.2 *.*.*.* Gi0/0* 216.58.211.142 11 CE41 01BB 6
Gi0/1.2 *.*.*.* Gi0/0* 157.55.235.169 11 7F19 9C42 1
Gi0/1.2 *.*.*.* Gi0/0* 157.55.235.167 11 7F19 9C5F 1
Gi0/1.2 *.*.*.* Gi0/0* 192.0.73.2 06 C845 01BB 1
Gi0/1.2 *.*.*.* Gi0/0* 192.0.73.2 06 C846 01BB 1
Gi0/1.1 10.1.0.10 Null 8.8.8.8 11 EDC4 0035 1
Gi0/1.2 10.1.1.12 Gi0/0 10.0.2.15 06 EA75 0CD3 1
Gi0/1.2 10.1.1.10 Gi0/0 216.58.209.133 11 C6B1 01BB 10
Gi0/1.2 *.*.*.* Gi0/0* 54.243.114.45 06 C895 0050 3
Gi0/1.2 *.*.*.* Gi0/0* 54.243.114.45 06 C894 0050 4
Gi0/1.2 10.1.1.10 Gi0/0 136.243.53.2 06 F239 2726 2
Gi0/1.1 10.1.0.14 Gi0/1.4 10.1.3.3 06 2FA9 C945 5
Gi0/1.4 10.1.3.3 Gi0/1.1 10.1.0.14 06 C945 2FA9 5
Gi0/1.4 10.1.3.3 Gi0/1.2 10.1.1.8 06 01BD D0D6 16
Gi0/1.2 10.1.1.8 Gi0/1.4 10.1.3.3 06 D0D6 01BD 17
Gi0/1.2 10.1.1.8 Gi0/1.4 10.1.3.3 06 D0D6 01BD 15
Gi0/1.2 10.1.1.10 Gi0/0 54.175.239.65 06 F03D 01BB 2
Gi0/1.2 10.1.1.10 Gi0/0 216.58.211.142 06 C897 01BB 9
Gi0/1.2 10.1.1.10 Gi0/0 157.55.235.167 11 7F19 9C5F 1
Gi0/1.2 10.1.1.10 Gi0/0 157.55.235.169 11 7F19 9C42 1
Gi0/1.2 10.1.1.10 Gi0/0 216.58.211.142 06 C839 01BB 1
Gi0/1.2 *.*.*.* Gi0/0* 216.58.209.142 06 C637 01BB 4
Gi0/1.2 10.1.1.10 Null 8.8.8.8 11 E25D 0035 1
Gi0/1.2 10.1.1.10 Null 8.8.4.4 11 CE40 0035 1
Gi0/1.2 *.*.*.* Gi0/0* 17.188.165.211 06 CCD3 1467 2
Gi0/1.2 10.1.1.10 Gi0/0 104.20.45.84 06 F22B 01BB 5
Gi0/1.1 10.1.0.14 Gi0/1.2 10.1.1.10 06 3264 C892 18
Gi0/1.1 10.1.0.14 Gi0/1.2 10.1.1.10 06 3264 C893 5
Gi0/1.2 10.1.1.10 Gi0/1.1 10.1.0.14 06 C896 3264 13
Gi0/1.1 10.1.0.14 Gi0/1.2 10.1.1.10 06 3264 C891 9
Gi0/1.1 10.1.0.14 Gi0/1.2 10.1.1.10 06 3264 C896 13
Gi0/1.2 10.1.1.10 Gi0/1.1 10.1.0.14 06 C891 3264 9
Gi0/1.2 10.1.1.10 Gi0/1.1 10.1.0.14 06 C893 3264 6
Gi0/1.2 10.1.1.10 Gi0/1.1 10.1.0.14 06 C892 3264 19

SrcIf SrcIPaddress DstIf DstIPaddress Pr SrcP DstP Pkts
Gi0/1.2 10.1.1.10 Gi0/0 50.16.185.14 06 C842 01BB 1
Gi0/1.2 10.1.1.10 Gi0/0 104.16.3.9 06 C84C 01BB 3
Gi0/1.4 *.*.*.* Gi0/0* 50.31.164.146 06 EE44 01BB 4
Gi0/1.2 10.1.1.10 Gi0/1.1 10.1.0.14 06 C836 3264 17
Gi0/1.1 10.1.0.14 Gi0/1.2 10.1.1.10 06 3264 C836 20
Gi0/1.2 10.1.1.10 Gi0/0 216.58.211.142 11 CE41 01BB 6
Gi0/1.1 10.1.0.14 Gi0/1.2 10.1.1.10 06 3264 C828 159
Gi0/1.2 10.1.1.10 Gi0/1.1 10.1.0.14 06 C828 3264 165
Gi0/1.2 10.1.1.10 Gi0/0 65.55.223.40 11 7F19 9C43 1
Gi0/1.5 *.*.*.* Gi0/0* 193.151.200.249 11 1AE1 E6B3 1
Gi0/1.2 *.*.*.* Gi0/0* 10.0.2.15 06 EA75 0CD3 1
Gi0/1.2 10.1.1.10 Gi0/1.5 10.1.4.11 06 C093 0D3D 662
Gi0/1.5 10.1.4.11 Gi0/1.2 10.1.1.10 06 0D3D C093 667
Gi0/1.2 10.1.1.10 Gi0/0 31.13.93.36 06 FAD7 01BB 11
Gi0/1.1 10.1.0.10 Null 8.8.8.8 11 F14B 0035 1
Gi0/1.2 *.*.*.* Gi0/0* 157.55.56.170 11 7F19 9C5F 1
Gi0/1.5 10.1.4.11 Gi0/0 193.151.200.249 11 1AE1 E6B3 1
Gi0/1.2 *.*.*.* Gi0/0* 52.72.3.68 06 C83E 01BB 1
Gi0/1.2 10.1.1.10 Gi0/0 52.59.43.154 06 C840 01BB 4
Gi0/1.2 10.1.1.10 Gi0/0 52.59.43.154 06 C841 01BB 1
Gi0/1.2 *.*.*.* Gi0/0* 157.55.130.161 11 7F19 9C4E 1
Gi0/1.2 *.*.*.* Gi0/0* 157.55.130.157 11 7F19 9C44 1
Gi0/1.2 10.1.1.10 Gi0/0 178.62.228.83 06 F243 0050 2
Gi0/1.2 10.1.1.10 Gi0/0 54.243.114.45 06 C894 0050 4
Gi0/1.2 10.1.1.10 Gi0/0 54.243.114.45 06 C895 0050 3
Gi0/1.2 10.1.1.8 Gi0/0 10.0.2.13 06 D0DA 0CD3 4
Gi0/1.2 10.1.1.8 Gi0/0 10.0.2.15 06 D0D8 0CD3 4
Gi0/1.2 10.1.1.8 Gi0/0 10.0.2.15 06 D0D9 0CD3 4
Gi0/1.2 10.1.1.8 Gi0/0 10.0.2.13 06 D0D7 0CD3 4
Gi0/1.2 *.*.*.* Gi0/0* 52.51.193.54 06 EFEF 01BB 2
Gi0/1.2 *.*.*.* Gi0/0* 64.233.163.189 11 CD47 01BB 154
Gi0/1.2 *.*.*.* Gi0/0* 136.243.53.2 06 F239 2726 2
Gi0/1.4 10.1.3.10 Gi0/0 50.31.164.146 06 EE44 01BB 4
Gi0/1.2 *.*.*.* Gi0/0* 64.233.163.125 06 EF95 1466 1

SrcIf SrcIPaddress DstIf DstIPaddress Pr SrcP DstP Pkts
Gi0/1.2 *.*.*.* Gi0/0* 64.233.162.188 06 EF6D 146C 1
Gi0/1.2 10.1.1.10 Gi0/0 31.13.64.16 06 F190 01BB 3
Gi0/1.2 *.*.*.* Gi0/0* 216.58.201.174 06 C81F 01BB 1
Gi0/1.2 *.*.*.* Gi0/0* 192.176.163.248 06 C82F 01BB 1
Gi0/1.2 *.*.*.* Gi0/0* 192.176.163.248 06 C82E 01BB 1
Gi0/1.2 10.1.1.10 Gi0/0 162.247.242.20 06 C7F4 01BB 4
Gi0/1.2 *.*.*.* Gi0/0* 216.58.209.133 11 C6B1 01BB 11
Gi0/1.2 10.1.1.10 Gi0/0 54.247.71.211 06 C07A 0050 2
Gi0/1.5 10.1.4.11 Null 255.255.255.255 11 0044 0043 1