09-04-2012 03:10 PM - edited 03-04-2019 05:28 PM
We have a problem with 100% CPU usage and a small packet loss when the router can't keep up at full speed (100Mb/s) NAT.
We are not using any inspect commands, so there are no overheads there.
Why is the router slowing down and grinding to a halt?
We are running a basic NAT and our ISP has provided us 100Mb/s VDSL connection. It's when we hit these high speeds that the router CPU usage hits 100% and we experience packet loss when pinging for example (intermittent no replies... etc).
Below is our running config and process information.
Your thoughts, fixes, comments and suggestions are greatly appreciated.
show proc cpu sort
r1.xxx.xxxx.com#show proc cpu sort
CPU utilization for five seconds: 96%/96%; one minute: 96%; five minutes: 96%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
98 340532 13421485 25 1.89% 1.28% 1.21% 0 Ethernet Msec Ti
2 35928 21401 1678 1.34% 1.04% 1.03% 0 Load Meter
92 2372784 528420 4490 0.63% 0.83% 0.96% 0 COLLECT STAT COU
146 1192 16749 71 0.47% 0.06% 0.01% 0 TCP Timer
289 93284 3293892 28 0.39% 0.25% 0.24% 0 PPP Events
281 18392 828131 22 0.23% 0.07% 0.06% 0 PPPoE Background
115 156872 146421 1071 0.23% 0.19% 0.15% 0 IP Input
288 134128 3293930 40 0.23% 0.41% 0.42% 0 PPP manager
97 23132 775596 29 0.15% 0.06% 0.07% 0 Ethernet Timer C
111 74836 3279452 22 0.15% 0.24% 0.23% 0 IPAM Manager
63 69968 555090 126 0.15% 0.23% 0.23% 0 LED Timers
283 17560 209076 83 0.07% 0.03% 0.05% 0 IP NAT Ager
274 7452 21461 347 0.07% 0.03% 0.00% 0 Compute load avg
188 7740 207739 37 0.07% 0.02% 0.00% 0 Inspect process
68 4680 106699 43 0.07% 0.01% 0.00% 0 Console redirect
32 7896 111262 70 0.07% 0.03% 0.00% 0 ARP Background
17 4212 104127 40 0.07% 0.02% 0.00% 0 IPC Periodic Tim
25 1380 21372 64 0.07% 0.00% 0.00% 0 IPC Loadometer
56 6512 54449 119 0.07% 0.02% 0.00% 0 Fast Throttle Ti
244 2860 189 15132 0.07% 0.14% 0.03% 8 Virtual Exec
AND MORE... but I omitted it because I was getting the message "This message can not be displayed due to its content. Please use the Contact Us link with any questions"...
Our Running Config
version 15.1
no service pad
service timestamps debug datetime msec
service timestamps log datetime msec
no service password-encryption
!
hostname r1.essex.xxxx.xxx
!
boot-start-marker
boot system flash c880data-universalk9-mz.151-4.M3.bin
boot-end-marker
!
!
no logging buffered
enable secret 5 xxxxxx
enable password xxxxxx
!
no aaa new-model
memory-size iomem 10
no ip source-route
!
!
!
ip dhcp excluded-address 192.168.0.1
ip dhcp excluded-address 192.168.0.50 192.168.0.255
!
ip dhcp pool NET-POOL
network 192.168.0.0 255.255.255.0
default-router 192.168.0.1
dns-server 8.8.8.8 8.8.4.4
!
!
ip cef
ip name-server 8.8.8.8
ip name-server 8.8.4.4
no ipv6 cef
!
!
!
!
!
!
!
!
controller VDSL 0
!
no ip ftp passive
!
!
!
!
!
!
!
interface Ethernet0
no ip address
!
interface Ethernet0.101
encapsulation dot1Q 101
pppoe-client dial-pool-number 1
!
interface ATM0
no ip address
shutdown
no atm ilmi-keepalive
!
interface FastEthernet0
no ip address
!
interface FastEthernet1
no ip address
shutdown
!
interface FastEthernet2
no ip address
shutdown
!
interface FastEthernet3
no ip address
shutdown
!
interface Vlan1
ip address 192.168.0.1 255.255.255.0
ip nat inside
ip virtual-reassembly in
ip tcp adjust-mss 1452
!
interface Dialer0
ip address 81.138.131.190 255.255.255.248
no ip redirects
no ip unreachables
no ip proxy-arp
ip mtu 1492
ip nat outside
ip virtual-reassembly in
encapsulation ppp
dialer pool 1
ppp authentication chap callin
ppp chap hostname xxxxxxxx
ppp chap password 0 xxxxxxxxx
ppp ipcp route default
no cdp enable
!
ip forward-protocol nd
no ip http server
ip http secure-server
!
ip nat inside source list 101 interface Dialer0 overload
ip nat inside source static 192.168.0.250 xxx.xxx.xxx.xxx
!
access-list 101 permit ip any any
!
!
!
!
!
Solved! Go to Solution.
09-06-2012 06:57 AM
Please bear in mind that I currently have virtual-reassembly disabled on both interfaces, however here is the output:
Dialer0:
Virtual Fragment Reassembly (VFR) is DISABLED [in]
Concurrent reassemblies (max-reassemblies): 16
Fragments per reassembly (max-fragments): 32
Reassembly timeout (timeout): 3 seconds
Drop fragments: OFF
Current reassembly count:0
Current fragment count:0
Total reassembly count:1334
Total reassembly timeout count:93
Vlan1:
Virtual Fragment Reassembly (VFR) is DISABLED [in]
Concurrent reassemblies (max-reassemblies): 16
Fragments per reassembly (max-fragments): 32
Reassembly timeout (timeout): 3 seconds
Drop fragments: OFF
Current reassembly count:0
Current fragment count:0
Total reassembly count:2
Total reassembly timeout count:0
Does this look normal? I can enable virtual-reassembly again and max out the router to see what happens? Thank you for yout time.
09-06-2012 09:02 AM
I presume that the counters of virtual-reassembly can not be cleared (need a reload of router / or maybe shut-no shut of the interfaces).
But as already indicated by JosephDoherty
"
Your configuration looks pretty "clean", so your only real solution would be a "faster" device.
"
09-07-2012 01:22 AM
>We are running a basic NAT and our ISP has provided us 100Mb/s VDSL connection. It's when we hit these high speeds that the router CPU usage hits 100% and we experience packet loss when pinging for example (intermittent no replies... etc).
can you indicate when the CPU rise to 100% ? when the throughput is .... Mbps
how you verify the throughput ?
can you paste the sh tech ? or send me by email ?
09-07-2012 02:38 AM
Hey Roberto,
The CPU hits the 100% mark when we are pushing around 90Mb/s inbound from the Dialer0 to the Vlan1 (downloading a file for example). There are not many NAT clients behind the network yet, so it's purely throughput not bloating of the NAT translation table. I think even when we hit 100% CPU, the translation table only has 100 entries.
I'll private message you the output of sh tech now!
Thank you for your help!
Best Regards,
Ryan
09-09-2012 11:13 AM
The CPU hits the 100% mark when we are pushing around 90Mb/s inbound from the Dialer0 to the Vlan1 (downloading a file for example). There are not many NAT clients behind the network yet, so it's purely throughput not bloating of the NAT translation table. I think even when we hit 100% CPU, the translation table only has 100 entries.
That is normal, and consistent or exceeding the performances tested by Cisco. See attachment, NAT testing.
With such a fast circuit, you will need a faster router.
09-09-2012 10:09 AM
Hey Roberto,
I have private messaged you with regards to the output of "sh tech". It's far too large to paste into the message box, it just makes all of my browsers hang.
I initially asked for your email address, but I have just uploaded it to one of our servers so you can view it directly from there.
I have private messaged you the link.
Thank you for your help Roberto.
09-10-2012 09:52 AM
Ok
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide