cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
10116
Views
4
Helpful
21
Replies

100% CPU Usage on 887VA when network traffic is heavy

Ian Stephens
Level 1
Level 1

We have a problem with 100% CPU usage and a small packet loss when the router can't keep up at full speed (100Mb/s) NAT.

We are not using any inspect commands, so there are no overheads there.

Why is the router slowing down and grinding to a halt?

We are running a basic NAT and our ISP has provided us 100Mb/s VDSL connection.  It's when we hit these high speeds that the router CPU usage hits 100% and we experience packet loss when pinging for example (intermittent no replies... etc).

Below is our running config and process information. 

Your thoughts, fixes, comments and suggestions are greatly appreciated.

show proc cpu sort

r1.xxx.xxxx.com#show proc cpu sort

CPU utilization for five seconds: 96%/96%; one minute: 96%; five minutes: 96%

PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process

  98      340532    13421485         25  1.89%  1.28%  1.21%   0 Ethernet Msec Ti

   2       35928       21401       1678  1.34%  1.04%  1.03%   0 Load Meter   

  92     2372784      528420       4490  0.63%  0.83%  0.96%   0 COLLECT STAT COU

146        1192       16749         71  0.47%  0.06%  0.01%   0 TCP Timer     

289       93284     3293892         28  0.39%  0.25%  0.24%   0 PPP Events   

281       18392      828131         22  0.23%  0.07%  0.06%   0 PPPoE Background

115      156872      146421       1071  0.23%  0.19%  0.15%   0 IP Input     

288      134128     3293930         40  0.23%  0.41%  0.42%   0 PPP manager   

  97       23132      775596         29  0.15%  0.06%  0.07%   0 Ethernet Timer C

111       74836     3279452         22  0.15%  0.24%  0.23%   0 IPAM Manager 

  63       69968      555090        126  0.15%  0.23%  0.23%   0 LED Timers   

283       17560      209076         83  0.07%  0.03%  0.05%   0 IP NAT Ager   

274        7452       21461        347  0.07%  0.03%  0.00%   0 Compute load avg

188        7740      207739         37  0.07%  0.02%  0.00%   0 Inspect process

  68        4680      106699         43  0.07%  0.01%  0.00%   0 Console redirect

  32        7896      111262         70  0.07%  0.03%  0.00%   0 ARP Background

  17        4212      104127         40  0.07%  0.02%  0.00%   0 IPC Periodic Tim

  25        1380       21372         64  0.07%  0.00%  0.00%   0 IPC Loadometer

  56        6512       54449        119  0.07%  0.02%  0.00%   0 Fast Throttle Ti

244        2860         189      15132  0.07%  0.14%  0.03%   8 Virtual Exec 

AND MORE... but I omitted it because I was getting the message "This message can not be displayed due to its content. Please use the Contact Us link with any questions"...

Our Running Config

version 15.1

no service pad

service timestamps debug datetime msec

service timestamps log datetime msec

no service password-encryption

!

hostname r1.essex.xxxx.xxx

!

boot-start-marker

boot system flash c880data-universalk9-mz.151-4.M3.bin

boot-end-marker

!

!

no logging buffered

enable secret 5 xxxxxx

enable password xxxxxx

!

no aaa new-model

memory-size iomem 10

no ip source-route

!

!

!

ip dhcp excluded-address 192.168.0.1

ip dhcp excluded-address 192.168.0.50 192.168.0.255

!

ip dhcp pool NET-POOL

network 192.168.0.0 255.255.255.0

default-router 192.168.0.1

dns-server 8.8.8.8 8.8.4.4

!

!

ip cef

ip name-server 8.8.8.8

ip name-server 8.8.4.4

no ipv6 cef

!

!

!

!

!

!

!

!

controller VDSL 0

!

no ip ftp passive

!

!

!

!

!

!

!

interface Ethernet0

no ip address

!

interface Ethernet0.101

encapsulation dot1Q 101

pppoe-client dial-pool-number 1

!

interface ATM0

no ip address

shutdown

no atm ilmi-keepalive

!

interface FastEthernet0

no ip address

!

interface FastEthernet1

no ip address

shutdown

!

interface FastEthernet2

no ip address

shutdown

!

interface FastEthernet3

no ip address

shutdown

!

interface Vlan1

ip address 192.168.0.1 255.255.255.0

ip nat inside

ip virtual-reassembly in

ip tcp adjust-mss 1452

!

interface Dialer0

ip address 81.138.131.190 255.255.255.248

no ip redirects

no ip unreachables

no ip proxy-arp

ip mtu 1492

ip nat outside

ip virtual-reassembly in

encapsulation ppp

dialer pool 1

ppp authentication chap callin

ppp chap hostname xxxxxxxx

ppp chap password 0 xxxxxxxxx

ppp ipcp route default

no cdp enable

!

ip forward-protocol nd

no ip http server

ip http secure-server

!

ip nat inside source list 101 interface Dialer0 overload

ip nat inside source static 192.168.0.250 xxx.xxx.xxx.xxx

!

access-list 101 permit ip any any

!

!

!

!

!


21 Replies 21

Please bear in mind that I currently have virtual-reassembly disabled on both interfaces, however here is the output:

Dialer0:

   Virtual Fragment Reassembly (VFR) is DISABLED [in]

   Concurrent reassemblies (max-reassemblies): 16

   Fragments per reassembly (max-fragments): 32

   Reassembly timeout (timeout): 3 seconds

   Drop fragments: OFF

   Current reassembly count:0

   Current fragment count:0

   Total reassembly count:1334

   Total reassembly timeout count:93

Vlan1:

   Virtual Fragment Reassembly (VFR) is DISABLED [in]

   Concurrent reassemblies (max-reassemblies): 16

   Fragments per reassembly (max-fragments): 32

   Reassembly timeout (timeout): 3 seconds

   Drop fragments: OFF

   Current reassembly count:0

   Current fragment count:0

   Total reassembly count:2

   Total reassembly timeout count:0

Does this look normal?  I can enable virtual-reassembly again and max out the router to see what happens?  Thank you for yout time.

I presume that the counters of virtual-reassembly can not be cleared (need a reload of router / or maybe shut-no shut of the interfaces).

But as already indicated by JosephDoherty

"

Your configuration looks pretty "clean", so your only real solution would be a "faster" device.

"

>We are running a basic NAT and our ISP has provided us 100Mb/s VDSL connection.  It's when we hit these high speeds that the router CPU usage hits 100% and we experience packet loss when pinging for example (intermittent no replies... etc).

can you indicate when the CPU rise to 100% ? when the throughput is .... Mbps

how you verify the throughput ?

can you paste the sh tech ? or send me by email ?

Hey Roberto,

The CPU hits the 100% mark when we are pushing around 90Mb/s inbound from the Dialer0 to the Vlan1 (downloading a file for example).  There are not many NAT clients behind the network yet, so it's purely throughput not bloating of the NAT translation table.  I think even when we hit 100% CPU, the translation table only has 100 entries.

I'll private message you the output of sh tech now!

Thank you for your help!

Best Regards,

Ryan

The CPU hits the 100% mark when we are pushing around 90Mb/s inbound from the Dialer0 to the Vlan1 (downloading a file for example).  There are not many NAT clients behind the network yet, so it's purely throughput not bloating of the NAT translation table.  I think even when we hit 100% CPU, the translation table only has 100 entries.

That is normal, and consistent or exceeding the performances tested by Cisco. See attachment, NAT testing.

With such a fast circuit, you will need a faster router.

Hey Roberto,

I have private messaged you with regards to the output of "sh tech".  It's far too large to paste into the message box, it just makes all of my browsers hang.

I initially asked for your email address, but I have just uploaded it to one of our servers so you can view it directly from there.

I have private messaged you the link.

Thank you for your help Roberto.

Ok

Review Cisco Networking products for a $25 gift card