No 100mbit/s with cisco 892 router, cpu maxes out

[jwillemadhese]

Hi there,

We have a basic corporate internet connection (100mbit/s down, 10mbit/s up, some guaranties) with eight fixed ip addresses (in the configuration .72 to .79).  To hook up our network to the internet, we are using a cisco 892 router.

I am new to cisco equipment, so I had some trouble getting things to work, but I got there.

Now we are facing this bandwith limitation problem: whenever we are using more than about 60mbit/s down bandwidth, the cpu of this router is maxing out (98 - 99% with 'show processes cpu history').

When I'm downloading a torrent (debian dvd) at 4MB/s (or 48mbit/s) cpu is running at around 46% (tops at 49%).  Stopping the download results in 14% cpu usage tops.

When using the 'show processes cpu sorted' command, I get this:

maximilian#show processes cpu sorted

CPU utilization for five seconds: 46%/43%; one minute: 33%; five minutes: 15%

PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process

  82    35978164    10389766       3462  2.31%  2.08%  2.06%   0 COLLECT STAT COU

  90      203264   266300211          0  0.31%  0.31%  0.28%   0 Ethernet Msec Ti

  31         664         844        786  0.31%  0.19%  0.08%   8 SSH Process     

108     2039472     5100352        399  0.23%  0.34%  0.30%   0 IP Input        

334       35468     4269539          8  0.23%  0.09%  0.04%   0 IP NAT Ager     

324       21204     2077221         10  0.07%  0.03%  0.02%   0 Per-Second Jobs 

104       24240    64783802          0  0.07%  0.04%  0.02%   0 IPAM Manager    

336        7404      108003         68  0.07%  0.02%  0.00%   0 IP VFR proc     

  33       66952      321851        208  0.07%  0.00%  0.00%   0 ARP Input       

   9           0           2          0  0.00%  0.00%  0.00%   0 Timers          

...

So cpu utilization is around 46%, while no process is actually using more than 2.31%.  Also, these numbers don't change if I stop the download.

This is our configuration (with some parts obscured):

maximilian#show run        

Building configuration...

Current configuration : 8035 bytes

!

! Last configuration change at 09:49:27 UTC Wed May 30 2012 by jan

! NVRAM config last updated at 13:55:40 UTC Tue May 22 2012 by jan

! NVRAM config last updated at 13:55:40 UTC Tue May 22 2012 by jan

version 15.2

service timestamps debug datetime msec

service timestamps log datetime msec

no service password-encryption

!

hostname maximilian

!

boot-start-marker

boot config flash:maxi-config

boot-end-marker

!

!

logging buffered 51200 warnings

!

no aaa new-model

!

crypto pki token default removal timeout 0

!

crypto pki trustpoint TP-self-signed-3260749506

enrollment selfsigned

subject-name cn=IOS-Self-Signed-Certificate-3260749506

revocation-check none

rsakeypair TP-self-signed-3260749506

!

crypto pki trustpoint tti

revocation-check crl

!

!

crypto pki certificate chain TP-self-signed-3260749506

certificate self-signed 01

...........[snip].....

            quit

crypto pki certificate chain tti

!

!

!

ip dhcp excluded-address 10.10.10.1

ip dhcp excluded-address 192.168.1.0 192.168.1.49

!

ip dhcp pool cvo-pool

import all

network 10.10.10.0 255.255.255.248

default-router 10.10.10.1

dns-server 8.8.8.8

lease 0 2

!

ip dhcp pool maxi-pool

import all

network 192.168.1.0 255.255.255.0

default-router 192.168.1.1

dns-server 8.8.8.8

lease infinite

!

!

ip domain name adhese.org

ip name-server 8.8.8.8

ip inspect name DEFAULT100 ftp

ip inspect name DEFAULT100 h323

ip inspect name DEFAULT100 icmp

ip inspect name DEFAULT100 netshow

ip inspect name DEFAULT100 rcmd

ip inspect name DEFAULT100 realaudio

ip inspect name DEFAULT100 rtsp

ip inspect name DEFAULT100 esmtp

ip inspect name DEFAULT100 sqlnet

ip inspect name DEFAULT100 streamworks

ip inspect name DEFAULT100 tftp

ip inspect name DEFAULT100 tcp

ip inspect name DEFAULT100 udp

ip inspect name DEFAULT100 vdolive

ip cef

no ipv6 cef

!

!

!

!

multilink bundle-name authenticated

!

!

!

!

!

!

license udi pid CISCO892-K9 sn FC[snip]

!

!

username cisco privilege 15 secret 5 [snip]

username jan privilege 15

!

!

!

!

!

ip ssh pubkey-chain

  username jan

   key-hash ssh-rsa [snip] jan@[snip]

  quit

!

!

!

!

!

!

!

!

!

interface BRI0

no ip address

encapsulation hdlc

shutdown

isdn termination multidrop

!

interface FastEthernet0

no ip address

spanning-tree portfast

!

interface FastEthernet1

no ip address

spanning-tree portfast

!

interface FastEthernet2

switchport access vlan 2

no ip address

spanning-tree portfast

!

interface FastEthernet3

no ip address

spanning-tree portfast

!

interface FastEthernet4

no ip address

spanning-tree portfast

!

interface FastEthernet5

no ip address

spanning-tree portfast

!

interface FastEthernet6

switchport access vlan 2

no ip address

spanning-tree portfast

!

interface FastEthernet7

switchport access vlan 2

no ip address

spanning-tree portfast

!

interface FastEthernet8

ip address 192.168.3.2 255.255.255.0

duplex auto

speed auto

!

interface GigabitEthernet0

ip address [snip].94 255.255.255.252

ip access-group 101 in

no ip redirects

no ip unreachables

no ip proxy-arp

ip flow ingress

ip nat outside

ip inspect DEFAULT100 out

ip virtual-reassembly in

duplex auto

speed auto

!

interface Vlan1

ip address 10.10.10.1 255.255.255.248

ip access-group 100 in

ip nat inside

ip virtual-reassembly in

ip tcp adjust-mss 1452

!

interface Vlan2

ip address 192.168.1.1 255.255.255.0

ip flow ingress

ip nat inside

ip virtual-reassembly in

!

ip forward-protocol nd

!

ip flow-capture fragment-offset

ip flow-capture packet-length

ip flow-capture ttl

ip flow-capture vlan-id

ip flow-capture icmp

ip flow-capture ip-id

ip flow-capture mac-addresses

!

ip http server

ip http access-class 23

ip http authentication local

ip http secure-server

ip http timeout-policy idle 60 life 86400 requests 10000

ip nat pool public3-74 [snip].74 [snip].74 prefix-length 29

ip nat inside source list 74 pool public3-74 overload

ip nat inside source static tcp 192.168.1.42 22 [snip].72 22 extendable

ip nat inside source static udp 192.168.1.42 1194 [snip].72 1194 extendable

ip nat inside source static tcp 192.168.1.42 22 [snip].72 1489 extendable

ip route 0.0.0.0 0.0.0.0 [snip].93

ip route 0.0.0.0 0.0.0.0 192.168.3.1 5

ip route 10.8.0.0 255.255.255.0 192.168.1.42

!

access-list 1 permit 10.10.10.0 0.0.0.7

access-list 23 permit 192.168.1.42

access-list 23 permit 10.10.10.0 0.0.0.7

access-list 74 permit 10.10.10.0 0.0.0.7

access-list 74 permit 192.168.1.0 0.0.0.255

access-list 100 deny   ip host 255.255.255.255 any

access-list 100 deny   ip 127.0.0.0 0.255.255.255 any

access-list 100 permit ip any any

access-list 101 deny   ip 10.0.0.0 0.255.255.255 any

access-list 101 deny   ip 172.16.0.0 0.15.255.255 any

access-list 101 deny   ip 192.168.0.0 0.0.255.255 any

access-list 101 deny   ip 127.0.0.0 0.255.255.255 any

access-list 101 deny   ip host 255.255.255.255 any

access-list 101 permit tcp any host [snip].72 eq 1489

access-list 101 permit tcp any host [snip].72 eq 22

access-list 101 permit udp any host [snip].72 eq 1194

access-list 199 deny   ip any host 74.209.133.138

access-list 199 permit ip any any

no cdp run

!

!

!

!

!

snmp-server community [snip] RO

!

control-plane

!

!

!

!

mgcp profile default

!

!

!

!

banner exec ^C

% Password expiration warning.

-----------------------------------------------------------------------

Cisco Virtual Office (CVO) is installed on this device and it provides the

default username "cisco".

It is strongly suggested that you create a new username with a privilege level

of 15 using the following command.

username <myuser> privilege 15 secret 0 <mypassword>

Replace <myuser> and <mypassword> with the username and password you want to

use.

-----------------------------------------------------------------------

^C

banner login ^C

-----------------------------------------------------------------------

Cisco Virtual Office (CVO) is installed on this device and it provides the

default username "cisco".

It is strongly suggested that you create a new username with a privilege level

of 15 using the following command.

username <myuser> privilege 15 secret 0 <mypassword>

Replace <myuser> and <mypassword> with the username and password you want to

use.

For more information about CVO please go to http://www.cisco.com/go/cvo

-----------------------------------------------------------------------

^C

!

line con 0

login local

line aux 0

line vty 0 4

access-class 23 in

login local

length 0

transport input telnet ssh

line vty 5 15

access-class 23 in

privilege level 15

login local

transport input telnet ssh

!

end

Please tell me what I can do about this.  Or is this router not capable of doing 100mbit/s?

The net effect of running out of cpu is random connections dropping and no communication possible with the router.  I could cap the bandwidth at 90mbit/s or 80mbit/s, but I'd rather not.

The system image file is:  "flash:c890-universalk9-mz.152-1.T1.bin"

Thanks in advance!

Jan.

[Daniele Giordano]

If no necessary, you can try to disable ip inspect and ip flow.

You can also try to upgrade the IOS.

But I think in no better performance than your result:

http://www.cisco.com/web/AP/partners/ANZ_PE/borderless_network/Cisco_routerperformance.pdf

Regards.

[jwillemadhese]

Hi Daniele,

Thank you for your quick reply!

If I understand correctly, we need ip inspect to bypass the implicit deny ip any any of access-list 101, or else any incoming packet will be dropped?  For sure, we only need to inspect tcp, udp, ftp and icmp.

We turned on ip flow to be able to track which user was maxing out the router.  Is there a more cpu-friendly way to do this?  I once enabled mac accounting and ip accounting, and after 5 minutes the router just crashed...

I'd like to upgrade the IOS, but we have no subscription.  We were unaware of the cisco ways, and just bought the router on amazon..

If I read the performance data correcty, the 892 can handle 100,000 64-byte packets per second, resulting in 51.20 Mbps.  I would assume the packets in our download are MTU-sized, and I'd expect the router doing easily more than 100 Mbps then?

Regards,

Jan.

[jwillemadhese]

Hi Daniele,

I now turned off ip flow like you suggested, I think it got the cpu usage a little lower.  But I can stil easily max out the cpu with a simple download.

I could not turn off ip inspect (even without access-list 101), and I did trim down to tcp, udp, ftp, icmp, tftp and esmtp.  I don't see much difference though with this last change.

Regards,

Jan.

[Leo Laohoo]

We have a basic corporate internet connection (100mbit/s down, 10mbit/s up, some guaranties) with eight fixed ip addresses (in the configuration .72 to .79).  To hook up our network to the internet, we are using a cisco 892 router.

The 890 will not be able to handle traffic beyond 51.02 Mbps (half duplex and no encryption).

[jwillemadhese]

Thanks.  It's kind of sad that our ISP recommended a router that can't handle the bandwidth that they provide to us.  But then again, I guess the next step up cisco router is probably more than twice as expensive.

I was wondering though: when the router is flooded with more traffic than it can handle, the result is a failing network for all users.  What are my options to handle this situation?

I'd prefer to just drop (or maybe shape) traffic that can't be handled because of too high cpu usage.  However, I think the only option is to shape on a fixed bandwidth instead of on cpu usage?  Or will the router do something better if I define a quality of service strategy?  Better as in: cpu maxes out, but only low priority traffic is affected.

[Joseph W. Doherty · Accepted solution]

Disclaimer

The  Author of this posting offers the information contained within this  posting without consideration and with the reader's understanding that  there's no implied or expressed suitability or fitness for any purpose.  Information provided is for informational purposes only and should not  be construed as rendering professional advice of any kind. Usage of this  posting's information is solely at reader's own risk.

Liability Disclaimer

In  no event shall Author be liable for any damages whatsoever (including,  without limitation, damages for loss of use, data or profit) arising out  of the use or inability to use the posting's information even if Author  has been advised of the possibility of such damage.

Posting

Actually, the 890 series is rated at 100 Kpps, which for minimum size Ethernet packets is about 51 Mbps (as also noted in other posts) but Cisco also documents the 890 providing up to 1,400 Mbps for 1500 byte size packets.  Unfortunately "your mileage may vary"; i.e. actual throughput is very, very dependent on your particular traffic and what you configure your router to do against that traffic.  For example, you have NAT/PAT, interface ACLs, firewall inspection and NetFlow, all which consume additional CPU while processing packets.

Not knowing what exactly a customer will do with a router, Cisco makes very conservative usage recommendations, and for the 890, it recommends the WAN side doesn't exceed 15 Mbps (duplex).  Again, this is very conservative, and as you've discovered, your configuration hits the wall at about twice this recommendation, although unfortunately this is not enough to handle your bandwidth capacity.

As the other posters have also noted, long term or preferred solution is probably obtaining and using a faster router.  You probably can get some more capacity out your 892 with some additional "tuning".  I.e. eliminating anything you don't really, really need, and doing what you need as efficiently as possible.  For example, deactivation of NetFlow (as already noted in some posts), deactivation of stateful firewall as you also have NAT/PAT and ACLs; and resequencing (if logically possible) ACEs.

Regarding your question of using policing or shaping, to avoid overrunning the CPU (which really do want to avoid!), yes something can be done there, and it could be very beneficial, but you will use some CPU for that and a really smart approach would be complex.  (An example of "smart" approach would be an embedded script that either polls CPU frequently or traps on high CPU, which then finds high rate flows and dynamically polices those flows slower.  A not-so-smart approach would be a static policer for all inbound traffic, or police just some kinds of inbound traffic.)

[jwillemadhese]

Thanks for your thorough explanation.  I had a hunch cisco spec'ed their routers on the safe side.

I think I found the best solution so far: turning off virtual-assembly.  It's probably a little less secure (the firewall inspection won't work as well on split-up packages), but we're not letting anything in anyway.  And it leaves us way more bandwidth for a given cpu load.

I'll keep your suggestion for the embedded script in mind.  Although it will probably be less troublesome for me to just shape the traffic.

Thanks!

[Leo Laohoo]

It's kind of sad that our ISP recommended a router that can't handle the bandwidth that they provide to us.

Do you have this in writing?  You can get your ISP to REPLACE your router with something that is appropriate with the bandwidth.

You can make it "official" by creating a TAC Case. 

[jwillemadhese]

We have this in writing yes, but we summed the monthly cost over the three year period, and it would cost us way more than buying it ourselves from Amazon.  It was probably not the best decision, since now we also don't have a service contract with cisco or anything (no idea what it would cost though).

So thanks for the suggestion, but it doesn't apply :-)

[darren.g]

Jan Willem wrote:
Hi there,
We have a basic corporate internet connection (100mbit/s down, 10mbit/s up, some guaranties) with eight fixed ip addresses (in the configuration .72 to .79).  To hook up our network to the internet, we are using a cisco 892 router.

Leo is 100% correct. The 890 series router is rated to just over 51 Mbps throughput - which roughly equates to what you're getting. That's half duplex - so, given that most torrents upload at the same time they download, your 48 Mbps download plus maybe 3 Mbps upload for your torrent, and you're pushing the router as fast as it'll go.

If you want to go to the limit of your 100/10 link, you need a bigger router.

Cheers.

[Karsten Iwen]

Another option is to use an ASA instead of the router. The ASA gives you more throuput for your money compared to an IOS-router. And your config doesn't show anything that can't be done with the ASA.

In a similar situation with 100/10 I use the 5505. It's running with a high CPU (up to 80%) from time to time, but it still workes great.

[HumanTorch]

Agreed with you.

An ASA 5506 has 100 Mbps VPN throughput for less price than an ISR 891 which is 50 Mbps or less with VPN.

We have a situation where the customer has Meraki devices and we want to manage their equipment remotely. The Merakis cannot do NAT over VPN so we have to place a router or firewall just for an always on remote management/monitoring of their network using our managed services cloud.

We used the ISR891 for some cases but now considering asa5506 for the better throughput and price point!

[billyray1985]

Hello:

I am observing the following casuistry:
- if the LAN port of the switch module negotiates at 100 Mbps, speeds of 40 Mbps are not exceeded.
- if the LAN port of the switch module negotiates at 1 Gbps, the maximum access speeds (in this case 100 Mbps) are exceeded.

Can it be true?