06-14-2012 06:35 AM - edited 03-07-2019 07:15 AM
Hi there,
We have a basic corporate internet connection (100mbit/s down, 10mbit/s up, some guaranties) with eight fixed ip addresses (in the configuration .72 to .79). To hook up our network to the internet, we are using a cisco 892 router.
I am new to cisco equipment, so I had some trouble getting things to work, but I got there.
Now we are facing this bandwith limitation problem: whenever we are using more than about 60mbit/s down bandwidth, the cpu of this router is maxing out (98 - 99% with 'show processes cpu history').
When I'm downloading a torrent (debian dvd) at 4MB/s (or 48mbit/s) cpu is running at around 46% (tops at 49%). Stopping the download results in 14% cpu usage tops.
When using the 'show processes cpu sorted' command, I get this:
maximilian#show processes cpu sorted
CPU utilization for five seconds: 46%/43%; one minute: 33%; five minutes: 15%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
82 35978164 10389766 3462 2.31% 2.08% 2.06% 0 COLLECT STAT COU
90 203264 266300211 0 0.31% 0.31% 0.28% 0 Ethernet Msec Ti
31 664 844 786 0.31% 0.19% 0.08% 8 SSH Process
108 2039472 5100352 399 0.23% 0.34% 0.30% 0 IP Input
334 35468 4269539 8 0.23% 0.09% 0.04% 0 IP NAT Ager
324 21204 2077221 10 0.07% 0.03% 0.02% 0 Per-Second Jobs
104 24240 64783802 0 0.07% 0.04% 0.02% 0 IPAM Manager
336 7404 108003 68 0.07% 0.02% 0.00% 0 IP VFR proc
33 66952 321851 208 0.07% 0.00% 0.00% 0 ARP Input
9 0 2 0 0.00% 0.00% 0.00% 0 Timers
...
So cpu utilization is around 46%, while no process is actually using more than 2.31%. Also, these numbers don't change if I stop the download.
This is our configuration (with some parts obscured):
maximilian#show run
Building configuration...
Current configuration : 8035 bytes
!
! Last configuration change at 09:49:27 UTC Wed May 30 2012 by jan
! NVRAM config last updated at 13:55:40 UTC Tue May 22 2012 by jan
! NVRAM config last updated at 13:55:40 UTC Tue May 22 2012 by jan
version 15.2
service timestamps debug datetime msec
service timestamps log datetime msec
no service password-encryption
!
hostname maximilian
!
boot-start-marker
boot config flash:maxi-config
boot-end-marker
!
!
logging buffered 51200 warnings
!
no aaa new-model
!
crypto pki token default removal timeout 0
!
crypto pki trustpoint TP-self-signed-3260749506
enrollment selfsigned
subject-name cn=IOS-Self-Signed-Certificate-3260749506
revocation-check none
rsakeypair TP-self-signed-3260749506
!
crypto pki trustpoint tti
revocation-check crl
!
!
crypto pki certificate chain TP-self-signed-3260749506
certificate self-signed 01
...........[snip].....
quit
crypto pki certificate chain tti
!
!
!
ip dhcp excluded-address 10.10.10.1
ip dhcp excluded-address 192.168.1.0 192.168.1.49
!
ip dhcp pool cvo-pool
import all
network 10.10.10.0 255.255.255.248
default-router 10.10.10.1
dns-server 8.8.8.8
lease 0 2
!
ip dhcp pool maxi-pool
import all
network 192.168.1.0 255.255.255.0
default-router 192.168.1.1
dns-server 8.8.8.8
lease infinite
!
!
ip domain name adhese.org
ip name-server 8.8.8.8
ip inspect name DEFAULT100 ftp
ip inspect name DEFAULT100 h323
ip inspect name DEFAULT100 icmp
ip inspect name DEFAULT100 netshow
ip inspect name DEFAULT100 rcmd
ip inspect name DEFAULT100 realaudio
ip inspect name DEFAULT100 rtsp
ip inspect name DEFAULT100 esmtp
ip inspect name DEFAULT100 sqlnet
ip inspect name DEFAULT100 streamworks
ip inspect name DEFAULT100 tftp
ip inspect name DEFAULT100 tcp
ip inspect name DEFAULT100 udp
ip inspect name DEFAULT100 vdolive
ip cef
no ipv6 cef
!
!
!
!
multilink bundle-name authenticated
!
!
!
!
!
!
license udi pid CISCO892-K9 sn FC[snip]
!
!
username cisco privilege 15 secret 5 [snip]
username jan privilege 15
!
!
!
!
!
ip ssh pubkey-chain
username jan
key-hash ssh-rsa [snip] jan@[snip]
quit
!
!
!
!
!
!
!
!
!
interface BRI0
no ip address
encapsulation hdlc
shutdown
isdn termination multidrop
!
interface FastEthernet0
no ip address
spanning-tree portfast
!
interface FastEthernet1
no ip address
spanning-tree portfast
!
interface FastEthernet2
switchport access vlan 2
no ip address
spanning-tree portfast
!
interface FastEthernet3
no ip address
spanning-tree portfast
!
interface FastEthernet4
no ip address
spanning-tree portfast
!
interface FastEthernet5
no ip address
spanning-tree portfast
!
interface FastEthernet6
switchport access vlan 2
no ip address
spanning-tree portfast
!
interface FastEthernet7
switchport access vlan 2
no ip address
spanning-tree portfast
!
interface FastEthernet8
ip address 192.168.3.2 255.255.255.0
duplex auto
speed auto
!
interface GigabitEthernet0
ip address [snip].94 255.255.255.252
ip access-group 101 in
no ip redirects
no ip unreachables
no ip proxy-arp
ip flow ingress
ip nat outside
ip inspect DEFAULT100 out
ip virtual-reassembly in
duplex auto
speed auto
!
interface Vlan1
ip address 10.10.10.1 255.255.255.248
ip access-group 100 in
ip nat inside
ip virtual-reassembly in
ip tcp adjust-mss 1452
!
interface Vlan2
ip address 192.168.1.1 255.255.255.0
ip flow ingress
ip nat inside
ip virtual-reassembly in
!
ip forward-protocol nd
!
ip flow-capture fragment-offset
ip flow-capture packet-length
ip flow-capture ttl
ip flow-capture vlan-id
ip flow-capture icmp
ip flow-capture ip-id
ip flow-capture mac-addresses
!
ip http server
ip http access-class 23
ip http authentication local
ip http secure-server
ip http timeout-policy idle 60 life 86400 requests 10000
ip nat pool public3-74 [snip].74 [snip].74 prefix-length 29
ip nat inside source list 74 pool public3-74 overload
ip nat inside source static tcp 192.168.1.42 22 [snip].72 22 extendable
ip nat inside source static udp 192.168.1.42 1194 [snip].72 1194 extendable
ip nat inside source static tcp 192.168.1.42 22 [snip].72 1489 extendable
ip route 0.0.0.0 0.0.0.0 [snip].93
ip route 0.0.0.0 0.0.0.0 192.168.3.1 5
ip route 10.8.0.0 255.255.255.0 192.168.1.42
!
access-list 1 permit 10.10.10.0 0.0.0.7
access-list 23 permit 192.168.1.42
access-list 23 permit 10.10.10.0 0.0.0.7
access-list 74 permit 10.10.10.0 0.0.0.7
access-list 74 permit 192.168.1.0 0.0.0.255
access-list 100 deny ip host 255.255.255.255 any
access-list 100 deny ip 127.0.0.0 0.255.255.255 any
access-list 100 permit ip any any
access-list 101 deny ip 10.0.0.0 0.255.255.255 any
access-list 101 deny ip 172.16.0.0 0.15.255.255 any
access-list 101 deny ip 192.168.0.0 0.0.255.255 any
access-list 101 deny ip 127.0.0.0 0.255.255.255 any
access-list 101 deny ip host 255.255.255.255 any
access-list 101 permit tcp any host [snip].72 eq 1489
access-list 101 permit tcp any host [snip].72 eq 22
access-list 101 permit udp any host [snip].72 eq 1194
access-list 199 deny ip any host 74.209.133.138
access-list 199 permit ip any any
no cdp run
!
!
!
!
!
snmp-server community [snip] RO
!
control-plane
!
!
!
!
mgcp profile default
!
!
!
!
banner exec ^C
% Password expiration warning.
-----------------------------------------------------------------------
Cisco Virtual Office (CVO) is installed on this device and it provides the
default username "cisco".
It is strongly suggested that you create a new username with a privilege level
of 15 using the following command.
username <myuser> privilege 15 secret 0 <mypassword>
Replace <myuser> and <mypassword> with the username and password you want to
use.
-----------------------------------------------------------------------
^C
banner login ^C
-----------------------------------------------------------------------
Cisco Virtual Office (CVO) is installed on this device and it provides the
default username "cisco".
It is strongly suggested that you create a new username with a privilege level
of 15 using the following command.
username <myuser> privilege 15 secret 0 <mypassword>
Replace <myuser> and <mypassword> with the username and password you want to
use.
For more information about CVO please go to http://www.cisco.com/go/cvo
-----------------------------------------------------------------------
^C
!
line con 0
login local
line aux 0
line vty 0 4
access-class 23 in
login local
length 0
transport input telnet ssh
line vty 5 15
access-class 23 in
privilege level 15
login local
transport input telnet ssh
!
end
Please tell me what I can do about this. Or is this router not capable of doing 100mbit/s?
The net effect of running out of cpu is random connections dropping and no communication possible with the router. I could cap the bandwidth at 90mbit/s or 80mbit/s, but I'd rather not.
The system image file is: "flash:c890-universalk9-mz.152-1.T1.bin"
Thanks in advance!
Jan.
Solved! Go to Solution.
06-28-2012 10:08 AM
Disclaimer
The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.
Liability Disclaimer
In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.
Posting
Actually, the 890 series is rated at 100 Kpps, which for minimum size Ethernet packets is about 51 Mbps (as also noted in other posts) but Cisco also documents the 890 providing up to 1,400 Mbps for 1500 byte size packets. Unfortunately "your mileage may vary"; i.e. actual throughput is very, very dependent on your particular traffic and what you configure your router to do against that traffic. For example, you have NAT/PAT, interface ACLs, firewall inspection and NetFlow, all which consume additional CPU while processing packets.
Not knowing what exactly a customer will do with a router, Cisco makes very conservative usage recommendations, and for the 890, it recommends the WAN side doesn't exceed 15 Mbps (duplex). Again, this is very conservative, and as you've discovered, your configuration hits the wall at about twice this recommendation, although unfortunately this is not enough to handle your bandwidth capacity.
As the other posters have also noted, long term or preferred solution is probably obtaining and using a faster router. You probably can get some more capacity out your 892 with some additional "tuning". I.e. eliminating anything you don't really, really need, and doing what you need as efficiently as possible. For example, deactivation of NetFlow (as already noted in some posts), deactivation of stateful firewall as you also have NAT/PAT and ACLs; and resequencing (if logically possible) ACEs.
Regarding your question of using policing or shaping, to avoid overrunning the CPU (which really do want to avoid!), yes something can be done there, and it could be very beneficial, but you will use some CPU for that and a really smart approach would be complex. (An example of "smart" approach would be an embedded script that either polls CPU frequently or traps on high CPU, which then finds high rate flows and dynamically polices those flows slower. A not-so-smart approach would be a static policer for all inbound traffic, or police just some kinds of inbound traffic.)
06-14-2012 07:54 AM
If no necessary, you can try to disable ip inspect and ip flow.
You can also try to upgrade the IOS.
But I think in no better performance than your result:
http://www.cisco.com/web/AP/partners/ANZ_PE/borderless_network/Cisco_routerperformance.pdf
Regards.
06-14-2012 08:25 AM
Hi Daniele,
Thank you for your quick reply!
If I understand correctly, we need ip inspect to bypass the implicit deny ip any any of access-list 101, or else any incoming packet will be dropped? For sure, we only need to inspect tcp, udp, ftp and icmp.
We turned on ip flow to be able to track which user was maxing out the router. Is there a more cpu-friendly way to do this? I once enabled mac accounting and ip accounting, and after 5 minutes the router just crashed...
I'd like to upgrade the IOS, but we have no subscription. We were unaware of the cisco ways, and just bought the router on amazon..
If I read the performance data correcty, the 892 can handle 100,000 64-byte packets per second, resulting in 51.20 Mbps. I would assume the packets in our download are MTU-sized, and I'd expect the router doing easily more than 100 Mbps then?
Regards,
Jan.
06-14-2012 08:46 AM
Hi Daniele,
I now turned off ip flow like you suggested, I think it got the cpu usage a little lower. But I can stil easily max out the cpu with a simple download.
I could not turn off ip inspect (even without access-list 101), and I did trim down to tcp, udp, ftp, icmp, tftp and esmtp. I don't see much difference though with this last change.
Regards,
Jan.
06-14-2012 09:10 PM
We have a basic corporate internet connection (100mbit/s down, 10mbit/s up, some guaranties) with eight fixed ip addresses (in the configuration .72 to .79). To hook up our network to the internet, we are using a cisco 892 router.
The 890 will not be able to handle traffic beyond 51.02 Mbps (half duplex and no encryption).
06-28-2012 06:01 AM
Thanks. It's kind of sad that our ISP recommended a router that can't handle the bandwidth that they provide to us. But then again, I guess the next step up cisco router is probably more than twice as expensive.
I was wondering though: when the router is flooded with more traffic than it can handle, the result is a failing network for all users. What are my options to handle this situation?
I'd prefer to just drop (or maybe shape) traffic that can't be handled because of too high cpu usage. However, I think the only option is to shape on a fixed bandwidth instead of on cpu usage? Or will the router do something better if I define a quality of service strategy? Better as in: cpu maxes out, but only low priority traffic is affected.
06-28-2012 10:08 AM
Disclaimer
The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.
Liability Disclaimer
In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.
Posting
Actually, the 890 series is rated at 100 Kpps, which for minimum size Ethernet packets is about 51 Mbps (as also noted in other posts) but Cisco also documents the 890 providing up to 1,400 Mbps for 1500 byte size packets. Unfortunately "your mileage may vary"; i.e. actual throughput is very, very dependent on your particular traffic and what you configure your router to do against that traffic. For example, you have NAT/PAT, interface ACLs, firewall inspection and NetFlow, all which consume additional CPU while processing packets.
Not knowing what exactly a customer will do with a router, Cisco makes very conservative usage recommendations, and for the 890, it recommends the WAN side doesn't exceed 15 Mbps (duplex). Again, this is very conservative, and as you've discovered, your configuration hits the wall at about twice this recommendation, although unfortunately this is not enough to handle your bandwidth capacity.
As the other posters have also noted, long term or preferred solution is probably obtaining and using a faster router. You probably can get some more capacity out your 892 with some additional "tuning". I.e. eliminating anything you don't really, really need, and doing what you need as efficiently as possible. For example, deactivation of NetFlow (as already noted in some posts), deactivation of stateful firewall as you also have NAT/PAT and ACLs; and resequencing (if logically possible) ACEs.
Regarding your question of using policing or shaping, to avoid overrunning the CPU (which really do want to avoid!), yes something can be done there, and it could be very beneficial, but you will use some CPU for that and a really smart approach would be complex. (An example of "smart" approach would be an embedded script that either polls CPU frequently or traps on high CPU, which then finds high rate flows and dynamically polices those flows slower. A not-so-smart approach would be a static policer for all inbound traffic, or police just some kinds of inbound traffic.)
07-19-2012 01:33 AM
Thanks for your thorough explanation. I had a hunch cisco spec'ed their routers on the safe side.
I think I found the best solution so far: turning off virtual-assembly. It's probably a little less secure (the firewall inspection won't work as well on split-up packages), but we're not letting anything in anyway. And it leaves us way more bandwidth for a given cpu load.
I'll keep your suggestion for the embedded script in mind. Although it will probably be less troublesome for me to just shape the traffic.
Thanks!
06-28-2012 03:42 PM
It's kind of sad that our ISP recommended a router that can't handle the bandwidth that they provide to us.
Do you have this in writing? You can get your ISP to REPLACE your router with something that is appropriate with the bandwidth.
You can make it "official" by creating a TAC Case.
07-19-2012 01:24 AM
We have this in writing yes, but we summed the monthly cost over the three year period, and it would cost us way more than buying it ourselves from Amazon. It was probably not the best decision, since now we also don't have a service contract with cisco or anything (no idea what it would cost though).
So thanks for the suggestion, but it doesn't apply :-)
06-14-2012 10:16 PM
Jan Willem wrote:
Hi there,
We have a basic corporate internet connection (100mbit/s down, 10mbit/s up, some guaranties) with eight fixed ip addresses (in the configuration .72 to .79). To hook up our network to the internet, we are using a cisco 892 router.
Leo is 100% correct. The 890 series router is rated to just over 51 Mbps throughput - which roughly equates to what you're getting. That's half duplex - so, given that most torrents upload at the same time they download, your 48 Mbps download plus maybe 3 Mbps upload for your torrent, and you're pushing the router as fast as it'll go.
If you want to go to the limit of your 100/10 link, you need a bigger router.
Cheers.
06-29-2012 12:55 AM
Another option is to use an ASA instead of the router. The ASA gives you more throuput for your money compared to an IOS-router. And your config doesn't show anything that can't be done with the ASA.
In a similar situation with 100/10 I use the 5505. It's running with a high CPU (up to 80%) from time to time, but it still workes great.
11-07-2016 09:30 AM
Agreed with you.
An ASA 5506 has 100 Mbps VPN throughput for less price than an ISR 891 which is 50 Mbps or less with VPN.
We have a situation where the customer has Meraki devices and we want to manage their equipment remotely. The Merakis cannot do NAT over VPN so we have to place a router or firewall just for an always on remote management/monitoring of their network using our managed services cloud.
We used the ISR891 for some cases but now considering asa5506 for the better throughput and price point!
05-25-2018 04:56 AM
Hello:
I am observing the following casuistry:
- if the LAN port of the switch module negotiates at 100 Mbps, speeds of 40 Mbps are not exceeded.
- if the LAN port of the switch module negotiates at 1 Gbps, the maximum access speeds (in this case 100 Mbps) are exceeded.
Can it be true?
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide