High CPU load with IP input on Cisco 881

Greg Maaaag · ‎09-24-2012

Good day!

Here is my conf:

no service pad

service timestamps debug datetime msec

service timestamps log datetime msec

service password-encryption

!

hostname IT

!

boot-start-marker

boot-end-marker

!

enable secret 5 *

!

no aaa new-model

memory-size iomem 10

crypto pki token default removal timeout 0

!

crypto pki trustpoint TP-self-signed-3898150452

enrollment selfsigned

subject-name cn=IOS-Self-Signed-Certificate-3898150452

revocation-check none

!

crypto pki certificate chain TP-self-signed-3898150452

certificate self-signed 01

3082022B 30820194 A0030201 02020101 300D0609 2A864886 F70D0101 04050030

31312F30 2D060355 04031326 494F532D 53656C66 2D536967 6E65642D 43657274

69666963 6174652D 33383938 31353034 3532301E 170D3132 30383231 30383239

34395A17 0D323030 31303130 30303030 305A3031 312F302D 06035504 03132649

4F532D53 656C662D 5369676E 65642D43 65727469 66696361 74652D33 38393831

35303435 3230819F 300D0609 2A864886 F70D0101 01050003 818D0030 81890281

8100948E 9DB523FC 3358CFED 558C315B 662AEB5B 19A4DC22 3E9682B5 CD8A5554

619E9404 C043E3B0 78B63FCB 0088B310 FBF9C7BA 796D79B5 2369FF84 9F82B388

75C986FA B8C3CDE1 0F84FA59 A48C5A45 62648E87 2E777CE8 FBEB7F5E BAE5BABD

8F0CD6A0 B60BABAC 522E0989 1EE0BFA8 7F533FA5 6D443721 2204830A 6BE4541C

B5650203 010001A3 53305130 0F060355 1D130101 FF040530 030101FF 301F0603

551D2304 18301680 14723AD8 E7EBA8F3 60BF92D7 817761D4 D47D5A3F BD301D06

03551D0E 04160414 723AD8E7 EBA8F360 BF92D781 7761D4D4 7D5A3FBD 300D0609

2A864886 F70D0101 04050003 8181002F FDB5515E 7F1A8B8B BBB7EBD4 4C19F32B

7D30060A C7436AD3 8B1778E6 3AD65F5D 3C69785A 1748E814 78246D70 211ACC80

42CC42AE 9D280FED C8C06065 80E22C91 E56D4560 6359AF2E AC497B0D EAF92ECB

0C06D617 F18E9F5A C3F8E5E3 EDD32E18 8520997D 08673F19 339DBDA8 9282200D

C409A8D3 6408E667 42BFFE7C 7D56A2

quit

ip source-route

!

ip dhcp excluded-address 192.168.10.1 192.168.10.99

ip dhcp excluded-address 192.168.10.201 192.168.10.254

!

ip dhcp pool pool

import all

network 192.168.10.0 255.255.255.0

default-router 192.168.10.12

dns-server 192.168.10.100 192.168.240.100

!

ip cef

no ip domain lookup

ip domain name office

ip name-server 8.8.8.8

no ipv6 cef

!

license udi pid CISCO881W-GN-E-K9 sn FCZ1539C4KL

!

username admin privilege 15 secret 5 *

username wifiap privilege 15 secret 5 *

!

ip tcp synwait-time 5

ip ftp source-interface Vlan1

ip ftp username cisco

ip ftp password 7 08305B4B584B56

!

crypto isakmp policy 10

encr aes

hash md5

authentication pre-share

group 2

lifetime 3600

!

crypto isakmp policy 20

encr 3des

authentication pre-share

group 2

lifetime 3600

crypto isakmp key 6 * address 89.*.*.92

crypto isakmp key 6 * address *

!

crypto ipsec transform-set myset esp-aes esp-md5-hmac

crypto ipsec transform-set sharepoint esp-3des esp-sha-hmac

!

crypto map vpn 10 ipsec-isakmp

set peer 89.*.*92

set transform-set myset

match address 102

crypto map vpn 30 ipsec-isakmp

set peer *

set transform-set sharepoint

match address 103

!

interface FastEthernet0

no ip address

!

interface FastEthernet1

no ip address

!

interface FastEthernet2

no ip address

!

interface FastEthernet3

no ip address

!

interface FastEthernet4

ip address 89.*.*.226 255.255.255.240

ip flow ingress

ip nat outside

ip virtual-reassembly in

ip route-cache same-interface

ip route-cache policy

duplex auto

speed auto

crypto map vpn

!

interface wlan-ap0

description Service module interface to manage the embedded AP

ip address 192.168.110.110 255.255.255.0

arp timeout 0

!

interface Wlan-GigabitEthernet0

description Internal switch interface connecting to the embedded AP

switchport mode trunk

no ip address

!

interface Vlan1

description $ETH-SW-LAUNCH$$INTF-INFO-HWIC 4ESW$

ip address 192.168.10.12 255.255.255.0

ip nat inside

ip virtual-reassembly in

!

interface Vlan2

ip address 192.168.25.12 255.255.255.0

ip nat inside

ip virtual-reassembly in

!

ip forward-protocol nd

ip http server

ip http authentication local

ip http secure-server

!

ip nat pool NEW 89.*.*.226 89.*.*.226 prefix-length 24

ip nat pool AD 192.168.10.100 192.168.10.100 netmask 255.255.255.0 type rotary

ip nat inside source list 100 pool NEW overload

ip nat inside source static tcp 192.168.10.4 1723 89.*.*.226 1723 extendable

ip nat inside source static tcp 192.168.10.4 3389 89.*.*.226 3389 extendable

ip nat inside source static tcp 192.168.10.11 8081 89.*.*.226 8081 extendable

ip nat inside destination list 110 pool AD

ip route 0.0.0.0 0.0.0.0 89.*.*.225

ip route 192.168.25.0 255.255.255.0 192.168.10.4

!

access-list 100 deny ip 192.168.10.0 0.0.0.255 192.168.2.0 0.0.0.255

access-list 100 deny ip 192.168.10.0 0.0.0.255 192.168.3.0 0.0.0.255

access-list 100 deny ip 192.168.10.0 0.0.0.255 192.168.12.0 0.0.0.255

access-list 100 deny ip 192.168.10.0 0.0.0.255 192.168.15.0 0.0.0.255

access-list 100 deny ip 192.168.10.0 0.0.0.255 192.168.16.0 0.0.0.255

access-list 100 deny ip 192.168.10.0 0.0.0.255 192.168.17.0 0.0.0.255

access-list 100 deny ip 192.168.10.0 0.0.0.255 192.168.18.0 0.0.0.255

access-list 100 deny ip 192.168.10.0 0.0.0.255 192.168.19.0 0.0.0.255

access-list 100 deny ip 192.168.10.0 0.0.0.255 192.168.25.0 0.0.0.255

access-list 100 deny ip 192.168.10.0 0.0.0.255 192.168.181.0 0.0.0.255

access-list 100 deny ip 192.168.10.0 0.0.0.255 192.168.200.0 0.0.0.255

access-list 100 deny ip 192.168.10.0 0.0.0.255 192.168.240.0 0.0.0.255

access-list 100 deny ip 192.168.10.0 0.0.0.255 192.168.250.0 0.0.0.255

access-list 100 deny ip 192.168.10.0 0.0.0.255 10.0.1.0 0.0.0.255

access-list 100 permit ip 192.168.10.0 0.0.0.255 any

access-list 102 permit ip 192.168.10.0 0.0.0.255 192.168.2.0 0.0.0.255

access-list 102 permit ip 192.168.10.0 0.0.0.255 192.168.3.0 0.0.0.255

access-list 102 permit ip 192.168.10.0 0.0.0.255 192.168.12.0 0.0.0.255

access-list 102 permit ip 192.168.10.0 0.0.0.255 192.168.15.0 0.0.0.255

access-list 102 permit ip 192.168.10.0 0.0.0.255 192.168.16.0 0.0.0.255

access-list 102 permit ip 192.168.10.0 0.0.0.255 192.168.17.0 0.0.0.255

access-list 102 permit ip 192.168.10.0 0.0.0.255 192.168.18.0 0.0.0.255

access-list 102 permit ip 192.168.10.0 0.0.0.255 192.168.19.0 0.0.0.255

access-list 102 permit ip 192.168.10.0 0.0.0.255 192.168.181.0 0.0.0.255

access-list 102 permit ip 192.168.10.0 0.0.0.255 192.168.200.0 0.0.0.255

access-list 102 permit ip 192.168.10.0 0.0.0.255 192.168.240.0 0.0.0.255

access-list 102 permit ip 192.168.10.0 0.0.0.255 192.168.250.0 0.0.0.255

access-list 102 permit ip 192.168.10.0 0.0.0.255 10.0.1.0 0.0.0.255

access-list 103 permit ip 192.168.10.0 0.0.0.255 192.168.40.0 0.0.0.255

access-list 110 permit tcp any any range 3268 3269

access-list 110 permit tcp any any eq 389

access-list 110 permit udp any any eq 389

access-list 110 permit udp any any eq 636

access-list 110 permit tcp any any eq 636

no cdp run

!

route-map SHAREPOINT permit 20

match ip address 103

set interface FastEthernet4

!

route-map IPSEC-TRAF permit 10

match ip address 102

set interface FastEthernet4

!

line con 0

exec-timeout 30 30

privilege level 15

password 7 060506324F41584B56

logging synchronous

login

line aux 0

line 2

no activation-character

no exec

transport preferred none

transport input all

line vty 0 4

privilege level 15

password 7 072871551A0B2B5241

login local

transport input all

!

end

show proc cpu sorted 1

CPU utilization for five seconds: 99%/56%; one minute: 99%; five minutes: 99%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

106 12469004 635891 19608 39.23% 38.88% 42.83% 0 IP Input

82 443152 42173 10507 1.11% 1.27% 1.45% 0 COLLECT STAT COU

80 277664 49943 5559 1.19% 1.09% 1.10% 0 i2c sm_exchange

79 75788 128397 590 0.47% 0.46% 0.24% 0 SEC BATCH

show interfaces FastEthernet 4

FastEthernet4 is up, line protocol is up

Hardware is PQII_PRO_UEC, address is 7081.05a0.97be (bia 7081.05a0.97be)

Internet address is 89.104.102.226/28

MTU 1500 bytes, BW 100000 Kbit/sec, DLY 100 usec,

reliability 255/255, txload 30/255, rxload 41/255

Encapsulation ARPA, loopback not set

Keepalive set (10 sec)

Full-duplex, 100Mb/s, 100BaseTX/FX

ARP type: ARPA, ARP Timeout 04:00:00

Last input 00:00:00, output 00:00:00, output hang never

Last clearing of "show interface" counters never

Input queue: 30/75/14250/0 (size/max/drops/flushes); Total output drops: 0

Queueing strategy: fifo

Output queue: 0/40 (size/max)

5 minute input rate 16245000 bits/sec, 1946 packets/sec

5 minute output rate 12003000 bits/sec, 1907 packets/sec

29306702 packets input, 1552528471 bytes

Received 37493 broadcasts (0 IP multicasts)

867 runts, 0 giants, 4348 throttles

4746 input errors, 0 CRC, 0 frame, 4585 overrun, 161 ignored

0 watchdog

0 input packets with dribble condition detected

27453647 packets output, 1824988727 bytes, 0 underruns

0 output errors, 0 collisions, 2 interface resets

12353 unknown protocol drops

0 babbles, 0 late collision, 0 deferred

0 lost carrier, 0 no carrier

0 output buffer failures, 0 output buffers swapped out

show interfaces FastEthernet 4 switching

FastEthernet4

Throttle count 4383

Drops RP 14301 SP 0

SPD Flushes Fast 0 SSE 0

SPD Aggress Fast 0

SPD Priority Inputs 1233 Drops 0

Protocol IP

Switching path Pkts In Chars In Pkts Out Chars Out

Process 16185388 1276887927 9057611 596342143

Cache misses 0 - - -

Fast 13439853 699219156 18687667 1399098080

Auton/SSE 0 0 0 0

Protocol ARP

Switching path Pkts In Chars In Pkts Out Chars Out

Process 23934 1446298 72 4320

Cache misses 0 - - -

Fast 0 0 0 0

Auton/SSE 0 0 0 0

Protocol Other

Switching path Pkts In Chars In Pkts Out Chars Out

Process 12479 1731848 0 0

Cache misses 0 - - -

Fast 0 0 0 0

Auton/SSE 0 0 0 0

How can i prevent it?

skarthic · ‎09-24-2012

Hi Greg,

As per the router performance document, you might be seeing a platform limitation. Best way to confirm this would be to check with TAC.

http://www.cisco.com/web/partners/downloads/765/tools/quickreference/routerperformance.pdf

I do see a lot of process switching happening too which is not the ideal method of packet switching.

CPU utilization for five seconds: 99%/56%; one minute: 99%; five minutes: 99%

56% due to interrupts

43% due to processes out of which 40% is due to "IP Input" which is related to process switching.

Can you check "show ip cef switching statistics feature" to see which feature is causing the packets to get process switched?

The config mostly looks fine.

Thanks,
Karthic

.

Greg Maaaag · ‎10-01-2012

Thank you for your answer!

Can you please tell, if config mostly looks fine, what isn't fine in it?:)

#show ip cef switching statistics feature

IPv4 CEF input features:

Feature Drop Consume Punt Punt2Host Gave route

Virtual Fragment 30573 2482032 0 0 0

IPSec input clas 3808 233310808 0 13743 0

NAT Outside 5 0 0 448269286 0

Total 34386 235792840 0 448283029 0

IPv4 CEF output features:

Feature Drop Consume Punt Punt2Host New i/f

Post-routing NAT 2 0 0 248475585 0

IPSec output cla 1995 0 0 0 0

IPSec: to crypto 0 159380476 0 0 0

Total 1997 159380476 0 248475585 0

IPv4 CEF post-encap features:

Feature Drop Consume Punt Punt2Host New i/f

Total 0 0 0 0 0

IPv4 CEF for us features:

Feature Drop Consume Punt Punt2Host New i/f

Total 0 0 0 0 0

IPv4 CEF punt features:

Feature Drop Consume Punt Punt2Host New i/f

Total 0 0 0 0 0

IPv4 CEF local features:

Feature Drop Consume Punt Punt2Host Gave route

Total 0 0 0 0 0

and

show ip traffic

IP statistics:

Rcvd: 709394331 total, 7659030 local destination

0 format errors, 0 checksum errors, 102917 bad hop count

0 unknown protocol, 3 not a gateway

0 security failures, 0 bad options, 5189 with options

Opts: 0 end, 0 nop, 0 basic security, 0 loose source route

0 timestamp, 0 extended security, 0 record route

0 stream ID, 0 strict source route, 5189 alert, 0 cipso, 0 ump

0 other

Frags: 13625 reassembled, 8 timeouts, 0 couldn't reassemble

10233 fragmented, 20468 fragments, 29 couldn't fragment

Bcast: 513555 received, 3876 sent

Mcast: 0 received, 0 sent

Sent: 4930701 generated, 1333377882 forwarded

Drop: 151 encapsulation failed, 0 unresolved, 0 no adjacency

499 no route, 0 unicast RPF, 0 forced drop

0 options denied

Drop: 0 packets with source IP address zero

Drop: 0 packets with internal loop back IP address

12237 physical broadcast

Reinj: 0 in input feature path, 46 in output feature path

ICMP statistics:

Rcvd: 39 format errors, 0 checksum errors, 0 redirects, 2467 unreachable

2071 echo, 7 echo reply, 0 mask requests, 0 mask replies, 1 quench

0 parameter, 0 timestamp, 0 timestamp replies, 0 info request, 0 other

0 irdp solicitations, 0 irdp advertisements

16 time exceeded, 0 info replies

Sent: 0 redirects, 327552 unreachable, 98 echo, 2071 echo reply

0 mask requests, 0 mask replies, 0 quench, 0 timestamp, 0 timestamp repl ies

0 info reply, 47 time exceeded, 0 parameter problem

0 irdp solicitations, 0 irdp advertisements

TCP statistics:

Rcvd: 6654094 total, 28 checksum errors, 12701 no port

Sent: 4591011 total

PIMv2 statistics: Sent/Received

Total: 0/0, 0 checksum errors, 0 format errors

Registers: 0/0 (0 non-rp, 0 non-sm-group), Register Stops: 0/0, Hellos: 0/0

Join/Prunes: 0/0, Asserts: 0/0, grafts: 0/0

Bootstraps: 0/0, Candidate_RP_Advertisements: 0/0

Queue drops: 0

State-Refresh: 0/0

IGMP statistics: Sent/Received

Total: 0/0, Format errors: 0/0, Checksum errors: 0/0

Host Queries: 0/0, Host Reports: 0/0, Host Leaves: 0/0

DVMRP: 0/0, PIM: 0/0

Queue drops: 0

UDP statistics:

Rcvd: 996430 total, 0 checksum errors, 979398 no port

Sent: 9959 total, 0 forwarded broadcasts

OSPF statistics:

Last clearing of OSPF traffic counters never

Rcvd: 0 total, 0 checksum errors

0 hello, 0 database desc, 0 link state req

0 link state updates, 0 link state acks

Sent: 0 total

0 hello, 0 database desc, 0 link state req

0 link state updates, 0 link state acks

ARP statistics:

Rcvd: 1778041 requests, 187 replies, 17226 reverse, 0 other

Sent: 490798 requests, 63567 replies (1025 proxy), 0 reverse

Drop due to input queue full: 0

I see too many incoming ARP requests and many bad hoop count...

As you see, we've got vpn tunnels with many subnets, can it be connected with it?)

About external brandwith, it's 20mb\s so i guess it isn't limit for 881w )

paolo bevilacqua · ‎10-02-2012

I would start with 'no ip reassemly" and summarizing the ACL entries.

Greg Maaaag · ‎10-02-2012

Summarazing like 192.168.0.0\16 ?

'no ip reassemly' this command isn't found

Joseph W. Doherty · ‎10-02-2012

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

Summarazing like 192.168.0.0\16 ?

e.g.

access-list 100 deny ip 192.168.10.0 0.0.0.255 192.168.16.0 0.0.0.255

access-list 100 deny ip 192.168.10.0 0.0.0.255 192.168.17.0 0.0.0.255

access-list 100 deny ip 192.168.10.0 0.0.0.255 192.168.18.0 0.0.0.255

access-list 100 deny ip 192.168.10.0 0.0.0.255 192.168.19.0 0.0.0.255

can be

access-list 100 deny ip 192.168.10.0 0.0.0.255 192.168.16.0 0.0.3.255

paolo bevilacqua · ‎10-02-2012

Greg Maaaag wrote:

Summarazing like 192.168.0.0\16 ?

'no ip reassemly' this command isn't found

Typo. no ip virtual-reassembly.