ASR1000 CPU jump

[jonathan johnson]

Hi,

We have a ASR1001x and I saw a jump in CPU utilization when using the command "show proc cpu"  I know this is showing load on the CPU for management etc, however, does anyone have a reason why we may see a jump from 3-5% to 15%.

From the "show proc cpu" output the following is shown - IP Input looks a bit high when compared to other similar routers that we have.

Any assistance would be appreciated.

Jono

CPU utilization for five seconds: 16%/9%; one minute: 15%; five minutes: 15%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
186 5614669 96096041 58 7.11% 7.72% 7.44% 0 IP Input
123 24310922 887480168 27 0.79% 1.06% 0.96% 0 IOSXE-RP Punt Se
210 39448140 2278960115 0 0.31% 0.15% 0.16% 0 Ethernet Msec Ti
76 16506979 150062684 110 0.23% 0.09% 0.08% 0 IOSD ipc task
134 24201780 1142263493 21 0.15% 0.09% 0.10% 0 HSRP Common
15 17839390 500631944 35 0.15% 0.12% 0.14% 0 ARP Input
611 17973664 908044888 19 0.07% 0.07% 0.07% 0 MFIB_mrib_write
314 2386438 1820454 1310 0.07% 0.01% 0.00% 0 QoS stats proces
619 21983111 582459416 37 0.07% 0.08% 0.08% 0 PPP Events
604 3460827 182052403 19 0.07% 0.00% 0.00% 0 Mwheel Process
595 5000203 1819815 2747 0.07% 0.05% 0.05% 0 MFI LFD Stats Pr
638 4114810 49248526 83 0.07% 0.03% 0.00% 0 BGP Router
288 11723070 269461296 43 0.07% 0.03% 0.05% 0 TCP Timer
180 7568364 284382874 26 0.07% 0.04% 0.02% 0 VRRS Main thread
618 11829229 571563952 20 0.07% 0.04% 0.05% 0 PPP manager
322 19060940 804625799 23 0.07% 0.09% 0.09% 0 L2X Data Daemon
17 0 2 0 0.00% 0.00% 0.00% 0 ATM Idle Timer
18 0 1 0 0.00% 0.00% 0.00% 0 ATM ASYNC PROC
16 1154470 18990152 60 0.00% 0.00% 0.00% 0 ARP Background
20 0 1 0 0.00% 0.00% 0.00% 0 Policy Manager
21 685 105309 6 0.00% 0.00% 0.00% 0 DDR Timers
19 1 38 26 0.00% 0.00% 0.00% 0 AAA_SERVER_DEADT

[Mark Malone]

Looks like you have interrupts , they would casue the cpu to jump 16%/9%

these troubleshooting docs will give you some things to check specifically for those issues

http://www.cisco.com/c/en/us/support/docs/routers/7500-series-routers/41120-highcpu-interrupts.html

http://www.cisco.com/c/en/us/support/docs/routers/7500-series-routers/41160-highcpu-ip-input.html

[jonathan johnson]

Thanks Mark.  I'll have a read.

[jonathan johnson]

I found the cause of the problem but I can't work out how to stop it.  One the sub-interfaces on the router is sending out lots and lots of arp requests.  Below is a capture from the tcpdump on a host on the same vlan (actually apart from the Cisco router, it's the only host...) 192.168.0.3 is my cisco router.

22:29:57.906934 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.0.102 tell 192.168.0.3, length 46
22:29:58.094408 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.0.247 tell 192.168.0.3, length 46
22:29:58.430472 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.0.53 tell 192.168.0.3, length 46

Any ideas on what is causing it?

Thanks

Jono.

[Mark Malone]

sorry slow reply

If you are getting constant arp requests it can be from an infected host on the network a worm/malware etc i wiould run a scan to be sure

have you altered any arp timers ? are the timers flushing quicker than they should be

another thing seen before high inputs due to using interface instead of ip i.e ip route 0.0.0.0 0.0.0.0 g0/0-----use ip as next-hop

also logging an acl can cause high inputs

any STP issues on the local switch side for the vlan

these are just some of the things that can cause it

[jonathan johnson]

Hi Mark,

Thanks for the reply.  

The arp timers haven't changed and I had previously removed logging on some of the ACL's

I don't have any routes point to interfaces.  So it's pointing to an infected host, but, tracking it down will be very difficult.  Are you aware of any debug that will assist in finding it?

All the best,

Jono

[Mark Malone]

unfortunatly no i wouldnt know any debugs for that i would use an actual software scanner for that like mcafee to try an locate it if it is an infcted pc

[jonathan johnson]

ok, no worries.  Thanks for your help on this, I really appreciate it.

Jono

[jonathan johnson]

I've been trying to figure out how to track down the host that's causing the router to spew out arp requests and so far (apart from shutting the sub-interface) I haven't been able to.

Can anyone please offer some avice on how to track it down?

Thanks, 

Jono