High CPU usage

[salemmahara]

Hello everybody

Here are some attachments showing a router's cpu usage(1800 series) in my network.

It would be pleasant if we can discuss about it.

What do you think of if? High CPU usage with no heavy process at all !

Why the CPU usage is at the highest level ?

Photo 1 : shows a high cpu usageRouting, Edge Router

photo 2 : shows the same router just a few minutes later !

photo 4 : Same router, another day! Again no heavy process

Note: One of its interfaces is connected to a big broadcast domain. There was input queue drops in this interface but there isn't anymore since I increase input queue . But the cpu usage is the same.

[jalejand]

In your output, you are experiencing High CPU utilization due to interrupts, in this case from the 86% of that total utilization, 84% were interrupts and the remaining 2% were processes.

Interrupts are packets that need to be processed by CPU due to incorrect switching paths, running out of hardware resources or packets that require special handing from CPU like TTL of 1, IP options, Fragmentation, Broadcasts, ACL logging, ICMP redirects, ICMP unreachables, IGMP joins, ARP requests, Control Packets, etc.

A "show ip traffic" output might come in handy.

[salemmahara]

Thank you Jalejand

Here is the output of Show IP Traffic :

What do you think?

CPU usage at the time of show ip traffic : CPU utilization for five seconds: 3%/1%; one minute: 2%; five minutes: 2%

  Rcvd:  480683255 total, 7731371 local destination
         0 format errors, 0 checksum errors, 0 bad hop count
         0 unknown protocol, 0 not a gateway
         0 security failures, 0 bad options, 0 with options
  Opts:  0 end, 0 nop, 0 basic security, 0 loose source route
         0 timestamp, 0 extended security, 0 record route
         0 stream ID, 0 strict source route, 0 alert, 0 cipso, 0 ump
         0 other
  Frags: 0 reassembled, 0 timeouts, 0 couldn't reassemble
         0 fragmented, 0 fragments, 0 couldn't fragment
  Bcast: 7720323 received, 0 sent
  Mcast: 0 received, 0 sent
  Sent:  282911 generated, 472578209 forwarded
  Drop:  57047 encapsulation failed, 0 unresolved, 0 no adjacency
         307312 no route, 0 unicast RPF, 0 forced drop
         0 options denied
  Drop:  0 packets with source IP address zero
  Drop:  0 packets with internal loop back IP address
         9623 physical broadcast

ICMP statistics:
  Rcvd: 0 format errors, 0 checksum errors, 0 redirects, 0 unreachable
        1 echo, 0 echo reply, 0 mask requests, 0 mask replies, 0 quench
        0 parameter, 0 timestamp, 0 info request, 0 other
        0 irdp solicitations, 0 irdp advertisements
  Sent: 0 redirects, 272567 unreachable, 0 echo, 1 echo reply
        0 mask requests, 0 mask replies, 0 quench, 0 timestamp
        0 info reply, 0 time exceeded, 0 parameter problem
        0 irdp solicitations, 0 irdp advertisements

TCP statistics:
  Rcvd: 2286 total, 0 checksum errors, 0 no port
  Sent: 1534 total

IP-EIGRP statistics:
  Rcvd: 0 total
  Sent: 0 total

PIMv2 statistics: Sent/Received
  Total: 0/0, 0 checksum errors, 0 format errors
  Registers: 0/0 (0 non-rp, 0 non-sm-group), Register Stops: 0/0,  Hellos: 0/0
  Join/Prunes: 0/0, Asserts: 0/0, grafts: 0/0
  Bootstraps: 0/0, Candidate_RP_Advertisements: 0/0
  Queue drops: 0
  State-Refresh: 0/0

IGMP statistics: Sent/Received
  Total: 0/0, Format errors: 0/0, Checksum errors: 0/0
  Host Queries: 0/0, Host Reports: 0/0, Host Leaves: 0/0
  DVMRP: 0/0, PIM: 0/0
  Queue drops: 0

UDP statistics:
  Rcvd: 7729823 total, 0 checksum errors, 7720639 no port
  Sent: 8811 total, 0 forwarded broadcasts

OSPF statistics:
  Rcvd: 0 total, 0 checksum errors
        0 hello, 0 database desc, 0 link state req
        0 link state updates, 0 link state acks

  Sent: 0 total
        0 hello, 0 database desc, 0 link state req
        0 link state updates, 0 link state acks

ARP statistics:
  Rcvd: 3871043 requests, 865 replies, 110 reverse, 0 other
  Sent: 23939 requests, 24632 replies (0 proxy), 0 reverse

[jalejand]

Well, it is difficult to determine a root cause for your CPU spikes, the only thing I could suggest is to disable ICMP Unreachables as you are generating quite some on your layer 3 interfaces:

CMP statistics:
Rcvd: 0 format errors, 0 checksum errors, 0 redirects, 0 unreachable
1 echo, 0 echo reply, 0 mask requests, 0 mask replies, 0 quench
0 parameter, 0 timestamp, 0 info request, 0 other
0 irdp solicitations, 0 irdp advertisements
Sent: 0 redirects, 272567 unreachable, 0 echo, 1 echo reply <<<<<<<<<<
0 mask requests, 0 mask replies, 0 quench, 0 timestamp
0 info reply, 0 time exceeded, 0 parameter problem
0 irdp solicitations, 0 irdp advertisements

You can use the "no ip icmp unreachables" under any port you want to disable it.
On the other hand, as we are dealing with interrupt traffic, it would be better to setup a SPAN session on the port facing this router in transmit direction and monitor whenever you experience CPU spikes to stop the capture and then look for any of the previous symptoms I suggested.

For proactive action, although I'm not sure if your device or release supports it, you can implement Control Plane Policing to rate-limit incoming packets to the CPU:

https://www.cisco.com/c/en/us/about/security-center/copp-best-practices.html

[Joseph W. Doherty]

"Interrupts are packets that need to be processed by CPU due to incorrect switching paths . . ."

Yes, that's noted in Cisco's high CPU troubleshooting whitepaper, but I thought interrupts showed CPU usage for "fast path" packet processing too.

For example, in https://www.cisco.com/c/en/us/support/docs/routers/10000-series-routers/15095-highcpu.html#anc7 you find:

"High CPU Utilization due to Interrupts

For more information, refer to Troubleshooting High CPU Utilization Caused by Interrupts. If the level of the CPU rises due to interrupts that are likely due to CEF switching packets, the level of CPU does not affect the router performance."

BTW, when Cisco says the level of CPU doesn't affect router performance, that's true until you run out CPU.

I believe interface input queues are also sometimes a symptom of the CPU not being able to keep up.

BTW, on some of the later ISRs/IOSs, I thought I've noticed fragmentation now "uses" interrupt CPU where in earlier routers it showed its CPU usage also under "IP Process". (NB: To OP, in either case, you do want to avoid fragmentation.)

To OP, an 1800 has a CPU rated at only 70 KPPS (minimum Ethernet sized packets), so it's not difficult for traffic to run the CPU up.

[Georg Pauwen]

Hello,

in addition to 'jalejands' post, can you also post the running configuration of your 1800 router, as well as the output of 'show ver' ?

[salemmahara]

Hello George

Our configuration is secret so I can't send it here. Is there anything special which I can check in configuration? I mean, are you looking for any special commands ?

Its version is 12.4 . We need a contract to download Routers' IOS ( I can't understand why Cisco force us to have contract to download IOS ?? IOS for switches are available however . But it really make us upset because good providers are those who provide services easily).

A lot of customers have troubles related to software version ( Security vulnerabilities , new features , instability issues ) and they can't upgrade their equipment? Why Cisco needs contract? Do they think we use their IOS files on third party hardware ? silly ! I've experienced " sub-optimal " support from Cisco in critical situations where there is a need to a rapid upgrading or downgrading . If you send them an E-mail, they just reply : Could you pleassssse send us contract number and S/N??? I'll post some videos and articles about Cisco's weaknesses in support area.

They want everything legally, but they don't know " Legal means full software support for a specific Hardware." .

They need to know if we have a device or not? Okay, S/N and P/N and anything in SHOW VERSION could be enough.

If we have a device and we don't have an active contract, it means we should throw the device out because the corporation doesn't support its product! What does responsibility mean? it means nothing !

[Georg Pauwen]

Hello,

post the output of 'show ver' so we can see what hardware you have. Since you are running 12.4, which is extremely outdated, my assumption is that you have a 1st generation 1800 router that is end of life/end of sale. If that is the case, your router won't even be able to hold a newer image due to memory.

[salemmahara]

Cisco IOS Software, 1841 Software (C1841-IPBASE-M), Version 12.4(17b), RELEASE SOFTWARE (fc2)

[a.alekseev]

sh run

[salemmahara]

Hi Alek

Is there any special command which you're looking for? Runnig-config is confidential and also as simple as your assumption. some NAT and static Route statements