High CPU load Cisco router 2901/K9

[renejim11]

Hi fellow network enthousiaste,

I've been breaking my head over this problem the past month now,

Before i explain the situation i'd like to tell you a little about my expierence in this field.

Ik started CCNA course this year since it was included in my classes at the university.

Since passing the first 3 exams  I got a part-time job as network administrator in a small business ( +/- 50 workstations,1 router, 2 switches(1 Cisco) 10+ servers + vpn )

There are no network specialists currently working for this company, so there isn't anyone to turn to with my networking questions

After the configuration of the Router i was confronted with a high CPU load on the Router. I tried the  troubleshooting the problem:

http://www.cisco.com/c/en/us/support/docs/routers/10000-series-routers/15095-highcpu.html

And went on a google spree

The sh proc cpu sorted shows that the process is the IP input process. 

CPU utilization for five seconds: 58%/32%; one minute: 67%; five minutes: 64%
 PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process
 PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process
 117     6571204     15184551          432 24.95% 28.31%27.40%     0 IP Input       
  30         336                          92       3652  0.15%  0.38%  0.09%  390 SSH Process    
 124       13280               2303661          5  0.15%  0.22%  0.23%      0 Ethernet Msec Ti
   2         740        3766        196                  0.07%  0.01%  0.00%      0 Load Meter 

Correct me if I am wrong but how i understand it, this means the Router uses up most cpu for routing traffic.

When i try to debug the router freezes. Furthermore i do not have the luxary to try stuf out that crashes the router since it is in production.

There is a need for PAT (port forwarding in Cisco terms) and NATing for Wan connection.

This is done by an NVI NAT.(ip nat enable)

I tried reshaping the configuration using the old : " nat inside/ outside" commands

This turned out to help on the CPU load, however i could not confirm this since  could not leave the server unacceable for a long time. Also when i switched the NAT method the PAT table seized to work which could have caused this drestic decline in CPU usage.

My question to you network enthousiaste is:

Am i looking in the right direction to solve this problem( could NAT NVI really be the cause of the high cpu load?)

If not , what direction should i be looking?

I would like to thank you for reading this article, and hope i provided the right information.

[Dragan Ilic]

Before going deeper have you looked at this:

http://www.cisco.com/c/en/us/support/docs/routers/7500-series-routers/41160-highcpu-ip-input.html

BR,

Dragan

[renejim11]

Thank you for your replay, but i have looked at that article already

I think i have not made myself completly clear. I have looked all over the internet and tried various ways of troubleshooting, including the ip input truobleshooting page from Cisco. 

The problem i found while troubleshooting the router was that when i tried to debug as the article states, the router crashes. When this accours the router does not save the debugging in the log file, making it impossible for me to get an insight in what is actualy going on.

[Dragan Ilic]

Read document again especially section related to debug "Sample IP packet debugging session" and you will see that you shoud first disable all logging methods and leave only buffer as log destionation...that way you are not going to CPU kill your router 

Try again with this in mind...just follow steps from page I gave you...

HTH,

Dragan

[renejim11]

Hi Dragan,

It might be that i made a mistaking in the debugging proces. Unfortunately i cannot debug the router since it is in production. 

I will try to debug it again tonight. 

Thanks for you respondse,

Rene

[renejim11]

If u dont't mind i would still like to discuss the possability that the virtual interface doing the nat is causing the high cpu usage. 

Here is the output from sh ip nat nvi statistic:

otal active translations: 2511 (24 static, 2487 dynamic; 2511 extended)
NAT Enabled interfaces:
  GigabitEthernet0/0.1, GigabitEthernet0/0.2, GigabitEthernet0/0.3
  GigabitEthernet0/0.4, GigabitEthernet0/0/0
Hits: 143969720  Misses: 179924
CEF Translated packets: 81199480, CEF Punted packets: 165614
Expired translations: 268226
Dynamic mappings:
-- Source [Id: 1] access-list la-list pool la-ip refcount 2412
 pool la-ip: netmask 255.255.255.0
        start 92.111.139.170 end 92.111.139.170
        type generic, total addresses 1, allocated 1 (100%), misses 0

I was wondering if around 2500 translations is a lot to handle for a 2901 Cisco router?

[Dragan Ilic]

How many users is behind that router? Can you give me output of "show interface gi 0/0.." and all subinterfaces?

I think that 2.5k connections is not much for that router...

BR,

Dragan

[renejim11]

Sure,

There are about 50 local user behind the router, and +/- 10 connecting via vpn

its quit a lot tho, we use 2 interfaces 1  with 4 subinterfaces 1 native, 1 for workstations, 1 for the servers and one for wires and 1 interface for WAN

This is the output from the individual interfaces ( its quiet a lot tho)

GigabitEthernet0/0 is up, line protocol is up
  Hardware is CN Gigabit Ethernet, address is e4c7.22d8.1958 (bia e4c7.22d8.1958                                   )
  Description: Internal Network
  MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
     reliability 255/255, txload 2/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full Duplex, 1Gbps, media type is RJ45
  output flow-control is unsupported, input flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:00, output 00:00:00, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/8/18904 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 6231000 bits/sec, 5239 packets/sec
  5 minute output rate 10592000 bits/sec, 5282 packets/sec
     122788086 packets input, 4002740762 bytes, 0 no buffer
     Received 63914 broadcasts (0 IP multicasts)
     0 runts, 0 giants, 1 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 1369 multicast, 0 pause input
     125370654 packets output, 1058799981 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     932 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out
GigabitEthernet0/0.1 is up, line protocol is up
  Hardware is CN Gigabit Ethernet, address is e4c7.22d8.1958 (bia e4c7.22d8.1958                                   )
  Description: native data vlan
  Internet address is 192.x.x.254/24
  MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
     reliability 255/255, txload 2/255, rxload 1/255
  Encapsulation 802.1Q Virtual LAN, Vlan ID  1.
  ARP type: ARPA, ARP Timeout 04:00:00
  Keepalive set (10 sec)
  Last clearing of "show interface" counters never
GigabitEthernet0/0.2 is up, line protocol is up
  Hardware is CN Gigabit Ethernet, address is e4c7.22d8.1958 (bia e4c7.22d8.1958                                   )
  Description: wireless
  Internet address is 192.x.x.1/24
  MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
     reliability 255/255, txload 2/255, rxload 1/255
  Encapsulation 802.1Q Virtual LAN, Vlan ID  2.
  ARP type: ARPA, ARP Timeout 04:00:00
  Keepalive set (10 sec)
  Last clearing of "show interface" counters never
GigabitEthernet0/0.3 is up, line protocol is up
  Hardware is CN Gigabit Ethernet, address is e4c7.22d8.1958 (bia e4c7.22d8.1958                                   )
  Description: workstation (wired)
  Internet address is 192.x.x.1/24
  MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
     reliability 255/255, txload 2/255, rxload 1/255
  Encapsulation 802.1Q Virtual LAN, Vlan ID  3.
  ARP type: ARPA, ARP Timeout 04:00:00
  Keepalive set (10 sec)
  Last clearing of "show interface" counters never
GigabitEthernet0/0.4 is up, line protocol is up
  Hardware is CN Gigabit Ethernet, address is e4c7.22d8.1958 (bia e4c7.22d8.1958                                   )
  Description: server
  Internet address is 192.x.x.1/24
  MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
     reliability 255/255, txload 2/255, rxload 1/255
  Encapsulation 802.1Q Virtual LAN, Vlan ID  4.
  ARP type: ARPA, ARP Timeout 04:00:00
  Keepalive set (10 sec)
  Last clearing of "show interface" counters never

GigabitEthernet0/0/0 is up, line protocol is up
  Hardware is EHWIC-1GE-SFP-CU, 
  Description:  Internet
  Internet address is 92.x.x.170/29
  MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full Duplex, 1Gbps, media type is RJ45
  output flow-control is XON, input flow-control is XON
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:00, output 00:00:00, output hang never
  Last clearing of "show interface" counters never
  Input queue: 1/75/0/108088 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 4012000 bits/sec, 595 packets/sec
  5 minute output rate 864000 bits/sec, 528 packets/sec
     31664617 packets input, 1331044057 bytes, 0 no buffer
     Received 452 broadcasts (0 IP multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 0 multicast, 0 pause input
     27125471 packets output, 4024100849 bytes, 0 underruns
     0 output errors, 0 collisions, 1 interface resets
     0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     1 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out
NVI0 is up, line protocol is up
  Hardware is NVI
  Interface is unnumbered. Using address of GigabitEthernet0/0.1 (192.x.x.254)
  MTU 1514 bytes, BW 56 Kbit/sec, DLY 5000 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation UNKNOWN, loopback not set
  Keepalive set (10 sec)
  Last input never, output never, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     0 packets input, 0 bytes, 0 no buffer
     Received 0 broadcasts (0 IP multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
     0 packets output, 0 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 unknown protocol drops
     0 output buffer failures, 0 output buffers swapped out

This are the stats from the active interfaces

[Dragan Ilic]

Gather debug to see what is happening...maybe it's all fine because you have some traffic there which is not small...maybe 2911/2921 would fit here better than the lowest one - but it should handle that. 

Also - what's your WAN link speed?

BR,

Dragan

[renejim11]

Our current WAN speed is like 50mbit up and 10 mbit down. Our fiber provider went bankrupt and stopped their services. They put a new line in, next week we should be connectied with a proper fiber speed again.

could that be of any influence on the CPU load?

I am hestitant to use any debug commands due to the fact that the router in in production atm i will provide this information when i am able.

Thanks for thinking with me on this issue it is really bugging me 

Rene