Solved: High CPU Utilization on 7606 - Page 2

sabuz_banik · ‎09-29-2012

Dear concern,

Our Router 7606 is in a problem with high CPU Utilization. We are using one EBGP and one IBGP session. We are also running OSPF with area 0 between two IBGP router to exchange few local networks. When bandwidth cross 100 MB than only CPU use become 100%. when bandwidth consumption is 80MB CPU use become 80% end so on.

Pls find the bellow out put and assist us.

----------------------------------------------------------------------------------------------------------------------

Out put is when bandwidth consumption is 34MB

----------------------------------------------------------------------------------------------------------------------

CPU utilization for five seconds: 44%/43%; one minute: 42%; five minutes: 30%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

78 44 9625 4 0.39% 0.03% 0.00% 2 Virtual Exec

13 40 375 106 0.07% 0.00% 0.00% 0 ARP Input

30 116 1098 105 0.07% 0.01% 0.00% 0 IPC Seat Manager

164 524 42660 12 0.07% 0.04% 0.05% 1 Virtual Exec

186 0 22945 0 0.07% 0.03% 0.02% 0 ACE Tunnel Task

259 12 89401 0 0.07% 0.10% 0.08% 0 Ethernet Msec Ti

1 0 8 0 0.00% 0.00% 0.00% 0 Chunk Manager

2 4 156 25 0.00% 0.01% 0.00% 0 Load Meter

4 0 47 0 0.00% 0.00% 0.00% 0 SCP async: Draco

5 0 173 0 0.00% 0.00% 0.00% 0 Retransmission o

6 0 3 0 0.00% 0.00% 0.00% 0 IPC ISSU Dispatc

7 0 1 0 0.00% 0.00% 0.00% 0 PF Redun ICC Req

8 8684 466 18635 0.00% 0.86% 0.81% 0 Check heaps

9 0 1 0 0.00% 0.00% 0.00% 0 Pool Manager

10 0 1 0 0.00% 0.00% 0.00% 0 DiscardQ Backgro

11 0 2 0 0.00% 0.00% 0.00% 0 Timers

12 4 226 17 0.00% 0.00% 0.00% 0 WATCH_AFS

14 8 853 9 0.00% 0.00% 0.00% 0 ARP Background

15 0 2 0 0.00% 0.00% 0.00% 0 ATM Idle Timer

16 0 1 0 0.00% 0.00% 0.00% 0 ATM ASYNC PROC

End = e Freeze = f

Enter Command:

-------------------------------------------------------------------------------------------------------------------------------

RTLSGR1#sh memory statistics

Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)

Processor 1322CA94 1692218732 522133152 1170085580 1168835904 1169364236

I/O 78000000 134217728 50319772 83897956 83897956 83896252

---------------------------------------------------------------------------------------------------------------------------------------

RTLSGR1#sh cef drop

% Command accepted but obsolete, see 'show (ip|ipv6) cef switching statistics [feature]'

IPv4 CEF Drop Statistics

Slot Encap_fail Unresolved Unsupported No_route No_adj ChkSum_Err

RP 0 0 422 10 4 0

5/0 0 0 0 0 0 0

3/0 0 0 0 0 0 0

RTLSGR1#

-------------------------------------------------------------------------------------------------------------

SUP

RSP720-3C-GE

--------------------------------------------------------------------------------------------------------------------

IOS

c7600rsp72043_rp-ADVENTERPRISEK9-M

---------------------------------------------------------------------------------------------------------------------

Regards

Raju Sekharan · ‎10-01-2012

Here is the issue

"***** FIB is in exception state for IPv4 unicast *****"

You FIB has gone into exception state. This could be due to some flood of prefixes in the past.

To check the current FIB TCAM usage, you can try the command " show platform hardware capacity forwarding"

To recover from this, we need to Reload the router.

Afte reload check the status using " show mls cef exception status"

sabuz_banik · ‎10-01-2012

I thoougt it was an IOS bug and replaced the IOS and restarted once but found same problem. Do i need to go with hard reboot?

RTLSGR1#sh platform hardware capacity forwarding

L2 Forwarding Resources

MAC Table usage: Module Collisions Total Used %Used

5 0 98304 6 1%

VPN CAM usage: Total Used %Used

512 0 0%

L3 Forwarding Resources

Module FIB TCAM usage: Total Used %Used

5 72 bits (IPv4, MPLS, EoM) 196608 196194 100%

144 bits (IP mcast, IPv6) 32768 9 1%

detail: Protocol Used %Used

IPv4 194146 99%

MPLS 2048 1%

EoM 0 0%

IPv6 2 1%

IPv4 mcast 4 1%

IPv6 mcast 3 1%

Adjacency usage: Total Used %Used

1048576 217 1%

Forwarding engine load:

Module pps peak-pps peak-time

5 340819 757369 16:31:57 UTC Sun Sep 30 2012

RTLSGR1#

Raju Sekharan · ‎10-01-2012

It is not a Bug

Module FIB TCAM usage: Total Used %Used

5 72 bits (IPv4, MPLS, EoM) 196608 196194 100%

You FIB TCAM is 100% utilized

For non-XL mode systems:

–IPv4 unicast and MPLS—192,000 routes

–IPv6 unicast and IPv4 multicast—32,000 routes

reference link:

http://www.cisco.com/en/US/docs/ios/ipswitch/command/reference/isw_i1.html#wp1087660

one option is to reduce the number of ipv4 prefixes you receive

Another option is to reduce the FIB TCAM used for ipv6 and multicast if you are not using them so that ipv4 FIB TCAM gets more TCAM space

example commands

mls cef maximum-routes ip-multicast 2

mls cef maximum-routes ipv6 2

This will release 28Kb of TCAM which can be used for IPv4.

This requires reload of the router. Also please keep only one boot statement when you boot

Check show ip route summary to see the number of ipv4 prefixes you have

sabuz_banik · ‎10-02-2012

OK, Let me check. I will update you shortly.

sabuz_banik · ‎10-02-2012

Shall i do the same for MPLS aswell ? As i am not running any MPLS network.

Raju Sekharan · ‎10-02-2012

MPLS is using only 2k. You may reduce it to 1k

How is your show ip route summary looking? D owe have more than 200K routes?

sabuz_banik · ‎10-02-2012

OK I will make it 1K. I am unable to reboot now as device is running in a production.Watch my IP route Sum. I am unable to filter BGP table as source of carrier can be any real ip.

RTLSGR1#sh ip route summary

IP routing table name is default (0x0)

IP routing table maximum-paths is 32

Route Source Networks Subnets Replicates Overhead Memory (bytes)

static 3 2 0 300 860

connected 0 6 0 400 1032

ospf 1 1 2 0 240 528

Intra-area: 3 Inter-area: 0 External-1: 0 External-2: 0

NSSA External-1: 0 NSSA External-2: 0

bgp 58636 152873 268794 0 25300020 72526724

External: 421660 Internal: 7 Local: 0

internal 4818 17317728

Total 157695 268804 0 25300960 89846872

RTLSGR1#

Raju Sekharan · ‎10-02-2012

Hi

You have more than 260K routes.

Your Total FIB TCAM is 252 K inluding ipv6, mpls,ipv6 and multicast. So reducing the TCAM for IPV6 and multicast is not going to help

You need to redue the routes you are receving on this router.

sabuz_banik · ‎10-02-2012

Hi Rajs,

Is there any way to increase Tcam Size ?

How shall i reduce route/prefixes as source ip is not fixed?

Shall i replace the SUP?

If i connect with ASR1002-10G router will it be able to handle this much route ?

Any way to stick with this router with tuning guideline ?

Raju Sekharan · ‎10-02-2012

Hi Sabuz,

If your requirement is to receive complete BGP routes, RSP 720-3C is not the right SUP for you

You can't increase the TCAM size. if you change the SUP to RSP 720-3CXL you will be able to handle upto 1M routes

Here is the comparison

http://www.cisco.com/en/US/prod/collateral/routers/ps368/product_data_sheet0900aecd8057f3b6.html

ASR1K will be able to handle it

http://www.cisco.com/en/US/prod/collateral/routers/ps9343/data_sheet_c78-441072.html

If you want to stick with the same router, either you need to reduce the routes or change the SUP to 3CXL

sabuz_banik · ‎10-02-2012

Hi Raj,

It was nice having views from you. Thanks a lot for your kind help.I hope your extened help will continue in future aswell.

Have a very nice time.

Thanks again.

Raju Sekharan · ‎10-02-2012

Hi Sabuz,

Sure. You are welcome. Wish you nice time

Raju Sekharan · ‎10-02-2012

Please don't forget to mark the query as answered if you are satisfied with the answer