High CPU usage BGP router process

[krun_shah]

I have an issue with 7600 router where CPU goes up to 60-70% and memory is also high. Both due to BGP Router process.

http://www.cisco.com/en/US/tech/tk365/technologies_tech_note09186a00809d16f0.shtml#process

According to our baseline it should not be more than 40% at any given time. We see high CPU uptp 70% consistently.

CPU utilization for five seconds: 99%/0%; one minute: 57%; five minutes: 55%

PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process

442    66173704  90234125        733 96.86% 46.09% 46.30%   0 BGP Router

   7   509291060  26330202      19342  1.17%  3.90%  2.99%   0 Check heaps

Router has 1 eBGP session with ISP from where it downloads whole IPv4 internet routing table and two IBGP session with other two rotuer. When I look at BGP summary table I see many updates received from ISP and sent out to IBGP neighbors. Also did debug BGP updates to confirm it.

We are seeing this issue since ISP has upgraded their router 20 days ago. Router also seen following error.

%BGP-6-BIGCHUNK: Big chunk pool request 628 for community. Replenishing with malloc

I have not reset the BGP session with ISP yet. Is there any way I can supress BGP updates coming from ISP and see if CPU and memory USAGE reduces. IOS version 12.2(33)SRD and RSP720 with PFC 3cXL

[Giuseppe Larosa]

Hello Krun,

the system with RSP 720 and PFC 3CXL is well equipped for this kind of job, and a single upstream provider with full internet table should not be a problem.

May I ask you what kind of router is the new device used by your upstream ISP?

The loading phase from the ISP should last some minutes it is not possible it is going on for 20 days.

The error message might be related to some BGP advertisement with a long list of BGP communities attriibutes attached to it and the need to allocate memory to store it invoking the malloc routine instead of other routine.

Hope to help

Giuseppe

[krun_shah]

Yes You are right we are not seeing issue during neighbor establishment process. debug bgp updates show a lot of BGP updates (new prefix added/removed etc) and router needs memory to process it. We do not know ISP's router model.

Is there a nerd knobb that I can enable which reduces processing time of the updates received. OR Can I supress updates received?

[Giuseppe Larosa]

Hello Krun,

BGP route dampening might be of help.

http://www.cisco.com/en/US/docs/ios/12_1/iproute/command/reference/1rdbgp.html#wp1017894

A useful troubleshooting link for high cpu caused by BGP

http://www.cisco.com/en/US/tech/tk365/technologies_tech_note09186a00809d16f0.shtml

Anyway, you can also open a ticket with the upstream ISP as it is not normal to have so many updates coming from their device. You should organize a joint troubleshooting session with them to understand the issue.

Hope to help

Giuseppe

[krun_shah]

I would try one more thing, which is to put my 2 IBGP neighbors into a peer-group, that way BGP builds one set of update message for the peer-group, apply routing policies for entire group rather tahn doing one neighbor at a time thereby reducing some CPU and memory overhead.

But not sure why only this router has such problem, Other router with exact same config has two ISP eBGP connection getting full internet routing table from both ISP do not see this problem.

[Giuseppe Larosa]

Hello Krun,

in recent IOS versions there are the BGP update-groups and you can see them as implicit peer-group. They are used to save resources in building BGP updates.

You can check that both iBGP peers are in the same update-group using

show ip bgp update-groups

So you shouldn't see a great change configuring the peer-group, however it is wise to try.

If the other C7600 device has exactly the same HW/SW and similar configuration the issue is in the interaction with  the new ISP router.

Hope to help

Giuseppe