6509 bgp reconverges after flap and gives "no memory" condition

[miked]

Neighbor 1.1.1.1 flapped causing the reconvergence to eat up memory. We are using Sup2 @ 512MB and gettign full routes from each of our neighbors.

the IOS version is: sup-bootflash:c6sup22-psv-mz.121-22.E2.bin

This is all I have from the logs:

01:11:52: %BGP-5-ADJCHANGE: neighbor 1.1.1.1 Down No memory

01:11:52: %BGP-5-ADJCHANGE: neighbor 2.2.2.2 Down No memory

01:11:52: %FIB-3-NOMEM: Malloc Failure, disabling DCEF

01:12:05: %FIB-2-FIBDOWN: CEF has been disabled due to a low memory condition.

It can be re-enabled by configuring "ip cef [distributed]"

01:12:07: %BGP-3-NOTIFICATION: sent to neighbor 2.2.2.2 3/1 (update malformed) 0 bytes

01:12:07: %BGP-5-ADJCHANGE: neighbor 1.1.1.1 Up

01:12:45: %BGP-5-ADJCHANGE: neighbor 2.2.2.2 Up

01:13:05: %FIB-2-FIBDOWN: CEF has been disabled due to a low memory condition.

It can be re-enabled by configuring "ip cef [distributed]"

Anybody had an issue with this before? IOS memory leak maybe? Thanks in advance.

[Giuseppe Larosa]

Hello Mike,

unfortunately your hardware is in end of support.

see

http://www.cisco.com/en/US/prod/collateral/switches/ps5718/ps708/prod_end-of-life_notice0900aecd80423d31.html

This means that no new IOS will be released by Cisco for your system.

It might be a SW bug, but also the fact that you are pushing the box over its own limits as now the full internet table is in excess of 370,000 routes and you receive two full tables from two different neighbors.

I may be wrong but the CEF table limit in your system may be very close (I couldnìt find the sup2 datasheet on CCO)

I had issues with C6500 and Sup720 3BXL when a linecard was changed and it was installed with a wrong non XL DFC daughter card. The linecard was not able to store more then 256k routes and so it has fallen back to process switching for many routes. Device cpu started to be very high.

I think you should consider an HW upgrade in the mid term. In the short term you can try to download last SW for your HW that should be something like 12.1. (26)E but I'm not sure it is still available on CCO

Hope to help

Giuseppe

[miked]

Thank you for the information. I'll dig into the cef limits. The device is currently not taking in full routes due to the issue we ran into. If I figure it's too much for the device I'll keep the minimal routes or seek a separate router based device to manage the bgp. Thanks again.

[Edison Ortiz]

You can not take full routes from the internet while using such an old Supervisor.

You need at least Sup720-3BXL or 3CXL for full IPv4 routing without surpassing the TCAM limit.

[miked]

Thanks Edison,

I recieved more details from the logs and it points to this IOS bug: CSCec43573

From other sources I've dug up it supposedly fixed in 12.1(26)E1, I still need to dig through the fixes in the release but I assume I'll find it.

Just for on going informational purposes, this was the info that was sent to me today:

4d18h: %BGP-5-ADJCHANGE: neighbor 1.1.1.1 Down Interface flap

4d18h: %SYS-SP-2-MALLOCFAIL: Memory allocation of 65536 bytes failed from 0x40101E64, alignment 4

Pool: Processor  Free: 1454260  Cause: Memory fragmentation

Alternate Pool: None  Free: 0  Cause: No Alternate pool

-Process= "FIB Control Task", ipl= 0, pid= 73

-Traceback= 40106B88 40108610 40101E6C 40187968 40182E7C 40183180 4017A558 400FB22C 400FB218

4d18h: %BGP-5-ADJCHANGE: neighbor 1.1.1.1 Up

That's what I have so far. I'll definitely dig into the hardware side of it too if I want to revert to full routes again. Thanks guys!