Cisco 7609 Routing Issue

[batchley2945]

The ISP company I work for owns a Cisco 7609 as a CORE router and we are having some major issues.

All of a sudden, when we perform ping/traceroute to specific subnets (Usually AT&T subnets) from our edge routers, we will receive around 70% packet loss.

E.G --- Edge Router > CORE Router > WAN > AT&T Subnet = 70% Packet Loss

However, if we perform the same tasks on the CORE Router, the issues do not persist. we will receive 0% packet loss.

E.G --- CORE Router > WAN > AT&T Subnet = 0% Packet Loss

The only way around this was to add static routes in our CORE Router which are redistributing via OSPF to the routers that are having issues then the above scenario works with no issues on our edge routers.

Any help would be appreciated, I will try to provide as much information as possible but I'm limited on what I can provide.

Thanks.

[Peter Paluch]

Hi,

The only way around this was to add static routes in our CORE Router which are redistributing via OSPF to the routers that are having issues then the above scenario works with no issues on our edge routers.

If I understand you correctly than what you are saying is that

• Pinging from CORE is fine, pinging from EDGE through CORE is lossy
• Adding static routes into CORE routers and having them redistributed via OSPF to EDGE routers solves the packet loss issue

Am I correct?

Can you be more specific about these questions please?

1. What are the static routes about? Where do they point to? What routing information do they add to the routing information already present on EDGE routers?
2. How does the routing on EDGE routers look like? Without the redistributed static routes from CORE routers, how would EDGE routers know how to reach themselves?

I am somewhat puzzled by the fact that modifying the routing information has impact on packet losses. That, in my opinion, can be caused by one of these possibilities: Either the static routes specify a different path than the one that would be taken ordinarily, a path that is less utilized and so not that prone to packet losses; Or you have a TCAM overfill issue, with some routes possibly being handled in software (hence the packet losses) and some other in hardware.

Can you perhaps check the following threads and the checks mentioned there to see if the TCAM could be the culprit?

https://supportforums.cisco.com/discussion/11357821/7600-fm-4-tcamentry-hardware-tcam-entry-capacity-exceeded

http://www.cisco.com/c/en/us/support/docs/switches/catalyst-6500-series-switches/117712-problemsolution-cat6500-00.html

Looking forward to your response.

Best regards,
Peter

[batchley2945]

1. One static route points to the DSL connection that were testing off of that has issues. it traceroutes confirm it takes the same path regardless of the static route in place or not since its the only connection we have up at the moment.

2. EDGE routers are redistributing the static routes from the CORE through OSPF or BGP depending on the configuration. Some actually have default routes to the CORE. Sorry I wasn't clear about that earlier.

The path is not over utilized considering we add a static route in the CORE router and then the issues go away.

It could possibly be a TCAM issue, our Network Director is going on-site tonight to reboot the CORE router and failover to the secondary if nessecary. I will let you know of the results.

Thanks for your assistance Peter.

Sincerely,

Blynn

[batchley2945]

Looks like a reboot along with allocating more for the TCAM resolved the issue. 

We were pushing around 99% for the 520k routes in the TCAM. Allocating that to 750 and disabling soft configuration inbound across our internet providers resolved the issues.


Thanks!

[Peter Paluch]

Hi Blynn,

Indeed - sounds really like the TCAM and possibly memory exhaustion was at the core of this issue. I am glad you were able to solve it and I am also thankful that you let us know it worked.

Regarding the soft reconfiguration inbound - I am actually surprised to hear you had it configured. Did you require any specific functionality out of this? The soft reconfiguration inbound was an ugly hack around an ancient BGP deficiency but since 2000, all decent BGP implementations support the RFC 2918 Route Refresh feature allowing a router to ask its BGP peer to re-send all routes of a particular address family, instead of requiring to store all unfiltered routes locally and consuming lots of memory in the process. The Route Refresh capability is negotiated dynamically and you do not need to activate it in any way. It is actually surprising how many SPs still use the soft-reconfig inbound and waste their memory uselessly.

Best regards,
Peter