We have a few instances of Cisco ACI Fabric running in a 3 completely unique data centers. All on different versions.
5.2(8i) being the newest
5.2(7g) in the middle
4.2(5l) being the oldest
All 3 started at 4.2 at some point. We will be moving to 6.x this year if all goes as planned.
We do not change our static routes very often, literally only a few times and every time we see the exact same behavior.
When we remove a static route from an L3Out the leafs that had EGPs with the contract out for that specific L3Out still retain the old route path.
We can see the destination subnet in the "show system internal policy-mgr prefix" table but only on the leafs in the fabric where the EGPs are learned that consume the contract for that specific L3Out.
The only work around to force the traffic via a different L3Out is to create a better static route.
We had 10.0.0.0/16 routing out one L3out and there was maintenance to be done on a firewall a few hops down that path.
I removed the 10.0.0.0/16 static and removed it from the L3Out Subnet export/shared route controls.
"show ip route" no longer showed the route.
I set the static for the same subnet out a different L3out and the new path shows in the routing table but traffic does not work.
Elam shows its still trying to use the old path/dclass.
When I run "show system internal policy-mgr prefix" I can see the old path is showing there but only on the leafs where the consumer EPGs are running.
Only the new path shows on the other leafs.
We moved a VM to a different VM Host being learned on a different leaf and the traffic started working for that one VM only.
I created a couple /17 routes out the new L3out and everything started working and I can see both the /16 and /17 now show in the original leafs.
THEN - when the maintenance was done I deleted the /17 routes hoping the /16 would again work but to my surprise the /17 were still showing in the table!!!
I had to create the /18 routes out the original L3Out path to get this traffic working again.
Now I can see the /16, /17 and /18 routes all in the table.
/16 and /18 go out the old L3out. /17 stuck with dclass out the temp L3out path.
I thought this was a bug in this version of Fabric, just have not had chance to reload/update yet.
We just had the exact same issue in a different fabric. Same exact issue, same exact workaround to get it working but this time it took down production traffic during a standard maintenance window.
TAC told me to do a clean reboot on the leafs with the "stuck" routes but in my opinion that is a horrible fix for the most basic config change I can think of in a network.
Any thoughts on this? Something we might be doing wrong? Any way to forcefully remove an old entry that someone might know.
Appreciate the help!
