Access list consolidation in the Policy based routing

[Gurbinder-Kabbay]

Hi Experts

I am facing a problem in network of high utilization of TCAM in Nexus 9k in VXlan Fabric. I have large number of access-list and when I am pushing it to Nexus 9K switching then TCAM utilization is going up to 90 percent. to avoid this I am thinking of workaround to consolidate the large access list of /32 host to subnet level. these access list I am calling in the route-map and applying this route-map to SVI interface. In this route-map I am setting next hop to force the traffic to some specific next hop to apply the PBR.

for example:

                      (Source)-----(Destination)

ip access-list 10.1.1.60/32 20.1.1.1.1/32

ip access-list 10.1.1.61/32 20.1.1.1.1/32

ip access-list 10.1.1.62/32 20.1.1.1.1/32

ip access-list 10.1.1.63/32 20.1.1.1.1/32
ip access-list 10.1.1.64/32 20.1.1.1.1/32
ip access-list 10.1.1.65/32 20.1.1.1.1/32

ip access-list 10.1.1.66/32 20.1.1.1.1/32

ip access-list 10.1.1.67/32 20.1.1.1.1/32

ip access-list 10.1.1.68/32 20.1.1.1.1/32

ip access-list 10.1.1.69/32 20.1.1.1.1/32

ip access-list 10.1.1.70/32 20.1.1.1.1/32

ip access-list 10.1.1.71/32 20.1.1.1.1/32

ip access-list 10.1.1.72/32 20.1.1.1.1/32

ip access-list 10.1.1.73/32 20.1.1.1.1/32

Query : "if I try to consolidate the above access list like below, will it work ?, because in consolidation of 10.1.1.64/29 the unusable IPs are coming like 10.1.1.64 and 10.1.1.71.  so will this consolidation will cover these two unusable IPs from subnet 10.1.1.64/29? and PBR will work or not?"

ip access-list 10.1.1.60/32 20.1.1.1.1/32

ip access-list 10.1.1.61/32 20.1.1.1.1/32

ip access-list 10.1.1.62/32 20.1.1.1.1/32

ip access-list 10.1.1.63/32 20.1.1.1.1/32

ip access-list 10.1.1.64/29 20.1.1.1.1/32
ip access-list 10.1.1.72/32 20.1.1.1.1/32

ip access-list 10.1.1.73/32 20.1.1.1.1/32

Regards

Gurbinder

[MHM Cisco World]

Pbr in vxlan?

Can you elaborate 

MHM

[Gurbinder-Kabbay]

I am just telling that my DC environment is VXlan and we have 9ks, we are using NDFC to manage our DC Network.
However, My question is general question, nothing to do with VXlan.

[MHM Cisco World]

You fill Tcam with /32

You need to summary these IP /32 to /29

MHM

[Gurbinder-Kabbay]

Yes, I need to reduce the access-list lines , so the utilization can be brought down. but I have doubt that IPs and subnet which I mentioned in the query that will impact the connectivity for unusable IPs in the subnet.

[MHM Cisco World]

10.1.1.60/30 → covers 60–63

10.1.1.72/31 → covers 72–73

What you need is use some online subnet calculator 

If IP not need include large subnet use deny in top of pbr

If single IP need you can use /32 

MHM

[Gurbinder-Kabbay]

I cannot consolidate big subnet , because my destination in access-list changes. from bigger /24 I have to do consolidation like /29 or /28. So I just need clarity on below point. please advise.

Query : "if I try to consolidate the above access list like below, will it work ?, because in consolidation of 10.1.1.64/29 the unusable IPs are 10.1.1.64 and 10.1.1.71.  so will this consolidation will cover these two unusable (10.1.1.64 and 10.1.1.71) IPs from subnet 10.1.1.64/29? and PBR will work or not for these unusable (10.1.1.64 and 10.1.1.71) IPs ?"

[MHM Cisco World]

ACL

Deny 10.1.1.64/32 20.1.1.1.1/32

Deny 10.1.1.71/32 20.1.1.1.1/32

Permit 10.1.1.64/29 20.1.1.1.1/32

This how you need to config ACL of PBR

MHM

[Gurbinder-Kabbay]

Sorry , If I confused you, I dont want to deny these two IPs in subnet. I want PBR and acl effect to these two IPs 10.1.1.64/32, 10.1.1.71/32.
I just want to know, If I apply access list on 10.1.1.64/29 : will these two IPs 10.1.1.64/32, 10.1.1.71/32 will be covered or not ?
As I have doubt from subnetting concept that these IPs will be ignored. because first and last IPs are not usable IPs.

[Richard Burts · Accepted solution]

We do not know much about your environment, but based on what we have so far I can say this: if you have 10.1.1.64/29 this defines a subnet with 8 addresses. Normally we would consider 10.1.1.64 as the subnet address and 10.1.1.71 as a broadcast address in that subnet. And normally there are not hosts with those addresses. But for the purpose of PBR I believe that it would match those addresses and would policy route packets with those addresses. 

In the original post you say "consolidate the large access list of /32 host to subnet level". Are you saying that 10.1.1.64/29 is a subnet? In which case .64  and .71 would not be host addresses and matching does not matter.

[Joseph W. Doherty · Accepted solution]

Yes (as also noted by @Richard Burts ), an ACE IP address and its mask, matches the complete IP address block.  I.e. the first and last IPs of an address block are NOT excluded.

BTW, it's my understanding that Cisco devices optimize TCAM usage, so you may not see as huge a TCAM usage resource savings as you might expect.

For example say we have following six host IPs, we want to treat "special":

192.168.1.1-5 and .7.

Even if your config file has a host ACE for each of the six IPs, a TCAM optimizer may transform that into an ACE matching the address block of eight IPs with two ACEs matching the two not included IPs, .0 and .6.   I.e. three ACEs rather than the six as seen in the config.

In other words, what you want to accomplish, may have already been done by the TCAM optimizer.

However, such optimization would be for what was actually configured.  If you can logically be less restrictive, you should be able to reduce the actual number of TCAM entries.  For example, in my above example, if it doesn't truly matter .0 and .6 are treated differently, then the whole address block, .0-7, can be matched with just one ACE.

I mentioned the above, because you might do a whole lot of work for almost no reduction of TCAM resource usage.

[MHM Cisco World · Accepted solution]

You edit the comment

Where network ID abd broadcast IP of subnet not use?

Only in config IP under interface & dhcp pool

ALL Other cases will use this IP it dont care about it network ID or broadcast IP

[Gurbinder-Kabbay]

Thanks @Joseph W. Doherty  for explanation, I have thousands acls on my devices, I got my answer. Thanks @Richard Burts , @MHM Cisco World , @Joseph W. Doherty  for giving your precious time to my post, highly appreciated.

[Richard Burts]

You are welcome. I am glad that our suggestions have been helpful. Thank you for marking the discussion as solved. This will help other members of the community to identify discussions that are helpful. This community is an excellent place to ask questions and to learn about networking. I hope to see you continue to be active in the community.

[Joseph W. Doherty]

BTW, you might find the following references interesting and/or useful:

ACL and QoS TCAM Exhaustion Avoidance on Catalyst 4500 Switches

Troubleshoot Security ACL TCAM Exhaustion on Catalyst 3850 Switches

TCAM Resource Issue Workarounds Explained

Cisco-Nexus-TCAM-usage-calculator

The above will lead you to these interesting references:

QoS on the Nexus 9k – Estimating TCAM usage, Part 1

For the above reference, you might want to look as the preceding and following articles.

ACL TCAM and LOUs in Catalyst 6500

Note: although you asked specifically about the Nexus 9K, I included other Cisco switch platforms, as Cisco TCAM usage appears similar across their switches.  However, Nexus 7K and 9K are included in the above.

If you really want to minimize TCAM usage, you need to understand how it maps ACL ACEs into TCAM.  Again, it's not just a simple every ACE is a single TCAM entry.