ASR9K IOS-XR TCAM verify

[mr.hein2013]

how to verify TCAM ASR9K IOS-XR ?

please help me

[smilstea]

What do you want to verify?

How many TCAM entries an ACE/ACL uses?

How many used / free TCAM entries there are?

How many TCAM entries does an ACL/ACE use?

Due to bit-boundaries and what ACL options are used a particular ACE may use multiple TCAM entries

1 ACE without any ranges (TCP ports, UDP ports, ttl) will typicall map to 1 TCAM entry.

ACEs with ranges occupy multiple TCAM entries

Note: 1 ACL cannot have more than 64K TCAM entries

Using bit boundaries helps to minimize the number of TCAM entries needed:

For example matching ports ranged 0 - 1023 would take a single TCAM entry

However, if you needed to match 0-1024, that would be two rules:

•  1 to match 0-1023
•  1 to match only 1024

As well interfaces on the same NP having the same ACL applied in the same direction share TCAM entries, this helps minimize the TCAM space that is used.

ACL Scaling

What affects the ACL scale for a LC?

• the physical size of the TCAM.  No matter what happens in software, the TCAM is a fixed size per linecard type.
• -E Trident cards have 256k 144b entries
• -B Trident cards have 128k 144b entries
• -L Trident cards have 64k 144b entries
• -TR Typhoon cards have 24k 160b entries
• -SE Typhoon cards have 96k 160b entries
• When required, four of these 144/160b entries can be logically combined into a larger 576/640b entry for more complex lookups.
• Note: 144/576 bit keys cannot accommodate for port ranges.

• the number and size of applications using the TCAM.  In the 9k, the primary applications that TCAMs are used for are:
• matching packets to an interface (i.e. which subinterface do frames with VLAN tag XXX actually belong to?)
• for classifying iFIB/for-us traffic (i.e. "what is the policer for BGP session XXX that needs to be punted?")
• QoS class maps, whether they explicitly use an ACL or not: ("Which queue do precedence 5 packets go to?")
• security ACLs ("Are packets matching ACL entry XXXX allowed or not?"
• Region sizing:
• the TCAMs are divided into regions, based on the applications above.  Within a region different applications may share entries (i.e. ACL entries can share space with QoS entries) but different types cannot share entries (meaning L2 entries have to be in a different region than 144b L3 entries, which have to be in a different region than 576b entries, etc)
• The complexity of the ACLs.
• Depending on what is being matched, the TCAMlookup key may fit into a 144/160b entry, or it may require a larger 576/640b entry (as above, this is just four 144b entries logically squashed together)
• IPv4 rules require 144/160b entries.
• IPv6 rules require 576/640b entries.

• Range expansion:
• if you use a range of port numbers in your ACL entry, this has to be expanded to fit into a set of binary rules.
• For example, if you say "match ports from 0-10" this ends up being THREE unique TCAMentries, because the range 0-10 has to be converted into binary-compatible rules
• entry #1: matches ports 0-7, the "value" is bitstring 0000 0000 and the "mask" is 0000 0xxx
• entry #2: matches ports 8-9, the "value" is bitstring 0000 1000 and the "mask" is 0000 000x
• entry #3: matches port 10, where the "value" is bitstring 0000 1010 and "mask" is 0000 0000

• Main restrictions as to how large an ACL can be:
  • No single ACL can ever be larger than 64k ACL entries (internal data structure limited to 16 bits)
  • -E cards, there are 96k entries in the v4 region, 4k entries in v6 (16k TCAM cells,  4 cells/entry)
  • -B cards, there are 48k entries in the v4 region, 2k entries in v6 (8k TCAM cells,  4 cells/entry)
  • -L cards, there are 24k entries in the v4 region, 1k entries in v6 (4k TCAM cells,  4 cells/entry)
  • -SE Typhoon cards, there are 96k entries in the v4 region, 4k entries in v6 (16k TCAM cells,  4 cells/entry)
  • -TR Typhoon cards, there are 24k entries in the v4 region, 1k entries in v6 (4k TCAM cells,  4 cells/entry)
  • -SE Tomahawk cards, there are 192k entries in the v4 region, 8k entries in v6 (32k TCAM cells,  4 cells/entry)
  • -TR Tomahawk cards, there are 48k entries in the v4 region, 2k entries in v6 (8k TCAM cells,  4 cells/entry)
  • If (starting in 4.3.1) scaled ACL is used then both IPv4 and IPv6 will use the 640 region for 16k total entries (SE cards)

show pfilter-ea fea ipv4-acl <ACL> loc <loc>

shows you how many ACEs, how many TCAM entries, and TCAM entries per ACE (must be applied to see)

ex:

ACE List for This Region:
        seq_num 94, tcam_entries 2, stats_ace_id 0x535880 (0x63c400) new 0
        seq_num 95, tcam_entries 2, stats_ace_id 0x535881 (0x63c408) new 0
        seq_num 96, tcam_entries 1, stats_ace_id 0x535882 (0x63c410) new 0
        seq_num 97, tcam_entries 1, stats_ace_id 0x535883 (0x63c418) new 0
        seq_num 98, tcam_entries 1, stats_ace_id 0x535884 (0x63c420) new 0
        seq_num 99, tcam_entries 1, stats_ace_id 0x535885 (0x63c428) new 0
        seq_num 100, tcam_entries 7, stats_ace_id 0x535886 (0x63c430) new 0
        seq_num 101, tcam_entries 2, stats_ace_id 0x535887 (0x63c438) new 0
        seq_num 102, tcam_entries 2, stats_ace_id 0x535888 (0x63c440) new 0
        seq_num 103, tcam_entries 2, stats_ace_id 0x535889 (0x63c448) new 0
        seq_num 104, tcam_entries 2, stats_ace_id 0x53588a (0x63c450) new 0
        seq_num 105, tcam_entries 2, stats_ace_id 0x53588b (0x63c458) new 0
        seq_num 106, tcam_entries 1, stats_ace_id 0x53588c (0x63c460) new 0
        seq_num 107, tcam_entries 2, stats_ace_id 0x53588d (0x63c468) new 0
        seq_num 108, tcam_entries 16, stats_ace_id 0x53588e (0x63c470) new 0

how pfilter-ea fea summary loc {location}

shows how many total ACEs/TCAM entries/stats counters are used on the linecard (per NP, where NP=“chan#”)

RP/0/RSP0/CPU0:ASR9006-E#show pfilter-ea fea summary loc 0/1/cpu0
Tue Jun 24 20:17:22.041 UTC

******** NP Resource Usage Summary ************

Chan #  144-bit TCAM Entries  576-bit TCAM Entries  Stats   SS Hash Entries
========================================================================
  0            36                   20            56            0
  1             0                    0             0            0
  2             0                    0             0            0
  3             0                    0             0            0
  4           139                    0            70            0
  5             0                    0             0            0
  6             0                    0             0            0
  7             0                    0             0            0

Here we have the number of TCAM entries used (139 out of 24k) and 70 ACE entries for NP 4

show access-lists ipv4 <acl> hardware [ingress | egress] resource-usage loc {location}

shows compiled ACL hardware stats (TCAM, compression, etc)

RP/0/RSP0/CPU0:ASR9006-E#show access-lists ipv4 test-acl hardware egress resource-usage loc 0/1/cpu0
Tue Jun 24 20:18:30.946 UTC

 NP                     : 4
 Rules (ACE)            : 70
 ACL compression level  : 0
 Fields compressed      : None
 TCAM Entries  used     : 139  ( 24k total)
 TCAM Key Width         : 160 ( 0 total for compressed fields)

Checking NP TCAM Summary

show prm server tcam summary all [ACL | AFMON | IFIB | LI | PBR | QOS | all] [all | np0 | npx] location {location}

shows TCAM utilization for the application(s) specified

RP/0/RSP0/CPU0:ASR9006-E#show prm server tcam summary all all np4 loc 0/1/cpu0
Wed Jun 25 16:07:18.618 UTC

                Node: 0/1/CPU0:
----------------------------------------------------------------

TCAM summary for NP4:

  TCAM Logical Table: TCAM_LT_L2 (1)
    Partition ID: 0, priority: 2, valid entries: 3, free entries: 317
    Partition ID: 1, priority: 2, valid entries: 0, free entries: 320
    Partition ID: 2, priority: 1, valid entries: 0, free entries: 320
    Partition ID: 3, priority: 1, valid entries: 0, free entries: 11840
    Partition ID: 4, priority: 0, valid entries: 76, free entries: 11700
  TCAM Logical Table: TCAM_LT_ODS2 (2), free entries: 16304, resvd 128
    ACL Common Region: 0 entries allocated. 0 entries free
    Application ID: NP_APP_ID_IFIB_IPV4 (0)
      Total: 1 vmr_ids, 8005 active entries, 8005 allocated entries.
    Application ID: NP_APP_ID_QOS (1)
      Total: 0 vmr_ids, 0 active entries, 0 allocated entries.
    Application ID: NP_APP_ID_IPV4_ACL (2)
      Total: 1 vmr_ids, 139 active entries, 139 allocated entries.
    Application ID: NP_APP_ID_AFMON (3)
      Total: 0 vmr_ids, 0 active entries, 0 allocated entries.
    Application ID: (null) (4)
      Total: 0 vmr_ids, 0 active entries, 0 allocated entries.
    Application ID: NP_APP_ID_PBR (5)
      Total: 0 vmr_ids, 0 active entries, 0 allocated entries.
  TCAM Logical Table: TCAM_LT_ODS8 (3), free entries: 1929, resvd 64
    ACL Common Region: 0 entries allocated. 0 entries free
    Application ID: NP_APP_ID_IFIB_IPV6 (0)
      Total: 1 vmr_ids, 2103 active entries, 2103 allocated entries.
    Application ID: NP_APP_ID_QOS_IPV6 (1)
      Total: 0 vmr_ids, 0 active entries, 0 allocated entries.
    Application ID: NP_APP_ID_ACL_IPV6 (2)
      Total: 0 vmr_ids, 0 active entries, 0 allocated entries.
    Application ID: (null) (3)
      Total: 0 vmr_ids, 0 active entries, 0 allocated entries.
    Application ID: (null) (4)
      Total: 0 vmr_ids, 0 active entries, 0 allocated entries.
    Application ID: NP_APP_ID_PBR_IPV6 (5)
      Total: 0 vmr_ids, 0 active entries, 0 allocated entries.

As we have seen before the NP is using 139 TCAM entries for ACLs

We also have 8005 IFIB TCAM entries

We are using 8144 entries on the local table TCAM_LT_ODS2 and have 16304 entries free (24K for -TR cards)

We also get the number of vmr_ids which is essentially how many unique ACLs are applied in the TCAM on the NP.

Sam

[gergo.andrasi]

Hi Sam,

Thank you for the insights, I know this is an old thread, but I have some addition to add for future reference and also some questions someone hopefully can answer.

You can view the actual TCAM entries that were generated for your ACL, for example with this ACL:

ipv4 access-list tcam-test
10 permit ipv4 any host 10.0.0.254
20 permit tcp 10.56.108.0 0.0.0.255 host 10.0.230.210 eq cmd
30 permit tcp 10.56.108.0 0.0.0.255 host 10.0.230.210 range 1514 1515
40 permit ipv4 10.56.108.0 0.0.0.255 host 10.165.130.2
50 deny ipv4 any 10.0.0.0 0.255.255.255
60 deny ipv4 any 192.168.0.0 0.0.255.255
70 permit ipv4 any any nexthop1 vrf tcamtestvrf ipv4 10.56.108.1
100 deny ipv4 any any

You can get the vmr-id of the ACL via the pfilter-ea fea command which is missing from the previous excerpt:

RP/0/RSP0/CPU0:ASR9001-TCAM#show pfilter-ea fea ipv4-acl tcam-test location 0/0/CPU0

Rgn tcam-test, lkup v4, Dir In, Chan 0, acl_id 3, vmr_id 3, num_aces 8, num_tcam_entries 11, refcnt 1.
...

You can view how many active entries the specifies vmr_id takes up in tcam:

RP/0/RSP0/CPU0:ASR9001-TCAM#show prm server tcam summary all all detail np0 location 0/0/CPU0

Node: 0/0/CPU0:
----------------------------------------------------------------

TCAM summary for NP0:

TCAM Logical Table: TCAM_LT_L2 (1)
Partition ID: 0, priority: 2, valid entries: 7, free entries: 2041
Partition ID: 1, priority: 2, valid entries: 0, free entries: 2048
Partition ID: 2, priority: 0, valid entries: 0, free entries: 2048
Partition ID: 3, priority: 0, valid entries: 6, free entries: 24570
Partition ID: 4, priority: 0, valid entries: 3, free entries: 67581
TCAM Logical Table: TCAM_LT_ODS2 (2), free entries: 89239, resvd 128
ACL Common Region: 448 entries allocated. 448 entries free
Application ID: NP_APP_ID_IFIB (0)
VMR ID: 1, active entries: 8005, total entries: 8005
Total: 1 vmr_ids, 8005 active entries, 8005 allocated entries.
Application ID: NP_APP_ID_QOS (1)
VMR ID: 21, active entries: 21, total entries: 21
Total: 1 vmr_ids, 21 active entries, 21 allocated entries.
Application ID: NP_APP_ID_ACL (2)
VMR ID: 3, active entries: 11, total entries: 11
VMR ID: 4, active entries: 7, total entries: 7
VMR ID: 6, active entries: 23, total entries: 23
VMR ID: 8, active entries: 23, total entries: 23
VMR ID: 9, active entries: 210, total entries: 210
VMR ID: 16, active entries: 94, total entries: 94
VMR ID: 19, active entries: 94, total entries: 94
Total: 7 vmr_ids, 463 active entries, 463 allocated entries.
Application ID: NP_APP_ID_AFMON (3)
Total: 0 vmr_ids, 0 active entries, 0 allocated entries.
Application ID: NP_APP_ID_LI (4)
Total: 0 vmr_ids, 0 active entries, 0 allocated entries.
Application ID: NP_APP_ID_PBR (5)
Total: 0 vmr_ids, 0 active entries, 0 allocated entries.
TCAM Logical Table: TCAM_LT_ODS8 (3), free entries: 15113, resvd 123
ACL Common Region: 448 entries allocated. 448 entries free
Application ID: NP_APP_ID_IFIB (0)
VMR ID: 1, active entries: 603, total entries: 603
Total: 1 vmr_ids, 603 active entries, 603 allocated entries.
Application ID: NP_APP_ID_QOS (1)
Total: 0 vmr_ids, 0 active entries, 0 allocated entries.
Application ID: NP_APP_ID_ACL (2)
VMR ID: 2, active entries: 22, total entries: 22
VMR ID: 5, active entries: 34, total entries: 34
VMR ID: 6, active entries: 34, total entries: 34
Total: 3 vmr_ids, 90 active entries, 90 allocated entries.
Application ID: NP_APP_ID_LI (4)
Total: 0 vmr_ids, 0 active entries, 0 allocated entries.
Application ID: NP_APP_ID_PBR (5)
VMR ID: 7, active entries: 6, total entries: 6
VMR ID: 8, active entries: 1, total entries: 1

And after that you can view the specific ACL in TCAM by the vmr_id:

RP/0/RSP0/CPU0:ASR9001-TCAM#show prm server tcam entries 144-LT vmr-id 3 100 np0 location 0/0/CPU0

Node: 0/0/CPU0:
----------------------------------------------------------------
ODS NP: 0, LT: 2, AppId: 2, VmrId 3, Offset 0, Entries 12, Shadow 0

5df02 V: 400c 0040 0000 0000 0000 0000 0000 0000 fe00 000a 11
      M: 0003 ffbf ffff ffff ffff ffff ffff ffff 0000 0000 
      R: 110a0000 00000003 28126100 00000000 00000000 00000000 00000000 00000000 
5df04 V: 400c 0041 0000 0000 0000 0202 006c 380a d2e6 000a 11
      M: 0003 ffbc ffff ffff ffff 0000 ff00 0000 0000 0000 
      R: 110a0000 00000003 b05d6300 00000000 00000000 00000000 00000000 00000000 
5df4a V: 400c 0043 0000 0000 0000 0000 006c 380a d2e6 000a 11
      M: 0003 ffbc ffff ffff ffff ffff ff00 0000 0000 0000 
      R: 110a0000 00000003 b05d6300 00000000 00000000 00000000 00000000 00000000 
5df4c V: 400c 0041 0000 0000 0000 05ea 006c 380a d2e6 000a 11
      M: 0003 ffbc ffff ffff ffff 0001 ff00 0000 0000 0000 
      R: 110a0000 00000003 c0156e00 00000000 00000000 00000000 00000000 00000000 
5df90 V: 400c 0043 0000 0000 0000 0000 006c 380a d2e6 000a 11
      M: 0003 ffbc ffff ffff ffff ffff ff00 0000 0000 0000 
      R: 110a0000 00000003 c0156e00 00000000 00000000 00000000 00000000 00000000 
5e302 V: 400c 0040 0000 0000 0000 0000 006c 380a 0282 a50a 11
      M: 0003 ffbf ffff ffff ffff ffff ff00 0000 0000 0000 
      R: 110a0000 00000003 30126100 00000000 00000000 00000000 00000000 00000000 
5eb02 V: 400c 0040 0000 0000 0000 0000 0000 0000 0000 000a 11
      M: 0003 ffbf ffff ffff ffff ffff ffff ffff ffff ff00 
      R: 111a0000 00000003 40126100 00000000 00000000 00000000 00000000 00000000 
5ef02 V: 400c 0040 0000 0000 0000 0000 0000 0000 0000 a8c0 11
      M: 0003 ffbf ffff ffff ffff ffff ffff ffff ffff 0000 
      R: 111a0000 00000003 48126100 00000000 00000000 00000000 00000000 00000000 
5f302 V: 400c 0040 0000 0000 0000 0000 0000 0000 0000 0000 11
      M: 0003 ffbf ffff ffff ffff ffff ffff ffff ffff ffff 
      R: 112a0100 11000003 50126100 016c380a 00000000 00000000 00000000 00000000 
5f304 V: 400c 0040 0000 0000 0000 0000 0000 0000 0000 0000 11
      M: 0003 ffbf ffff ffff ffff ffff ffff ffff ffff ffff 
      R: 111a0000 00000003 58126100 00000000 00000000 00000000 00000000 00000000 
5fb02 V: 400c 0040 0000 0000 0000 0000 0000 0000 0000 0000 11
      M: 0003 ffbf ffff ffff ffff ffff ffff ffff ffff ffff 
      R: 111a0000 00000003 60126100 00000000 00000000 00000000 00000000 00000000

Lets see this line for example:

5df4c V: 400c 0041 0000 0000 0000 05ea 006c 380a d2e6 000a 11
      M: 0003 ffbc ffff ffff ffff 0001 ff00 0000 0000 0000 
      R: 110a0000 00000003 c0156e00 00000000 00000000 00000000 00000000 00000000 

• protocol
• for tcp its same value as for ipv4... for some reason... for icmp V: 01 M: 00 which is 1 with mask so icmp, for udp its V: 11 M: 00 which is 17 udp, not sure why tcp is different
• action: permit or deny
• src ip and mask
• src port and mask
• dst ip and mask
• dst port and mask

Not sure about the other fields but this is enough to identify rules.

Now for the weird thing and my question... as you can see for the TCP permit rules, it actually takes up 2 TCAM entry spaces... one with the port information and one without, see here:

5df4c V: 400c 0041 0000 0000 0000 05ea 006c 380a d2e6 000a 11
      M: 0003 ffbc ffff ffff ffff 0001 ff00 0000 0000 0000 
      R: 110a0000 00000003 c0156e00 00000000 00000000 00000000 00000000 00000000 
5df90 V: 400c 0043 0000 0000 0000 0000 006c 380a d2e6 000a 11
      M: 0003 ffbc ffff ffff ffff ffff ff00 0000 0000 0000 
      R: 110a0000 00000003 c0156e00 00000000 00000000 00000000 00000000 00000000 

this is the same for UDP permit rules as well... regards to the doubling of permit rules

I did some testing and for deny rules the case as you would suppose, only one entry with the ports assigned correctly...

So here is my question, why would permit rules with a single port or ports which can be matched by a single rule use two TCAM slots? Model is an ASR9001.

[smilstea]

We have two entries for the sequence numbers that specify a source and destination instead of any, for the other permit statements they take up 1 entry such as sequence 10. You also don't necessarily need a deny any any at the end as there is an implicit deny any any applied.

RP/0/RSP0/CPU0:ASR-9001-B#show pfilter-ea fea ipv4-acl tcam-test loc 0/0/cPU0
Wed Mar 8 17:18:54.876 UTC

Rgn tcam-test, lkup v4, Dir In, Chan 1, acl_id 2, vmr_id 3, num_aces 9, num_tcam_entries 11, refcnt 1.
ACE List for This Region:
seq_num 10,tcam_entries 1,stats_ace_id 0x531760 (0x61bb00) new 0
seq_num 20,tcam_entries 2,stats_ace_id 0x531761 (0x61bb08) new 0
seq_num 30,tcam_entries 2,stats_ace_id 0x531762 (0x61bb10) new 0
seq_num 40,tcam_entries 1,stats_ace_id 0x531763 (0x61bb18) new 0
seq_num 50,tcam_entries 1,stats_ace_id 0x531764 (0x61bb20) new 0
seq_num 60,tcam_entries 1,stats_ace_id 0x531765 (0x61bb28) new 0
seq_num 70,tcam_entries 1,stats_ace_id 0x531766 (0x61bb30) new 0
seq_num 100,tcam_entries 1,stats_ace_id 0x531767 (0x61bb38) new 0
seq_num 2147483647,tcam_entries 1,stats_ace_id 0x531768 (0x61bb40) new 0
Intf List for This Region:
Te0/0/2/3, hw_count 0.

Sam

[gergo.andrasi]

Hi Sam,

thank you for the reply, it is true, that the deny any any at the end is not needed. I am just wondering why the double entry is needed for seq 20 and 30.... seq 40 also specifies a source and a destination and still uses only one entry.

Seems to me that specifying a port would be the trigger to this behavior.

Two rules are created for seq 30, which in the tcam list are referenced as 5df4c and 5df90. To me looks like 5df4c wholly covers 5df90, meaning that 5df90 seems to be redundant... i am wondering if that is the case or if not, what is the reason behind creating a second rule without the port definition in 5df90?

g

[smilstea]

I see what you mean, I changed the permit entries to deny and their tcam count went down to 1. I need to do some investigation internally to see why there is this difference.

Sam

[smilstea]

I got a response from development:

The behaviour seen is expected.

When we filter TCP/UDP protocol with port fields, two TCAM entries will be created. First entry is to match non-fragmented packet which has complete L3 and L4 header and the second entry is to match fragmented packets which has only L3 header and does not have any L4 header. Thats why in second TCAM entry there is no port field programmed. The purpose of this is to permit all the fragmented packets as well.

But incase of deny ACE there is no need to create another tcam entry to match fragmented packets because the initial packets which has L3/L4 header will anyways be dropped by first tcam entry and reassembly of fragmented packets anyhow will not happen and gets dropped. Thats the reason we don't create two tcam entries for deny ACE.

Sam

[gergo.andrasi]

Right, fragmentation...

Does the ASR reassemble fragmented packets before forwarding? Or will the fragments (which come after the original packet with L4 information was dropped) be forwarded for example if you have an acl like this?

10 deny udp any any
20 permit ipv4 any any

So my question is will the asr drop the fragmented packets or will the router pass the remaining fragments down to the destination and it is the destinations responsibility to drop the invalid packets?

[smilstea]

Reassembly only happens on for-us packets as that is cpu intensive.

For regular data traffic we forward the packets as is, otherwise the next hop would just have to fragment the packet, then the next hop reassemble, etc etc.

Fragments can be completely valid, but its not upto the transit node (asr9k) to determine that for transit packets.

Sam

[gergo.andrasi]

HI Sam,

was my intuition as well, just wanted to make sure, thanks.