04-30-2011 01:20 PM - edited 03-01-2019 04:39 PM
1. Purpose
Explain how Access Control Lists (ACL) configured in Catalyst 6500 fit into its ACL TCAM and LOUs, and also to optimize the ACL configurations.
2. Understanding the usage of ACL TCAM :
The ACL TCAM is organized into series of VALUES (aka Entries) and MASK.
Fig 1: ACL TCAM Organization – Masks, Values and Results
Let us consider an example ACL and see how the switch populates the TCAM.
Example:
ip access-list extended EXAMPLE
permit ip any host 10.1.2.100
deny ip any host 10.1.68.101
deny ip any host 10.33.2.25
permit tcp any any eq 22
deny tcp any any eq 23
deny udp any any eq 514
permit tcp any any eq 80
permit udp any any eq 161
First three ACEs check destination IP address only and rest of the ACEs check destination TCP/UDP port. So, the switch need two Masks. For every mask allocated, we can have maximum of 8 entries, as shown in the above figure.
For the above mentioned ACL, the TCAM will look like as mentioned below. A bit value of 0 in mask means that the corresponding bit position in the value is not compared with the actual traffic being classified (also known as don’t care bit).
Mask is 104 bits length and compares Source-IP, Destination-IP, L4 port type (TCP/UDP), Source L4 port and Destination L4 port.
Fig 2: Usage of TCAM Masks and Value
3. ACL TCAM and Logical Operation Units (LOUs)
ACLs use Logical Operations like:
Lesser than (lt)
Greater than (gt)
Equal to (eq)
Not equal to (neq)
Range
The Catalyst 6500 hardware provides Logical Operations in Hardware do these operations. The ACL hardware has 128 hardware LOUs and it is a global resource. Global LOUs are divided equally for source port and destination port operations and 2 LOUs needed to match port range. Totally, we can do 32 source port range matches and 32 destination port range matches in hardware.
Also per ACL we have 10 bits called L4Ops that select from the global LOUs, in order to perform a port operation for a given ACE within the ACL.
So across all ACLs combined we can do 32 source port range (or TCP flag) matches and 32 destination port range (or TCP flag) matches in hardware using the LOUs, and for any specific ACL we can do 10 port range (or TCP flag) operations using the LOUs.
If there are more port ranges in the ACL than the available LOUs and L4OPs, they are expanded in software into equivalent entries. This is known as PORT EXPANSION.
Fig 3: Logical Operation Units in Catalyst 6500
4. Monitoring the utilization of TCAM and LOUs
6500A# show tcam counts
Used Free Percent Used Reserved
---- ---- ------------ --------
Labels: 20 4076 0
ACL_TCAM
--------
Masks: 3549 547 86 72
Entries: 19740 13028 60 576
QOS_TCAM
--------
Masks: 4 4092 0 18
Entries: 22 32746 0 144
LOU: 64 64 50
ANDOR: 1 15 6
ORAND: 0 16 0
ADJ: 0 2048 0
6500A# show platform hardware capacity acl
ACL/QoS TCAM Resources
Key: ACLent - ACL TCAM entries, ACLmsk - ACL TCAM masks, AND - ANDOR,
QoSent - QoS TCAM entries, QOSmsk - QoS TCAM masks, OR - ORAND,
Lbl-in - ingress label, Lbl-eg - egress label, LOUsrc - LOU source,
LOUdst - LOU destination, ADJ - ACL adjacency
Module ACLent ACLmsk QoSent QoSmsk Lbl-in Lbl-eg LOUsrc LOUdst AND OR ADJ
5 60% 86% 0% 0% 1% 1% 50% 50% 0% 0% 0%
6500A# sho tcam lou
LOU # Opcode Port# CapMap Count Ace Count
----- ------ ----- ------------ ---------
0 1 1024 10 1944
64 1 33701 7 9
65 4 33433 7 9
66 1 1024 6 36
<snip>
Opcode: LT = 1, GT = 2, NEQ = 3 and RANGE = 4.
5. Optimization the usage of TCAM resources:
5.1 Defining ACEs and Order dependency:
Consider an ACL (similar to the one discussed in Section 2 above) starts with 9 ACEs checking destination IP address and 5 ACEs checking destination L4 port, then we need totally 3 masks.
Mask (104 bits) | Value | Result |
00000000 FFFFFFFF 00 0000 0000 | < totally 8 entries > | < Permit or Deny > |
00000000 FFFFFFFF 00 0000 0000 | < 1 entry > | < Permit or Deny > |
00000000 00000000 00 0000 FFFF | < totally 5 entries > | < Permit or Deny > |
It is NOT correct to directly correlate number of ACEs configured to number of masks/entries used.
As you might be aware, an ACL and the TCAM are order dependent. For 2 ACEs to share the same mask besides having the don’t care bits in the same locations they need to be adjacent to each other in the ACL definition. One example of entries that can mask share would be say a subnet:
Permit any host 192.10.10.0
Permit any host 192.10.10.1
192.10.10.0 192.10.10.1 both have the same mask 255.255.255.254 (the last bit is don’t care).
So the 2 entries result in 1 TCAM entry with 1 mask: Permit 192.10.10.0 255.255.255.254
If in another ACL adjacent 2 entries are:
Permit 192.11.11.0
Permit 192.11.11.1
This would result in 1 TCAM entry with 1 mask 255.255.255.254 and since the mask is the same we will use the second of the 8 entries corresponding to this mask.
TCAM entry for the above 2 will look thus:
Mask (104 bits) | Value | Result |
00000000 FFFFFFFE 00 0000 0000 | xxxxxxxx 192.10.10.0 xx xxxx xxxx | Permit |
xxxxxxxx 192.11.11.0 xx xxxx xxxx | Permit |
Definition of ACEs and its order determines the usage of ACL TCAM Masks and entries.
5.2 Defining Logical Operations in an ACL to optimize usage of TCAM/LOUs
Let us consider following ACL entry:
Permit any host 192.10.10.1 tcp range 1020 1023 any
If we cannot use the LoUs, it should be expanded it out to individual ACEs.
Permit any host 192.10.10.1 tcp 1020 any
Permit any host 192.10.10.1 tcp 1021 any
Permit any host 192.10.10.1 tcp 1022 any
Permit any host 192.10.10.1 tcp 1023 any
A LOU can be referenced by multiple ACEs in the same ACL. Taking our example above, we could have another set of ACEs in the same ACL as
Permit any host 195.1.2.3 tcp range 1020 1023 any
If we had an LOU allocated for the logical operation “range 1020 1023” the 2 ACEs will share it. Every usage of such Logical operation within an ACL is called an instance.
When expanding an ACL for lack of LOUs, the number of ACEs it expands to depends on 2 things:
After expansion the ACEs will increase by product of m and n. (m * n)
This document has an Appendix (Section 8) that lists methods to calculate the expansion factor (m)
(1) Calculate a priority (weight) to each Logical Operation.
Calculated priority for a particular port range is based on the following factors:
a) How many ACEs will an ACE with this port range expand to if an LOU cannot be used (expansion factor)? We need to include and consider any "gt"/"lt" operations used in the same way as other port ranges. These should be put in hardware since they may expand to a large number of aces.
b) How many ACE’s use this port range (ref count)?
The priority for each range will be determined by (expansion factor – 1) * ref count.
(2) Determine what LOUs are already allocated in hardware (for other ACLs or from the current successful HW programming of a given ACL). From this exercise we determine if the current LoU allocation is optimal or not. We want High priority operations (identified in Step1 above) to be allocated in hardware.
(3) ACEs with operations on TCP flags must always be allocated an LOU and hence are highest priority.
(4) It is possible to eliminate the need for a L4OP. As an example say an ACE of,
Permit any host 192.10.10.1 lt 1025
could be redefined as
Permit any host 192.10.10.1 eq 1024
Permit any host 192.10.10.1 lt 1024
The latter does not consume any LoU since the range 0 to 1023 can be represented using TCAM mask and hence no L4OP.
(5) Make sure one L4OP is available for TCP flags (within the first 10) for the "established" TCP flags match. Configure "permit tcp any any established" to make sure the TCP flags also gets allocated a L4OP. [ similarly, configure one ACE that use a given high priority port range, and do same thing for all port ranges, based on their priority ]
(6) Please be careful because you do not want to alter the behavior of the ACL by rearranging the ACEs. Consider adding some dummy ACEs (such as "deny tcp host 0.0.0.0 host 0.0.0.0 range 3500 3530") which are matching basically some invalid traffic at the top of the ACL for the critical ranges. Basically these ACEs do not have any effect except making sure the port range is allocated L4OP/LOU.
8. APPENDIX
8.1. Calculating the Expansion factor (m)
The expansion factor depends on the actual values since it is dependent on the bit position which is determined by the binary representation of the layer 4 port values. Explained below is the mathematics behind calculating the expansion factor.
(a) Expansion factor (m) for Greater Than (GT) case
To calculate a Greater Than case first convert the port number to binary. Add leading 0s until the total number of bits are 16 (since the Layer4 port numbers are maximum 16 bits)
Example: permit any host 192.10.1.1 gt 35 any
The binary representation of 35 (after adding leading 0 to make the number of bits 16) is
0000 0000 0010 0011
GT 35 expands into (- below means don’t care)
1--------------- covers range 32768 to 65535
-1-------------- covers range 16384 to 32767
--1------------- covers range 8192 to 16383
---1------------ covers range 4096 to 8191
----1----------- covers range 2048 to 4095
-----1---------- covers range 1024 to 2047
------1--------- covers range 512 to 1023
-------1-------- covers range 256 to 511
--------1------- covers range 128 to 255
---------1------ covers range 64 to 127
----------11---- covers range 48 to 63
----------1-1--- covers range 40 to 47
----------1--1-- covers range 36 to 39
So in case of GT the expansion factor m is the number of 0s in the binary representation of the number (after adding leading 0s to make the number of bits 16).
(b) Expansion factor (m) for Less Than (LT) case
To calculate a less than case first convert the port number to binary.
Example: permit any host 192.10.1.1 gt 35 any
The binary representation of 35 is 10 0011
LT 35 (i.e. entries from 0 to 34) expands into: (where – means don’t care)
0----- covers range from 0 to 31
10000- covers 32, 33
100010 covers 34
So in case of LT the expansion factor m is the number of 1s in the binary representation of the number.
(c) Expansion factor (m) for ranges
Let us consider the example of range 50 62
First convert them into binary representation:
62 = 11 1110
50 = 11 0010
If you take a look you see that the leading (most significant) 2 bits are same and they differ in only the last 4 bits. We need to disregard the same bits and concentrate on the bits that follow (including) the first bit where they start to differ (shown in orange above).
Form the diff group for the two numbers which are the lower bits including the orange one above.
In our example the diff group for 62 is 1110 and for 50 it is 0010
To get the worst case value for the expansion factor (m) count the number of 1 in the binary representation of higher number, limited to the group of bits that differ (aka diff group). In our example that would be for 62 and the number of 1s is 3. Then look at the number of bits that differ for the lower number and count the number of 0s. That in our example would be for 50 and the count of 0s is 3. Now get the sum of those 2 which is 6. Hence the worst case expansion for this range is 6.
For most cases this approximation may be good enough.
However, to get the exact number we need to weed out the overlaps. The overlap is found by comparing the first entries from the GT and LT expansion of the diff group and if they overlap then we remove them.
To find the overlaps take the diff group from the higher number and apply the LT algorithm to it. That gives (group LT):
0---
10--
110-
Then we place all 1 in the don’t care (-) above to give (group LT#):
0111
1011
1101
Also form a group LT## where we place all 0 in the don’t care (-) above to give (group LT##)
0000
1000
1100
Now we take the diff group from the lower number and apply the GT algorithm to it (no need to expand it to 16 bits). That gives (group GT):
1---
01--
0011
Next we place all 0 in the don’t care (-) above to give (GT#):
1000
0100
0011
Now compare the first entry from LT# (0111) with all the entries in GT#. If it is greater than any one of those entries (in this case it is greater than 0011), take the corresponding entry in LT## (which is 0011) and compare it with all entries in GT#. Since 0000 is less than 0011 the first entry 0--- in LT is part of overlap and can be discarded.
Next compare the first entry from GT# (1000) with all the entries in LT##. Since 1000 is less than at least one entry in the group LT## (in this case 1100) it is also part of the overlap and can be safely discarded.
So the accurate expansion count is 6 – 2 (since we are discarding 2 entries due to overlap) = 4
The actual expanded entries would look like this (you need to put back the bits besides the diff group):
11 10-- range 56 to 59 (from the LT group)
11 110- range 60 to 61 (from the LT group)
11 01-- range 52 to 55 (from the GT group)
11 0011 equal 51 (from the GT group)
If only Source (or destination) ports are used the worst case expansion is 16.
If range operation on both source and destination ports are used then first calculate the expansion factor for source and destination port ranges individually (say src_m and dest_m) and the final expansion factor is the product of the two (m = src_m * dest_m). Consequently, the worst case expansion factor is 256 (16 * 16), when ranges are used for both: source and destination ports.
Since all the above mathematical calculations are coded in an catalyst 6500 the easiest way to get the expansion factor (m) is to use the Catalyst 6500 to do the work. Hence better to use a staging box (rather than a production box).
Create a test ACL and define first 12 ACEs to use logical operations (this will lock in the available L4OPs and LoUs per ACL and force the FM to do the ACL expansion)
Now define an ACE with the range (or logical operation) for which you want to calculate the expansion factor as the 13th ACE of this test ACL.
Apply the test ACL to an interface (needs to be in Admin up state. The line protocol state does not matter). If using a production box please make sure that this interface has tcam priority low configured so that it does not pull out any TCAM entries from the TCAM.
Do a show fm interface <int> (where int is the interface to which the test ACL was applied above).
Look at the output to get the expansion factor.
Example:
Router#sh ip access-lists test
Extended IP access list test
10 permit tcp any any range 10 ftp-data
20 permit tcp any any range ftp-data 30
30 permit tcp any any range 30 40
40 permit tcp any any range 40 50
50 permit tcp any any range 50 60
60 permit tcp any any range 60 gopher
70 permit tcp any any range gopher www
80 permit tcp any any range www 90
90 permit tcp any any range 90 100
100 permit tcp any any range 100 pop3
110 permit tcp any any range pop3 120
120 permit tcp any any range 120 130
If we want to find the expansion factor for example range 14000-14026:
Router(config)#ip access-list extended test
Router(config-ext-nacl)#permit tcp any any range 14000 14026
Router(config-ext-nacl)#end
Router#sh ip access-lists test
Extended IP access list test
10 permit tcp any any range 10 ftp-data
20 permit tcp any any range ftp-data 30
30 permit tcp any any range 30 40
40 permit tcp any any range 40 50
50 permit tcp any any range 50 60
60 permit tcp any any range 60 gopher
70 permit tcp any any range gopher www
80 permit tcp any any range www 90
90 permit tcp any any range 90 100
100 permit tcp any any range 100 pop3
110 permit tcp any any range pop3 120
120 permit tcp any any range 120 130
130 permit tcp any any range 14000 14026
Router(config)#int f1/48
Router(config-if)#ip access-group test in
Router(config-if)#end
Router#sh fm int f1/48
Interface: FastEthernet1/48 IP is enabled
hw_state[INGRESS] = not reduced, hw_state[EGRESS] = not reduced
mcast = 0
priority = 0
flags = 0x4
parent[INGRESS] = none
inbound label: 104
Feature IP_ACCESS_INGRESS:
ACL: test
-----------------------------------------------------------------------------
FM_FEATURE_IP_ACG - Acl Name: test Direction:Ingress
=============================================================================
DPort - Destination Port SPort - Source Port Pro - Protocol
PT - Packet Type DPT - Dst. Packet Type SPT - Src. Packet Type
X - XTAG TOS - TOS Value Res - VMR Result
RFM - R-Recirc. Flag MRTNPC - M-Multicast Flag R - Reflexive flag
- F-Fragment flag - T-Tcp Control N - Non-cachable
- M-More Fragments - P-Mask Priority(H-High, L-Low)
Adj. - Adj. Index C - Capture Flag T - M(Mask)/V(Value)
FM - Flow Mask NULL - Null FM SAO - Source Only FM
DAO - Dest. Only FM SADA - Sour.& Dest. Only VSADA - Vlan SADA Only
ISADA - Intf. SADA FF - Full Flow VFF - Vlan Full Flow
IFF - Intf. FF F-VFF - Either FF or VFF IFF-FF - Either IFF or FF
A-VSD - Atleast VSADA A-FF - Atleast FF A-VFF - Atleast VFF
A-SON - Atleast SAO A-DON - Atleast DAO A-SD - Atleast SADA
SHORT - Shortest ISADA-L- ISADA Least FF-L - FF Least
IFF-L - IFF Least A-SFF - Any short than FF A-EFF - Any except FF
A-EVFF - Any except VFF SA-L - Source Least DA-L - Dest. Least
SADA-L - SADA Least FF-LESS- FF Less N-FF - Not FF
N-IFF - Not IFF A-LVFF - Any less than VFF FULL - Full Pkt Type
EUI - EUI 64 Pkt Type EMBD - Embedded Pkt Type ELNK - EUI Link Overlap
ESIT - EUI Site Overlap LINK - Link Pkt Type SITE - Site Pkt Type
ERR - Flowmask Error
+----+-+---------------+---------------+-----+-----+---+---+-+---+------+----+------+
|Indx|T| Dest Ip Addr | Source Ip Addr|DPort|SPort|Pro|RFM|X|ToS|MRTNPC|Adj.| FM |
+----+-+---------------+---------------+-----+-----+---+---+-+---+-----+----+------+
1 V 0.0.0.0 0.0.0.0 0 0 6 -F- 0 0 ----L- ---- SHORT
M 0.0.0.0 0.0.0.0 0 0 255 010 0 0
TM_PERMIT_RESULT
2 V 0.0.0.0 0.0.0.0 0 0 6 --- 0 0 ----L- ---- SHORT
M 0.0.0.0 0.0.0.0 0 0 255 000 0 0
Destination: RANGE 10 - 20
TM_PERMIT_RESULT
3 V 0.0.0.0 0.0.0.0 0 0 6 -F- 0 0 ----L- ---- SHORT
M 0.0.0.0 0.0.0.0 0 0 255 010 0 0
TM_PERMIT_RESULT
4 V 0.0.0.0 0.0.0.0 0 0 6 --- 0 0 ----L- ---- SHORT
M 0.0.0.0 0.0.0.0 0 0 255 000 0 0
Destination: RANGE 20 - 30
TM_PERMIT_RESULT
5 V 0.0.0.0 0.0.0.0 0 0 6 -F- 0 0 ----L- ---- SHORT
M 0.0.0.0 0.0.0.0 0 0 255 010 0 0
TM_PERMIT_RESULT
6 V 0.0.0.0 0.0.0.0 0 0 6 --- 0 0 ----L- ---- SHORT
M 0.0.0.0 0.0.0.0 0 0 255 000 0 0
Destination: RANGE 30 - 40
TM_PERMIT_RESULT
7 V 0.0.0.0 0.0.0.0 0 0 6 -F- 0 0 ----L- ---- SHORT
M 0.0.0.0 0.0.0.0 0 0 255 010 0 0
TM_PERMIT_RESULT
8 V 0.0.0.0 0.0.0.0 0 0 6 --- 0 0 ----L- ---- SHORT
M 0.0.0.0 0.0.0.0 0 0 255 000 0 0
Destination: RANGE 40 - 50
TM_PERMIT_RESULT
9 V 0.0.0.0 0.0.0.0 0 0 6 -F- 0 0 ----L- ---- SHORT
M 0.0.0.0 0.0.0.0 0 0 255 010 0 0
TM_PERMIT_RESULT
10 V 0.0.0.0 0.0.0.0 0 0 6 --- 0 0 ----L- ---- SHORT
M 0.0.0.0 0.0.0.0 0 0 255 000 0 0
Destination: RANGE 50 - 60
TM_PERMIT_RESULT
11 V 0.0.0.0 0.0.0.0 0 0 6 -F- 0 0 ----L- ---- SHORT
M 0.0.0.0 0.0.0.0 0 0 255 010 0 0
TM_PERMIT_RESULT
12 V 0.0.0.0 0.0.0.0 0 0 6 --- 0 0 ----L- ---- SHORT
M 0.0.0.0 0.0.0.0 0 0 255 000 0 0
Destination: RANGE 60 - 70
TM_PERMIT_RESULT
13 V 0.0.0.0 0.0.0.0 0 0 6 -F- 0 0 ----L- ---- SHORT
M 0.0.0.0 0.0.0.0 0 0 255 010 0 0
TM_PERMIT_RESULT
14 V 0.0.0.0 0.0.0.0 0 0 6 --- 0 0 ----L- ---- SHORT
M 0.0.0.0 0.0.0.0 0 0 255 000 0 0
Destination: RANGE 70 - 80
TM_PERMIT_RESULT
15 V 0.0.0.0 0.0.0.0 0 0 6 -F- 0 0 ----L- ---- SHORT
M 0.0.0.0 0.0.0.0 0 0 255 010 0 0
TM_PERMIT_RESULT
16 V 0.0.0.0 0.0.0.0 0 0 6 --- 0 0 ----L- ---- SHORT
M 0.0.0.0 0.0.0.0 0 0 255 000 0 0
Destination: RANGE 80 - 90
TM_PERMIT_RESULT
17 V 0.0.0.0 0.0.0.0 0 0 6 -F- 0 0 ----L- ---- SHORT
M 0.0.0.0 0.0.0.0 0 0 255 010 0 0
TM_PERMIT_RESULT
18 V 0.0.0.0 0.0.0.0 0 0 6 --- 0 0 ----L- ---- SHORT
M 0.0.0.0 0.0.0.0 0 0 255 000 0 0
Destination: RANGE 90 - 100
TM_PERMIT_RESULT
19 V 0.0.0.0 0.0.0.0 0 0 6 -F- 0 0 ----L- ---- SHORT
M 0.0.0.0 0.0.0.0 0 0 255 010 0 0
TM_PERMIT_RESULT
20 V 0.0.0.0 0.0.0.0 0 0 6 --- 0 0 ----L- ---- SHORT
M 0.0.0.0 0.0.0.0 0 0 255 000 0 0
Destination: RANGE 100 - 110
TM_PERMIT_RESULT
21 V 0.0.0.0 0.0.0.0 0 0 6 -F- 0 0 ----L- ---- SHORT
M 0.0.0.0 0.0.0.0 0 0 255 010 0 0
TM_PERMIT_RESULT
22 V 0.0.0.0 0.0.0.0 0 0 6 --- 0 0 ----L- ---- SHORT
M 0.0.0.0 0.0.0.0 0 0 255 000 0 0
Destination: RANGE 110 - 120
TM_PERMIT_RESULT
23 V 0.0.0.0 0.0.0.0 0 0 6 -F- 0 0 ----L- ---- SHORT
M 0.0.0.0 0.0.0.0 0 0 255 010 0 0
TM_PERMIT_RESULT
24 V 0.0.0.0 0.0.0.0 0 0 6 --- 0 0 ----L- ---- SHORT
M 0.0.0.0 0.0.0.0 0 0 255 000 0 0
Destination: RANGE 120 - 130
TM_PERMIT_RESULT
25 V 0.0.0.0 0.0.0.0 0 0 6 -F- 0 0 ----L- ---- SHORT
M 0.0.0.0 0.0.0.0 0 0 255 010 0 0
TM_PERMIT_RESULT
26 V 0.0.0.0 0.0.0.0 14026 0 6 --- 0 0 ----L- ---- SHORT
M 0.0.0.0 0.0.0.0 65535 0 255 000 0 0
TM_PERMIT_RESULT
27 V 0.0.0.0 0.0.0.0 14000 0 6 --- 0 0 ----L- ---- SHORT
M 0.0.0.0 0.0.0.0 65520 0 255 000 0 0
TM_PERMIT_RESULT
28 V 0.0.0.0 0.0.0.0 14016 0 6 --- 0 0 ----L- ---- SHORT
M 0.0.0.0 0.0.0.0 65528 0 255 000 0 0
TM_PERMIT_RESULT
29 V 0.0.0.0 0.0.0.0 14024 0 6 --- 0 0 ----L- ---- SHORT
M 0.0.0.0 0.0.0.0 65534 0 255 000 0 0
TM_PERMIT_RESULT
30 V 0.0.0.0 0.0.0.0 0 0 0 --- 0 0 ----L- ---- SHORT
M 0.0.0.0 0.0.0.0 0 0 0 000 0 0
M 0.0.0.0 0.0.0.0 0 0 0 000 0 0
TM_L3_DENY_RESULT
The entries 26, 27, 28, 29 are the expanded ACEs to cover the port range for 14000 - 14026. The last (30) is for the default deny ip any any ACE. So the expansion factor m in this case is 4.
Very very useful DOC
Super useful DOC, has details that are hard to find in configuration guides.
Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: