L2 Handoff Scalability

[dm2020]

Hi All,

I am planning on installing a Cisco 9300X switch within my fabric site that will act as a L2 handoff border node to support migration. Looking at the Catalyst Center data sheet, the C9300X supports a maxium of 32,000 endpoints when deployed as an L2 border node, with the foot notes stating the following

"These numbers are the sum of the total numbers of endpoints both inside and outside the fabric site when the site has a border node with a Layer-2 handoff. A maximum of 6000 hosts can be connected outside the fabric for all platforms that support Layer-2 border handoff."

Its not clear from the above if the 32,000 is the total number of endpoints that are supported in the IP pools/VLANs that have been enabled on the L2 border, or if this number is the maxium supported endpoints for the entire fabric. For example, if we have 10 VLANs in migration, with 6,000 endpoints outside of the fabric in these VLANs, and 26,000 endpoints inside the fabric in the associated IP pools for these VLANs, does 32,000 suggest and that no more endpoints are supported in these or additional VLANs that need to be enabled for L2 handoff?

Or does simply introducing the C9300X as an L2 border into the fabric lower the scalability for the entire site to 32,000? We currently have one fabric that is planned to have >40k endpoints. Can we add a layer 2 border to this fabric to support features such as gateway outside the fabric or will this not be supported?

Hope this makes sense

[Andrii Oliinyk]

It seems to be simple:

"A maximum of 6000 hosts can be connected outside the fabric for all platforms that support Layer-2 border handoff.
The border node with a Layer-2 handoff contains a combination of local and remote LISP entries.
Local entries = LISP database
Remote entries = LISP map-cache"
If you plan to have >40K in fabric, with 9300X u under the risk of failure

32K=6K(database)+26K(map-cache) doesnt mean your Fabric is limited to 32K since this point, but particular L2-handoff it is. 

[dm2020]

Ok its still not quite clear. Perhaps its understanding what endpoints will be populdated in the remote LISP entries/map cache. Will these just be endpoints that have been migrated to the fabric that are part of the L2 handoff, or will these be all endpoints in the fabric (wired/wireless etc) that are associated to VNs/IP pools that are not configured for L2 handoff? So in other words, will ALL endoints in the fabric be populated in the L2 border's map-cache?

[Andrii Oliinyk]

Not necessarily, L2NB will maintain its map-cache for remote endpoints it's interesting in. But imagine the case somebody mistaken with L2VN assignment somewhere in the Fabric & thus added extra Ks to summary u thought should have never exceeded l.s. 30K (remember that playing with maximum values in production is not recommended practice).  

[jedolphi]

Hi @dm2020 , if there is a packet from outside L2HO heading for the fabric then L2BN will need to create one or more map cache entries for the destination inside fabric. If EPs (endpoints) outside L2HO only send packets to DC/Internet (pure S-N comms) then L2BN will only have map cache entries for border nodes. If EPs outside L2HO send packets to all endpoints inside fabric (very unlikely!) then L2BN will build map cache for all destinations inside the fabric. In other words L2BN populates map cache on demand based on what destinations it needs to know inside of fabric. You can monitor L2BN utilisation with command "show lisp platform". The 6000 limit is redundant and I've been trying to get it removed from data sheet.

Below is a show lisp platform from my lab 17.12.x 9300X for reference, you can see the limit is 32K L2 and 40K L3, however as a general rule (noting all rules can be broken, with conditions attached) I recommend to plan for max 50% utilisation (16K / 20K) since below CLI output is one dimensional scale, and in the real world there is no such thing as 1 dimensional networks.

9300X#show lisp plat
Parallel LISP instance limit: 2000
RLOC forwarding support:
IPv4 RLOC, local: OK
IPv6 RLOC, local: OK
MAC RLOC, local: Unsupported
IPv4 RLOC, remote: OK
IPv6 RLOC, remote: OK
MAC RLOC, remote: Unsupported
Support for signal+forward:
IPv4: OK
IPv6: OK
MAC: OK
Platform reported limits:
L3 limit: 40448
Total Current utilization: 0%
IPv4 multiplier: 1
IPv4 local EID counter: 0
IPv4 remote EID counter: 0
IPv4 remote EID idle counter: 0
IPv4 mapping cache full: no
IPv6 multiplier: 2
IPv6 local EID counter: 0
IPv6 remote EID counter: 0
IPv6 remote EID idle counter: 0
IPv6 mapping cache full: no
L2 limit: 32768
Total Current utilization: 0%
MAC multiplier: 1
MAC local EID counter: 0
MAC remote EID counter: 0
MAC remote EID idle counter: 0
MAC mapping cache full: no
Latest supported config style: Service and instance
Current config style: Service and instance
9300X#